In-memory-fuzzing in Linux (with GDB and Python)

06 Jan 2012


Probably, if you're reading this article, you already know what fuzzing means. In short, fuzz testing is a technique for testing software and searching vulnerabilities: targeted software is feeded with malformed input, hoping for something abnormal to occur.

In recent years many excellent frameworks have been published to help the tester in the development of an effective fuzzer (I like spike, peach and sulley very much).  But, as always, there are some experimental techniques, which in future will allow the evolution of the field: in this case I'm talking about in-memory fuzzing.

In-Memory Fuzzing is an advanced, and therefore even more complex, technique (but we'll see how to manage this complexity), which allows the tester to fuzz individual subroutines of the targeted program. This focused type of test has many advantages:

But how does an in-memory fuzzer work?

If we consider an application as chain of function, that receives an input, parses and processes it, and produces an output, we can describe in-memory fuzzing as a process that tests only a few specific rings of the chain (those dealing with parsing and processing).

Having clarified this point, is now easy to illustrate the main techniques used today:

Mutation Loop Insertion

Mutation Loop Insertion (MLI) modifies the target program inserting an infinite loop on the parsing subroutine, isolating it from from the rest of the function chain.

in memory loop fuzzing

This loop can test the targeted function with a large quantity of inputs in a very short time, eliminating superfluous program's code, and requires no interaction from the outside. This makes this method the fastest.

However it has also negative aspects, being the method more difficult to implement... It requires at least some knowledge of reverse engineering, and the ability to write code safely injectable inside an active process.

Snapshot Restoration Mutation

Snapshot Restoration Mutation (SRM) handle the program function chain is a different way: no code is injected, but, through the use of breakpoints, the fuzzer takes and restores snapshots of the process at the beginning and end of the tested function.

in memory snapshot fuzzing

The effect is similar to that of an infinite loop, as in the previous case, but managed and monitored by an external process.

This method has several advantages including the ability to restore the program at a clean state, and, of course, it does not require the writing of assembly code. This at a cost of a certain performance degradation.

Implementation

Ok, the practical part, let's start with a question...  Did you know that, since version 7.0, GDB can be scripted in python?

Yeah, I agree, it's a wonderful thing:  although at the time the development is not yet complete, with some adjustments, it is possible to exploit all the capabilities of the (u)nix debugger par excellence, within our python scripts.

Moreover, since version 7.0, GDB is able to take and restore snapshots of the debugged process, with the introduction of the checkpoint command.

Having found a so confortable "framework" (that, IMHO, has good probabilities to overcome similar solutions present in the Windows environment, also excellent as PyDbg), I could not resist the temptation to write an implementation of in-memory fuzzer for (u)nix environments.

The result was the creation of a small library to support GDB python scripting, with the fuzzer inside the examples directory. Let's look at its practical use...

 In-memory fuzzing (in practice)

The fuzzer is composed of two scripts: the first, in-memory-break.py, is used to find functions to test.

The script inserts breakpoints at the beginning of every function of the program and prints out their arguments in search of text strings. The goal is to find the function that parses the input.

Let's try to run the script against the vulnerable program contained in the same directory:

crossbower@yotsubox$ ./in-memory-break.py getdomain test@email.com
Breakpoint 1 at 0x4004b0
Breakpoint 2 at 0x4004d8
Breakpoint 3 at 0x4004e8
Breakpoint 4 at 0x4004f8
Breakpoint 5 at 0x400508
Breakpoint 6 at 0x400518
Breakpoint 7 at 0x400528
Breakpoint 8 at 0x40056c
Breakpoint 9 at 0x400590
Breakpoint 10 at 0x400600
Breakpoint 11 at 0x40062c
Breakpoint 12 at 0x40064d
Breakpoint 13 at 0x4007f0

Function <__libc_start_main@plt> at *0x4004f8:
	argument0 = 4196079 "UH"
	argument1 = 2
	argument2 = 140737488348088 "L"
	argument3 = 4196192 "H"
	argument4 = 4196176
	argument5 = 140737351962048 "UH"

Function <_init> at *0x4004b0:
	argument0 = 2
	argument1 = 140737488348088 "L"
	argument2 = 140737488348112
	argument3 = 0
	argument4 = 140737351885568
	argument5 = 140737351962048 "UH"

Function <call_gmon_start> at *0x40056c:
	argument0 = 2
	argument1 = 140737488348088 "L"
	argument2 = 140737488348112
	argument3 = 0
	argument4 = 140737351885568
	argument5 = 140737351962048 "UH"

Function <frame_dummy> at *0x400600:
	argument0 = 2
	argument1 = 140737488348088 "L"
	argument2 = 140737488348112
	argument3 = 0
	argument4 = 140737351885568
	argument5 = 140737351962048 "UH"

Function <__do_global_ctors_aux> at *0x4007f0:
	argument0 = 2
	argument1 = 140737488348088 "L"
	argument2 = 140737488348112
	argument3 = 0
	argument4 = 140737351885568
	argument5 = 140737351962048 "UH"

Function <strdup@plt> at *0x400508:
	argument0 = 140737488348816 "test@email.com"
	argument1 = 140737488348088 "L"
	argument2 = 140737488348112
	argument3 = 0
	argument4 = 140737351885568
	argument5 = 140737351962048 "UH"

Function <parse> at *0x40064d:
	argument0 = 6295568 "test@email.com"
	argument1 = 140737488348831 "SSH_AGENT_PID=2952"
	argument2 = 0
	argument3 = 30803244232763745
	argument4 = 140737351888448
	argument5 = 140737348377640

Function <strtok@plt> at *0x400528:
	argument0 = 6295568 "test@email.com"
	argument1 = 4196426 "@"
	argument2 = 0
	argument3 = 30803244232763745
	argument4 = 140737351888448
	argument5 = 140737348377640

Function <strtok@plt> at *0x400528:
	argument0 = 0
	argument1 = 4196426 "@"
	argument2 = 6295573 "email.com"
	argument3 = 6295573 "email.com"
	argument4 = 6295568 "test"
	argument5 = 140737348377640

Function <strcpy@plt> at *0x400518:
	argument0 = 140737488346768
	argument1 = 6295573 "email.com"
	argument2 = 6295582
	argument3 = 6295583
	argument4 = 0
	argument5 = 140737348377640

Function <print_domain> at *0x40062c:
	argument0 = 140737488346768 "email.com"
	argument1 = 6295584
	argument2 = 140737488346777
	argument3 = 0
	argument4 = -72340172838076673
	argument5 = -72219847665292440

Function <printf@plt> at *0x4004d8:
	argument0 = 4196412 "Domain is %s\n"
	argument1 = 140737488346768 "email.com"
	argument2 = 140737488346777
	argument3 = 0
	argument4 = -72340172838076673
	argument5 = -72219847665292440

Domain is email.com
Function <printf@plt> at *0x4004d8:
	argument0 = 4196463 "Domain is valid? %s\n"
	argument1 = 4196428 "YES"
	argument2 = 140737351888368
	argument3 = 4196425
	argument4 = 1
	argument5 = 4196425

Domain is valid? YES
Function <__do_global_dtors_aux> at *0x400590:
	argument0 = 140737488347632 "("
	argument1 = 140737488347632 "("
	argument2 = 140737354127792
	argument3 = 4
	argument4 = 0
	argument5 = 4

[Inferior 1 (process 18083) exited normally]

I've highlighted the parsing function. This case was simple, because the binary was not stripped, making possible to print function names. But even in the case of stripped binaries, it's possible to find the function we are interested in, analyzing and trying to decode the arguments.

The vulnerable program simply try to extract the domain from an email address: an overly long domain triggers a buffer overflow on the stack.

Now that we know the function to test and the input type, let's try to see if our in-memory fuzzer is able to detect the bug:

cross@yotsubox$ ./in-memory-fuzz.py parse getdomain test@email.com
Breakpoint 1 at 0x400651

Breakpoint 1, 0x0000000000400651 in parse ()
fuzz loop: 1
string len: 15
0x601030:	 "test@email.com"
Domain is email.com
0x000000000040072f in main ()
Switching to process 4997
#0  0x0000000000400651 in parse ()
fuzz loop: 2
string len: 16
0x601030:	 "test@email.comA"
Domain is email.comA
0x000000000040072f in main ()
Switching to process 4998
#0  0x0000000000400651 in parse ()
fuzz loop: 3
string len: 17
0x601030:	 "test@email.comAA"
Domain is email.comAA
0x000000000040072f in main ()
Switching to process 4999
...
...
...
#0  0x0000000000400651 in parse ()
fuzz loop: 1031
string len: 1045
0x601030:	 "test@email.com", 'A' <repeats 186 times>...
Domain is email.comAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
0x000000000040072f in main ()
Switching to process 7202
#0 0x0000000000400651 in parse ()
fuzz loop: 1032
string len: 1046
0x601030:	 "test@email.com", 'A' <repeats 186 times>...
Domain is email.comAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Domain is valid? YES

Program received signal SIGSEGV, Segmentation fault.
0x0000000000400849 in ?? ()
Switching to process 7203
#0  0x0000000000400651 in parse ()
fuzz loop: 1033
string len: 1047
0x601030:	 "test@email.com", 'A' <repeats 186 times>...
[Switching to process 7204]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.

#
# The program has crashed! Stack exhaustion or bug???
# Now is your turn, have fun! :P
#

A debugging session is active.

	Inferior 1 [process 7204] will be killed.

Quit anyway? (y or n) y

As you can see, fuzzing loop number 1032 triggered the bug: the input strings are simply generated appending 'A' characters to a valid email address, and when the domain part reaches a critical length, a buffer on the stack is overflowed, overwriting the return address (and stack canaries).

Although this is script is very simple, it's a basis to build on to create more complex fuzzer. In addition it illustrates several GDB features very useful in this field:

If you are interested in GDB python scripting or in-memory fuzzing on (u)nix system, you can visit the project website:

GDB Python Utils:  https://github.com/crossbowerbt/GDB-Python-Utils/

You will find the scripts illustrated in this article under the examples directory of the project.

I'm also trying to maintain a good documentation for the support library, so you may give a look at the "snippet" page of the wiki (https://github.com/crossbowerbt/GDB-Python-Utils/wiki/Snippets/) to see the implemented features.

Post scripta

Some additions to the original article, born after discussing some issues with visitors in the comment section of the previous site.

Hints for 32bit systems

The fuzzer was specifically developed for 64bit systems: if you want to use it against 32bit application you must adapt it (if you send me an email I can give you a few hints...)

On 32-bit systems the arguments are passed on the stack: you can find them adding their offset to the %esp register.

Example, a 32bit process read() call:

(gdb) x/4x $sp
0xffffc09c:	0xf6bb17da	0x0000000b	0x09286d90	0x00002000

The first dword encountered is the return address, we are not interested in it...

The first argument (the file descriptor):

(gdb) x/x $sp+4
0xffffc0a0:	0x0000000b

The second argument (the buffer):

(gdb) x/x $sp+8
0xffffc0a4:	0x09286d90

The third argument (max read length):

(gdb) x/x $sp+12
0xffffc0a8:	0x00002000

To make the fuzzer work on 32-bit system you must change the code that read/modify the arguments of functions (I will probably change the fuzzer to make it multi-architecture, when I have time).

In the meanwhile the interface has changed...

Python integration in GDB has improved. Here is an interesting video that shows what is possible to do now: