In-memory fuzzing in Linux (with GDB and Python)
06 Jan 2012
Probably, if you're reading this article, you already know what fuzzing means. In short, fuzz testing is a technique for testing software and searching vulnerabilities: targeted software is feeded with malformed input, hoping for something abnormal to occur.
In recent years many excellent frameworks have been published to help the tester in the development of an effective fuzzer (I like spike, peach and sulley very much). But, as always, there are some experimental techniques, which in future will allow the evolution of the field: in this case I'm talking about in-memory fuzzing.
In-Memory Fuzzing is an advanced, and therefore even more complex, technique (but we'll see how to manage this complexity), which allows the tester to fuzz individual subroutines of the targeted program. This focused type of test has many advantages:
- The fuzzing process is faster and allows a complete code coverage, since it's possible to select the piece of code to test.
- Targeting a specific program subroutine, allows to bypass any obfuscation or decoding of input data, making the development of fuzzing tools simpler, from this point of view.
But how does an in-memory fuzzer work?
If we consider an application as chain of function, that receives an input, parses and processes it, and produces an output, we can describe in-memory fuzzing as a process that tests only a few specific rings of the chain (those dealing with parsing and processing).
Having clarified this point, is now easy to illustrate the main techniques used today:
Mutation Loop Insertion
Mutation Loop Insertion (MLI) modifies the target program inserting an infinite loop on the parsing subroutine, isolating it from from the rest of the function chain.
This loop can test the targeted function with a large quantity of inputs in a very short time, eliminating superfluous program's code, and requires no interaction from the outside. This makes this method the fastest.
However it has also negative aspects, being the method more difficult to implement... It requires at least some knowledge of reverse engineering, and the ability to write code safely injectable inside an active process.
Snapshot Restoration Mutation
Snapshot Restoration Mutation (SRM) handle the program function chain is a different way: no code is injected, but, through the use of breakpoints, the fuzzer takes and restores snapshots of the process at the beginning and end of the tested function.
The effect is similar to that of an infinite loop, as in the previous case, but managed and monitored by an external process.
This method has several advantages including the ability to restore the program at a clean state, and, of course, it does not require the writing of assembly code. This at a cost of a certain performance degradation.
Implementation
Ok, the practical part, let's start with a question... Did you know that, since version 7.0, GDB can be scripted in python?
Yeah, I agree, it's a wonderful thing: although at the time the development is not yet complete, with some adjustments, it is possible to exploit all the capabilities of the (u)nix debugger par excellence, within our python scripts.
Moreover, since version 7.0, GDB is able to take and restore snapshots of the debugged process, with the introduction of the checkpoint command.
Having found a so confortable "framework" (that, IMHO, has good probabilities to overcome similar solutions present in the Windows environment, also excellent as PyDbg), I could not resist the temptation to write an implementation of in-memory fuzzer for (u)nix environments.
The result was the creation of a small library to support GDB python scripting, with the fuzzer inside the examples directory. Let's look at its practical use...
In-memory fuzzing (in practice)
The fuzzer is composed of two scripts: the first, in-memory-break.py, is used to find functions to test.
The script inserts breakpoints at the beginning of every function of the program and prints out their arguments in search of text strings. The goal is to find the function that parses the input.
Let's try to run the script against the vulnerable program contained in the same directory:
crossbower@yotsubox$ ./in-memory-break.py getdomain test@email.com
Breakpoint 1 at 0x4004b0
Breakpoint 2 at 0x4004d8
Breakpoint 3 at 0x4004e8
Breakpoint 4 at 0x4004f8
Breakpoint 5 at 0x400508
Breakpoint 6 at 0x400518
Breakpoint 7 at 0x400528
Breakpoint 8 at 0x40056c
Breakpoint 9 at 0x400590
Breakpoint 10 at 0x400600
Breakpoint 11 at 0x40062c
Breakpoint 12 at 0x40064d
Breakpoint 13 at 0x4007f0
Function <__libc_start_main@plt> at *0x4004f8:
argument0 = 4196079 "UH"
argument1 = 2
argument2 = 140737488348088 "L"
argument3 = 4196192 "H"
argument4 = 4196176
argument5 = 140737351962048 "UH"
Function <_init> at *0x4004b0:
argument0 = 2
argument1 = 140737488348088 "L"
argument2 = 140737488348112
argument3 = 0
argument4 = 140737351885568
argument5 = 140737351962048 "UH"
Function <call_gmon_start> at *0x40056c:
argument0 = 2
argument1 = 140737488348088 "L"
argument2 = 140737488348112
argument3 = 0
argument4 = 140737351885568
argument5 = 140737351962048 "UH"
Function <frame_dummy> at *0x400600:
argument0 = 2
argument1 = 140737488348088 "L"
argument2 = 140737488348112
argument3 = 0
argument4 = 140737351885568
argument5 = 140737351962048 "UH"
Function <__do_global_ctors_aux> at *0x4007f0:
argument0 = 2
argument1 = 140737488348088 "L"
argument2 = 140737488348112
argument3 = 0
argument4 = 140737351885568
argument5 = 140737351962048 "UH"
Function <strdup@plt> at *0x400508:
argument0 = 140737488348816 "test@email.com"
argument1 = 140737488348088 "L"
argument2 = 140737488348112
argument3 = 0
argument4 = 140737351885568
argument5 = 140737351962048 "UH"
Function <parse> at *0x40064d:
argument0 = 6295568 "test@email.com"
argument1 = 140737488348831 "SSH_AGENT_PID=2952"
argument2 = 0
argument3 = 30803244232763745
argument4 = 140737351888448
argument5 = 140737348377640
Function <strtok@plt> at *0x400528:
argument0 = 6295568 "test@email.com"
argument1 = 4196426 "@"
argument2 = 0
argument3 = 30803244232763745
argument4 = 140737351888448
argument5 = 140737348377640
Function <strtok@plt> at *0x400528:
argument0 = 0
argument1 = 4196426 "@"
argument2 = 6295573 "email.com"
argument3 = 6295573 "email.com"
argument4 = 6295568 "test"
argument5 = 140737348377640
Function <strcpy@plt> at *0x400518:
argument0 = 140737488346768
argument1 = 6295573 "email.com"
argument2 = 6295582
argument3 = 6295583
argument4 = 0
argument5 = 140737348377640
Function <print_domain> at *0x40062c:
argument0 = 140737488346768 "email.com"
argument1 = 6295584
argument2 = 140737488346777
argument3 = 0
argument4 = -72340172838076673
argument5 = -72219847665292440
Function <printf@plt> at *0x4004d8:
argument0 = 4196412 "Domain is %s\n"
argument1 = 140737488346768 "email.com"
argument2 = 140737488346777
argument3 = 0
argument4 = -72340172838076673
argument5 = -72219847665292440
Domain is email.com
Function <printf@plt> at *0x4004d8:
argument0 = 4196463 "Domain is valid? %s\n"
argument1 = 4196428 "YES"
argument2 = 140737351888368
argument3 = 4196425
argument4 = 1
argument5 = 4196425
Domain is valid? YES
Function <__do_global_dtors_aux> at *0x400590:
argument0 = 140737488347632 "("
argument1 = 140737488347632 "("
argument2 = 140737354127792
argument3 = 4
argument4 = 0
argument5 = 4
[Inferior 1 (process 18083) exited normally]
I've highlighted the parsing function. This case was simple, because the binary was not stripped, making possible to print function names. But even in the case of stripped binaries, it's possible to find the function we are interested in, analyzing and trying to decode the arguments.
The vulnerable program simply try to extract the domain from an email address: an overly long domain triggers a buffer overflow on the stack.
Now that we know the function to test and the input type, let's try to see if our in-memory fuzzer is able to detect the bug:
cross@yotsubox$ ./in-memory-fuzz.py parse getdomain test@email.com Breakpoint 1 at 0x400651 Breakpoint 1, 0x0000000000400651 in parse () fuzz loop: 1 string len: 15 0x601030: "test@email.com" Domain is email.com 0x000000000040072f in main () Switching to process 4997 #0 0x0000000000400651 in parse () fuzz loop: 2 string len: 16 0x601030: "test@email.comA" Domain is email.comA 0x000000000040072f in main () Switching to process 4998 #0 0x0000000000400651 in parse () fuzz loop: 3 string len: 17 0x601030: "test@email.comAA" Domain is email.comAA 0x000000000040072f in main () Switching to process 4999 ... ... ... #0 0x0000000000400651 in parse () fuzz loop: 1031 string len: 1045 0x601030: "test@email.com", 'A' <repeats 186 times>... Domain is email.comAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 0x000000000040072f in main () Switching to process 7202 #0 0x0000000000400651 in parse () fuzz loop: 1032 string len: 1046 0x601030: "test@email.com", 'A' <repeats 186 times>... Domain is email.comAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Domain is valid? YES Program received signal SIGSEGV, Segmentation fault. 0x0000000000400849 in ?? () Switching to process 7203 #0 0x0000000000400651 in parse () fuzz loop: 1033 string len: 1047 0x601030: "test@email.com", 'A' <repeats 186 times>... [Switching to process 7204] Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists. # # The program has crashed! Stack exhaustion or bug??? # Now is your turn, have fun! :P # A debugging session is active. Inferior 1 [process 7204] will be killed. Quit anyway? (y or n) y
As you can see, fuzzing loop number 1032 triggered the bug: the input strings are simply generated appending 'A' characters to a valid email address, and when the domain part reaches a critical length, a buffer on the stack is overflowed, overwriting the return address (and stack canaries).
Although this is script is very simple, it's a basis to build on to create more complex fuzzer. In addition it illustrates several GDB features very useful in this field:
- Allocation of memory on the target process
- Snapshots/checkpoints and their restoration
- Breakpoint management
- Argument analysis
If you are interested in GDB python scripting or in-memory fuzzing on (u)nix system, you can visit the project website:
GDB Python Utils: https://github.com/crossbowerbt/GDB-Python-Utils/
You will find the scripts illustrated in this article under the examples directory of the project.
I'm also trying to maintain a good documentation for the support library, so you may give a look at the "snippet" page of the wiki (https://github.com/crossbowerbt/GDB-Python-Utils/wiki/Snippets/) to see the implemented features.
Post scripta
Some additions to the original article, born after discussing some issues with visitors in the comment section of the previous site.
Hints for 32bit systems
The fuzzer was specifically developed for 64bit systems: if you want to use it against 32bit application you must adapt it (if you send me an email I can give you a few hints...)
On 32-bit systems the arguments are passed on the stack: you can find them adding their offset to the %esp register.
Example, a 32bit process read() call:
(gdb) x/4x $sp 0xffffc09c: 0xf6bb17da 0x0000000b 0x09286d90 0x00002000
The first dword encountered is the return address, we are not interested in it...
The first argument (the file descriptor):
(gdb) x/x $sp+4 0xffffc0a0: 0x0000000b
The second argument (the buffer):
(gdb) x/x $sp+8 0xffffc0a4: 0x09286d90
The third argument (max read length):
(gdb) x/x $sp+12 0xffffc0a8: 0x00002000
To make the fuzzer work on 32-bit system you must change the code that read/modify the arguments of functions (I will probably change the fuzzer to make it multi-architecture, when I have time).
In the meanwhile the interface has changed...
Python integration in GDB has improved. Here is an interesting video that shows what is possible to do now: