Debugging Kernel with GDB in Kernel Crash

FreeBSD kernel debugging with GDB

In today's post we will cover how to build a testing setup (to test kernel mode rootkit components for instance) in a FreeBSD environment. For this matter we'll build a client-server scheme using VMWare and it's serial port emulation capabilities as a bridge between the two machines. In this setup, one of the boxes will act as a debugger and the other one as the debuggee thanks to GDB's remote debugging feature. The exact versions covered in this article are the following:

• FreeBSD 8.2-RELEASE (amd64)
• VMware vSphere Web Access Version 4.1.0 (VMware ESX)

 
First of all we need to grab a fresh copy of FreeBSD. Once we got our copy ISO file, we create a single new Virtual Machine (VM) with default FreeBSD settings (8GB of disk space + default disk layout should be more than enough for our setup). During the install process we should take care of selecting kernel developer packages as well as the ports system. The rest of the install process is fairly straightforward if we keep this guide at hand.
With a fresh system just ready for action, the next step will consist in installing rsync, which we'll use to move compiled source + binaries to the debugger on a later step. To install rsync one of the easiest methods is to go straight into the ports system and building it from there:

# cd /usr/ports/net/rsync  
# make install

OR

#cd /packages
#pkg_add rsync-3.0.7.tbz (or any other version u have)

And after a few seconds of output we will be back in a shell prompt with a fully functioning rsync deployment. Next, we will avoid ourselves the burden of having to install a second VM by making use of VMWare's VM cloning capability. In this case we will perform a full copy so that we get two separate disk images rather than a single linked copy. From now on, one of the VMs will work as a debuggee (VM A) and the other one as debugger (VM B).
In the next phase, inside VM A we will recompile the kernel such that we enable full debugging capabilities and tracing. For this, we will make a copy of the default kernel compilation configuration file and then add a couple of flags. After completing the kernel compilation and installing the new boot image, we will copy the generated debugging kernel + symbols to VM B. Let's begin:

# cd /usr/src/sys/amd64/  
# cp conf/GENERIC conf/MYDEBUGKERNEL  

With our favorite text editor we edit MYDEBUGKERNEL and add (note that some of the options might already be enabled and needn't be re-defined):

DEBUG -g  
OPTIONS KDB  
OPTIONS DDB  
OPTIONS GDB

And now, it's compile time:

# cd /usr/src/  
# make buildkernel KERNCONF=MYDEBUGKERNEL  
# make installkernel KERNCONF=MYDEBUGKERNEL  

After compilation and installation succeed, we will copy the generated resources to VM B using rsync. Assuming that a folder is ready at /usr/debug in VM B (192.168.45.139 in this case), we pull the trigger:

# cd /usr/obj/usr/src/sys  
# rsync -avz MYDEBUGKERNEL root@192.168.45.139::/usr/debug  

(NOTE : scp,curl and any other methods will not give the correct synchronus copy so use rsync)

The transfer shouldn't take very long. After it's over, we shutdown both machines to let the VM magic begin. By magic, I mean that we will emulate a serial port connection by connecting both machines through a named pipe. Procedure is as follows:

If directly accessing the ESX server follow below steps

1. In VM A's tab, right click -> Settings -> Add
2. Select "Serial Port" from the list then "output to socket"
3. In the next window, choose a filename for the file that will act as a named pipe (e.g. /tmp/com_1)
4. Select "From server" to "Virtual Machine"
5. That's it. Now repeat the previous steps with VM B but instead of "From Server" choose "From Client"

If using VMware web Access follow below steps
(Applies to both VM's)

1. Click "Add Hardware" --> Select "Serial Port"-->> select "Named Pipe"
2. Then give the pipe name (Ex : debug pipe,,Note : Name for both pipes must be same)
3. For VM1 : Near End -> Is a server and Far End->Is a virtual machine
4. For VM2 : Near End -> Is a client and Far End->Is a virtual machine 
5. Click : Next -> Finish

Now that both machines are ready and connected, we can start descending into the catacombs. Fire up both machines. Go to VM A and edit the file /boot/device.hints and change the first serial connection's flag parameter, which in my particular case is "hint.uart.0.flags" to the value "0xC0". Reboot VM A.
Now shift to VM B and navigate to the directory where we copied the symbols using rsync. From there launch GDB connected to the serial port (serial port device name might not match yours):

# cd /usr/debug/MYDEBUGKERNEL  
# kgdb -r /dev/cuau0 kernel.debug  
GNU gdb 6.1.1 [FreeBSD]  
Copyright 2004 Free Software Foundation, Inc.  
GDB is free software, covered by the GNU General Public License, and you are  
welcome to change it and/or distribute copies of it under certain conditions.  
Type "show copying" to see the conditions.  
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...    
switching to remote protocol  
ignoring packet error, continuing...  
Couldn't establish connection to remote target  

(kgdb) 

Notice at this point how GDB enters a semi-idle state waiting for active connections from VM A. Now back to VM A, directly in the shell we issue:

# sysctl debug.kdb.enter=1  
debug.kdb.enter: 0KDB: enter: sysctl debug.kdb.enter  
[thread pid 1219 tid 100083]  
Stopped at  kdb_enter+0x3a: movl    $0,kdb_why  
db> gdb  
Step to enter the remote GDB backend.  
db> 

Notice how now in VM B the debugging prompt changed to an active state. Once we type 's' to step in VM A, the debugger in VM B will break and take control of VM A (with full system control, yay!).

It's possible to confirm that we're actually debugging VM A by issuing a simple command such as 'bt', which will show the current executing thread's call stack:

(kgdb) bt  
#0  kdb_enter (why=0xc0c311a6 "sysctl",  
    msg=0xc0cd9a83 sysctl debug.kdb.enter)  
    at /usr/src/sys/kern/subr_kdb.c:354  
#1  0xc08eeaa in kdb_sysctl_enter (oidp=0xc0dbed00, arg1=0x0, arg2=0,  
    req=0xe6c89b74) at /usr/src/sys/kern/subr_kdb.c:174  
    (...output trimmed...)

Happy digging!