Linux Kernel Debugging using KGDB:
Kernel can be debugged using kgdb via serial port , virtual machine and via network configuration.
We required 2 systems to debug kernel using kgdb. Lets see how to debug using serial port connection.
Target system : Developing kernel to be run on this machine.
Host system : Using gdb on host machine we can debug kernel running on target machine.
Make sure KGDB related config is enabled .
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
Add boot parameters as "kgdbwait kgdboc=ttyS4,115200" in target system. For more clarity on adding boot parameters refer "http://ayyappa-ch.blogspot.in/2015/07/serial-console-logging.html"
Need to boot the target machine with mentioned boot parameters.
When i try to debug , after this screen target machine is not responding. So the actual limitation is KGDB will not support usb based key board. So , out target will support only USB based keyboard. So , we modified boot parameters as "console=ttyS4,115200n8 kgdbwait kgdboc=ttyS4".
This configuration will help COM port as console. Host and Target are connected via serial port.
On host serial we could see same message and able to enter to kdb mode. So now target is in waiting state for Host to connect via gdb. After target enters into kdb mode we need to close serial port terminal as we are going to use same serial port to debug the kernel.
Start the host machine and make sure source , map , vmlinux and other obj files available in host machine. Go to the location where vmlinux exist.
$ gdb ./vmlinux
$ gdb > set debug serial 1
$ gdb > break emmc_init -> set break point at emmc_init
$ gdb> target remote /dev/ttyS4
This will make the connection between host and target mechine. After this control comes to Host mechine and Target waits on host.
$gdb> cont
This will continue the boot process in target and control comes to target . On host terminal will be waiitng.
On target machine enter $ echo "g" > /proc/sysrq-trigger.
This will give control to host machine to continue debugging.
Continue debugging as usual with gdb.
$gdb> bt - gives back trace of the kernel
$gdb> cont
This will continue until break point hits or if no break point hits , control goes to target machine.
Again in target mechine enter $ echo "g" > /proc/sysrq-trigger.
This will again give control to host machine . So we can debug again.
We can use either Linux machine or Windows machine as Host system. If we use windows system use MinGW gdb at windows side
Kernel module debugging:
On target machine get required module information:
$ modprobe your_module
$cd /sys/module/your_module/sections
$ cat .text
$ cat .bss
$ cat .data
$ cat .rodata
On host machine add module for debugging:
$ gdb> add-symbol-file module_name_with complete_path \
text_segment_address \
-s .bss bss_segment_address \
-s .rodata rodata_segment_address \
-s .data data_segment_address
Now we can set break points even in our modules.
User Space Process debugging:
we will see how to debug Xorg process.
$ ps -e | grep Xorg - get Xorg process pid
$ sudo gdb -p (pidof Xorg) - connect gdb to Xorg process
$gdb > bt -> will get stack trace of Xorg
Below link has some good info about gdb list and disassemble commands examples.
https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks
gdb debugging commands:
step - step into sub-routine
next - run over sub-routine in a go
finish - run till current function returns
return - make selected stack frame return to its caller
jump [address] - continue program at specified line or address
list - print 10 more lines
info break - show list and status of break points
break function - setting break point at function
break file:line - setting a break point at a file of line .
bt - display back trace
info args - shows args of current frame
info locals - shows locals of current frame
thread apply all bt - Back trace of all the threads
info threads - information about threads
Kernel can be debugged using kgdb via serial port , virtual machine and via network configuration.
We required 2 systems to debug kernel using kgdb. Lets see how to debug using serial port connection.
Target system : Developing kernel to be run on this machine.
Host system : Using gdb on host machine we can debug kernel running on target machine.
Make sure KGDB related config is enabled .
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
Add boot parameters as "kgdbwait kgdboc=ttyS4,115200" in target system. For more clarity on adding boot parameters refer "http://ayyappa-ch.blogspot.in/2015/07/serial-console-logging.html"
Need to boot the target machine with mentioned boot parameters.
When i try to debug , after this screen target machine is not responding. So the actual limitation is KGDB will not support usb based key board. So , out target will support only USB based keyboard. So , we modified boot parameters as "console=ttyS4,115200n8 kgdbwait kgdboc=ttyS4".
This configuration will help COM port as console. Host and Target are connected via serial port.
On host serial we could see same message and able to enter to kdb mode. So now target is in waiting state for Host to connect via gdb. After target enters into kdb mode we need to close serial port terminal as we are going to use same serial port to debug the kernel.
Start the host machine and make sure source , map , vmlinux and other obj files available in host machine. Go to the location where vmlinux exist.
$ gdb ./vmlinux
$ gdb > set debug serial 1
$ gdb > break emmc_init -> set break point at emmc_init
$ gdb> target remote /dev/ttyS4
This will make the connection between host and target mechine. After this control comes to Host mechine and Target waits on host.
$gdb> cont
This will continue the boot process in target and control comes to target . On host terminal will be waiitng.
On target machine enter $ echo "g" > /proc/sysrq-trigger.
This will give control to host machine to continue debugging.
Continue debugging as usual with gdb.
$gdb> bt - gives back trace of the kernel
$gdb> cont
This will continue until break point hits or if no break point hits , control goes to target machine.
Again in target mechine enter $ echo "g" > /proc/sysrq-trigger.
This will again give control to host machine . So we can debug again.
We can use either Linux machine or Windows machine as Host system. If we use windows system use MinGW gdb at windows side
Kernel module debugging:
On target machine get required module information:
$ modprobe your_module
$cd /sys/module/your_module/sections
$ cat .text
$ cat .bss
$ cat .data
$ cat .rodata
On host machine add module for debugging:
$ gdb> add-symbol-file module_name_with complete_path \
text_segment_address \
-s .bss bss_segment_address \
-s .rodata rodata_segment_address \
-s .data data_segment_address
Now we can set break points even in our modules.
User Space Process debugging:
we will see how to debug Xorg process.
$ ps -e | grep Xorg - get Xorg process pid
$ sudo gdb -p (pidof Xorg) - connect gdb to Xorg process
$gdb > bt -> will get stack trace of Xorg
Below link has some good info about gdb list and disassemble commands examples.
https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks
gdb debugging commands:
step - step into sub-routine
next - run over sub-routine in a go
finish - run till current function returns
return - make selected stack frame return to its caller
jump [address] - continue program at specified line or address
list - print 10 more lines
info break - show list and status of break points
break function - setting break point at function
break file:line - setting a break point at a file of line .
bt - display back trace
info args - shows args of current frame
info locals - shows locals of current frame
thread apply all bt - Back trace of all the threads
info threads - information about threads
