The beauty of FTRACE

The kernel configuration options  need to be enabled for Ftrace:
CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE

Ftrace Sys path :
[~]# cd /sys/kernel/debug/tracing
[tracing]#

Stack Tracing:
The stack tracer checks the size of the stack at every function call. If it is greater than the last recorded maximum, it records the stack trace and updates the maximum with the new size. To see the current maximum, look at the stack_max_size file.

[tracing]# echo 1 > /proc/sys/kernel/stack_tracer_enabled
[tracing]# cat stack_max_size
2928
[tracing]# cat stack_trace


List of available tracers:
[tracing]# cat available_tracers 
function_graph function sched_switch nop

Setting current_tracer:
[tracing]# echo function > current_tracer
[tracing]# cat current_tracer
function

Setting trace buffer size:
[tracing]# echo 50 > buffer_size_kb

Adding module for ftrace filter:
[tracing]# echo ':mod:amdgpu' > set_ftrace_filter

This will ignore existing modules if any added and add all the functions available in amdgpu module for tracing.

Adding module for ftrace filter:
[tracing]# echo ':mod:ttm' >> set_ftrace_filter

note the '>>' is used. It will add ttm module to the existing list of modules. 

Adding set of functions start with specific name for Tracing:
[tracing]# echo 'sched*' > set_ftrace_filter
[tracing]# echo 'schedule:traceoff' >> set_ftrace_filter

All function names  start with sched are added for Tracing.

Adding specific pid for Tracing:
[tracing]# echo $$ > set_ftrace_pid

View Function graph for particular function:
[tracing]# echo kfree > set_graph_function
[tracing]# echo function_graph > current_tracer
[tracing]# cat trace

It will display the function flow for kfree.

   
Removing unwanted function contain specific name:
[tracing]# echo '!*lock*' >> set_ftrace_filter


The '!' symbol will remove functions listed in the filter file. As shown above, the '!' works with wildcards, but could also be used with a single function. Since '!' has special meaning in bash it must be wrapped with single quotes or bash will try to execute what follows it. Also note the '>>' is used. If you make the mistake of using '>' you will end up with no functions in the filter file.


References:
http://lwn.net/Articles/365835/  - ftrace part1
https://lwn.net/Articles/366796/  - ftrace part2
https://lwn.net/Articles/370423/  - ftrace secrets

Spin-lock usage with respect to Process, Bottom Half and Top Half Context

For kernels compiled without CONFIG_SMP, and without CONFIG_PREEMPT spinlocks do not exist at all. when no-one else can run at the same time, there is no reason to have a lock.

If the kernel is compiled without CONFIG_SMP, but CONFIG_PREEMPT is set, then spinlocks simply disable preemption, which is sufficient to prevent any races.

Linux guarantees the same interrupt will not be re-entered.


spin_lock(lock):
=>  Acquire the spin lock

=> Under certain circumstances, it is not necessary to disable local interrupts. For example, most filesystems only access their data structures from process context and acquire their spinlocks by calling spin lock(lock).

=> If another tasklet/timer wants to share data with your tasklet or timer , you will both need to use spin_lock() and spin_unlock() calls. spin_lock_bh() is unnecessary here, as you are already in a tasklet, and none will be run on the same CPU.


spin_lock_irq(lock)  :
=> Disable interrupts on local CPU
=> acquire the spin lock

=> If the code in process context is holding a spinlock and the code in interrupt context attempts to acquire the same spinlock, it will spin forever. For this reason, it is recommended that spin_lock_irq() is always used.

=> Data sharing between interrupt context and softirq or tasklet or process context needs to protect with spin_lock_irq().
 

spin_lock_irqsave(lock , flags) :
=> saves current interrupt state into flags
=> Disable interrupts on local CPU
=> acquire the spin lock

=> Sharing data bwtween two Hard IRQ Handlers ( interrupt contextes) use this locking technique

=> same code can be used inside an hard irq handler (where interrupts are already off) and in softirqs (where the irq disabling is required).


spin_lock_bh(lock):
=> Disbale softirq on current CPU
=> acquire the spin lock

=> If a data structure is accessed only from process and bottom half context, spin lock bh() can be used instead. This optimisation allows interrupts to come in while the spinlock is held, but doesn’t allow bottom halves to run on exit from the interrupt routine; they will be deferred until the spin unlock bh().

=> If another tasklet/timer wants to share data with your tasklet or timer , you will both need to use spin_lock() and spin_unlock()  calls. spin_lock_bh() is unnecessary here, as you are already in a tasklet, and none will be run on the same CPU.
 
 
Locking between same softirq sharing data :
The same softirq can run on the other CPUs: you can use a per-CPU array for better performance. If you're going so far as to use a softirq, you probably care about scalable performance enough to justify the extra complexity.You'll need to use spin_lock() and spin_unlock() for shared data.


Locking Between Hard IRQ and Softirqs/Tasklets:
If a hardware irq handler shares data with a softirq, you have two concerns. Firstly, the softirq processing can be interrupted by a hardware interrupt, and secondly, the critical region could be entered by a hardware interrupt on another CPU. This is where spin_lock_irq() is used. It is defined to disable interrupts on that cpu, then grab the lock. spin_unlock_irq() does the reverse.

The irq handler does not to use spin_lock_irq(), because the softirq cannot run while the irq handler is running: it can use spin_lock(), which is slightly faster. The only exception would be if a different hardware irq handler uses the same lock: spin_lock_irq() will stop that from interrupting us.

Deadlock Vs Livelock Vs Starvation

Deadlock: A situation in which two or more processes are unable to proceed because each is waiting for one the others to do something.

For example, consider two processes, P1 and P2, and two resources, R1 and R2. Suppose that each process needs access to both resources to perform part of its function. Then it is possible to have the following situation: the OS assigns R1 to P2, and R2 to P1. Each process is waiting for one of the two resources. Neither will release the resource that it already owns until it has acquired the other resource and performed the function requiring both resources. The two processes are deadlocked

Livelock: A situation in which two or more processes continuously change their states in response to changes in the other process(es) without doing any useful work:

For example , consider two processes each waiting for a resource the other has but waiting in a non-blocking manner. When each learns they cannot continue they release their held resource and sleep for some times, then they retrieve their original resource followed by trying to the resource the other process held, then left, then reacquired. Since both processes are trying to cope (just badly), this is a livelock.

Starvation: A situation in which a runnable process is overlooked indefinitely by the scheduler; although it is able to proceed, it is never chosen.

For example , consider three processes (P1, P2, P3) each require periodic access to resource R. Consider the situation in which P1 is in possession of the resource, and both P2 and P3 are delayed, waiting for that resource. When P1 exits its critical section, either P2 or P3 should be allowed access to R. Assume that the OS grants access to P3 and that P1 again requires access before P3 completes its critical section. If the OS grants access to P1 after P3 has finished, and subsequently alternately grants access to P1 and P3, then P2 may indefinitely be denied access to the resource, even though there is no deadlock situation.

Linux Kernel Debugging using KGDB and User space process debugging using gdb

Linux Kernel Debugging using KGDB:

Kernel can be debugged using kgdb via serial port , virtual machine and via network configuration.

We required 2 systems to debug kernel using kgdb. Lets see how to debug using serial port connection.

Target system : Developing kernel to be run on this machine.

Host system :  Using gdb on host machine we can debug kernel running on target machine.

Make sure KGDB related config is enabled .
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y

Add boot parameters as "kgdbwait kgdboc=ttyS4,115200" in target system. For more clarity on adding boot parameters refer "http://ayyappa-ch.blogspot.in/2015/07/serial-console-logging.html"

Need to boot the target machine with mentioned boot parameters.

When i try to debug , after this screen target machine is not responding. So the actual limitation is KGDB will not support usb based key board. So , out target will support only USB based keyboard. So , we modified boot parameters as "console=ttyS4,115200n8 kgdbwait kgdboc=ttyS4".

This configuration will help COM port as console. Host and Target are connected via serial port.
On host serial we could see same message and able to enter to kdb mode. So now target is in waiting state for Host to connect via gdb. After target enters into kdb mode we need to close serial port terminal as we are going to use same serial port to debug the kernel.

Start the host machine and make sure source , map , vmlinux and other obj files available in host machine. Go to the location where vmlinux exist.

$ gdb ./vmlinux

$ gdb > set debug serial 1

$ gdb > break emmc_init -> set break point at emmc_init

$ gdb> target remote /dev/ttyS4
This will make the connection between host and target mechine. After this control comes to Host mechine and Target waits on host.

$gdb> cont
This will continue the boot process in target and control comes to target . On host terminal will be waiitng.

On target machine enter  $  echo "g"  >  /proc/sysrq-trigger.
This will give control to host machine to continue debugging.

Continue debugging as usual with gdb.
$gdb> bt - gives back trace of the kernel
$gdb> cont
This will continue until break point hits or if no break point hits , control goes to target machine.

Again in target mechine enter $  echo "g"  >  /proc/sysrq-trigger.
This will again give control to host machine . So we can debug again.

We can use either Linux machine or Windows machine as Host system. If we use windows system use MinGW gdb at windows side

Kernel module debugging:
On target machine get required module information:
 $ modprobe  your_module
$cd /sys/module/your_module/sections
$ cat .text
$ cat .bss
$ cat .data
$ cat .rodata

On host machine add module for debugging:
 $ gdb> add-symbol-file  module_name_with complete_path  \
 text_segment_address \
-s .bss    bss_segment_address \
-s .rodata   rodata_segment_address \
-s .data    data_segment_address

Now we can set break points even in  our modules.

User Space Process debugging:
we will see how to debug Xorg process.

$ ps -e | grep Xorg - get Xorg process pid

$ sudo gdb -p (pidof Xorg) - connect gdb to Xorg process

$gdb > bt -> will get stack trace of Xorg

Below link has some good info about gdb list and disassemble commands examples.

https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks

gdb debugging commands:

step - step into sub-routine

next - run over sub-routine in a go

finish - run till current function returns

return - make selected stack frame return to its caller

jump [address] - continue program at specified line or address

list  - print 10 more lines

info break - show list and status of break points

break function - setting break point at function

break file:line - setting a break point at a file of line .

bt - display back trace

info args - shows args of current frame

info locals - shows locals of current frame

thread apply all bt  - Back trace of all the threads

info threads - information about threads

Serial Console logging

To debug boot-up issues we need to configure Serial port as console in Kernel . We can easily collect dmesg information with this.

Steps to enable Serial logging :

1) Make sure serial driver statically built with kernel image . If it built as a module , we can not get bootup logs as it will not loaded at that time. If it is built as modules , make sure module name is added to /etc/modules.conf. So that module will be loaded .

2) Need to add kernel boot parameters for enabling serial port as console

Edit /etc/default/grub file

Replace  GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"  with

GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS4,115200n8 debug ignore_loglevel"

close the file and execute "sudo update-grub".

Make sure target system UART is connected to other system and configure UART.

Reboot the system and observe the logs will be displayed in both test system and Other system.

console=tty0  =>  the kernel to continue using the standard console device. Boot messges will be dispayed on target system monitor.

console=ttyS4 => the kernel to also use the first serial device as a console. ttySn , n will be the serial port device available.  grep dmesg with "ttyS" to find available ports in your system.

115200n8 sets the serial speed to 115200 bits per second with no parity and 8 data bits per character. The other end of the serial line also must be configured to same settings.

debug ignore_loglevel allows you to see ALL kernel messages on the configured console.

drm.debug=14 => if we need to enable drm specifc logs other than all logs.

log_buf_len=16M => set the dmesg buffer size to get all required logs.

extern Vs EXPORT_SYMBOL ?

EXPORT_SYMBOL macro creates a new kernel symbol entry and puts it in a special section of the kernel image, in __ksymtab section. When modules are loaded dynamically, he loader resolves the symbols during run time by parsing the entries of this section.

For exporting a symbol , Function prototypes don't need to be written with extern. extern is assumed by the compiler.  Its use is as unnecessary as using auto to declare automatic/local variables in a block.
 
Extern can be used for non-static functions that are statically linked during compile time.

Why Serial Data Transmission Is Faster Than Parallel Data Transmission?

how serial data transmission can be faster than parallel data transmission ? But lets see what it means..

To be with basics , At a given frequency Parallel data transmission is better than Serial data transmission. With parallel transmission , for one clock cycle we can transfer 1 byte but only one bit with serial data transmission.

But we can not use parallel data transmission at high frequency. High frequency will cause data lost , data corruption and synchronization issue . So it may need to transmit the  data and finally data rate will be limited.

If we use high frequency for serial data transmission , we can achieve higher data rate because data will be transmitted at only one line . so there is no synchronization and stability issues.

This is the reason SATA AHCI interface makes faster than PATA.  In latest technology we use only SATA AHCI.

SATA can run at lower power compared to PATA due to no of pins are less.


So , latest serial data transmission  can be called it as High Speed Serial Data Transmission .

One more example is PCIe is faster than PCI.

PCIe is a high speed serial computer expansion bus.