The beauty of FTRACE

The kernel configuration options  need to be enabled for Ftrace:
CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE

Ftrace Sys path :
[~]# cd /sys/kernel/debug/tracing
[tracing]#

Stack Tracing:
The stack tracer checks the size of the stack at every function call. If it is greater than the last recorded maximum, it records the stack trace and updates the maximum with the new size. To see the current maximum, look at the stack_max_size file.

[tracing]# echo 1 > /proc/sys/kernel/stack_tracer_enabled
[tracing]# cat stack_max_size
2928
[tracing]# cat stack_trace


List of available tracers:
[tracing]# cat available_tracers 
function_graph function sched_switch nop

Setting current_tracer:
[tracing]# echo function > current_tracer
[tracing]# cat current_tracer
function

Setting trace buffer size:
[tracing]# echo 50 > buffer_size_kb

Adding module for ftrace filter:
[tracing]# echo ':mod:amdgpu' > set_ftrace_filter

This will ignore existing modules if any added and add all the functions available in amdgpu module for tracing.

Adding module for ftrace filter:
[tracing]# echo ':mod:ttm' >> set_ftrace_filter

note the '>>' is used. It will add ttm module to the existing list of modules. 

Adding set of functions start with specific name for Tracing:
[tracing]# echo 'sched*' > set_ftrace_filter
[tracing]# echo 'schedule:traceoff' >> set_ftrace_filter

All function names  start with sched are added for Tracing.

Adding specific pid for Tracing:
[tracing]# echo $$ > set_ftrace_pid

View Function graph for particular function:
[tracing]# echo kfree > set_graph_function
[tracing]# echo function_graph > current_tracer
[tracing]# cat trace

It will display the function flow for kfree.

   
Removing unwanted function contain specific name:
[tracing]# echo '!*lock*' >> set_ftrace_filter


The '!' symbol will remove functions listed in the filter file. As shown above, the '!' works with wildcards, but could also be used with a single function. Since '!' has special meaning in bash it must be wrapped with single quotes or bash will try to execute what follows it. Also note the '>>' is used. If you make the mistake of using '>' you will end up with no functions in the filter file.


References:
http://lwn.net/Articles/365835/  - ftrace part1
https://lwn.net/Articles/366796/  - ftrace part2
https://lwn.net/Articles/370423/  - ftrace secrets