Why Tracepoints needed:
It is not feasible for the debugger to interrupt the program's execution long enough for the developer to learn anything helpful about its behavior. If the program's correctness depends on its real-time behavior, delays introduced by a debugger might cause the program to change its behavior drastically, or perhaps fail, even when the code itself is correct. It is useful to be able to observe the program's behavior without interrupting it.
What is Trace Points:
A tracepoint placed in code provides a hook to call a function that you can provide at runtime.
A tracepoint can be "on" or "off".
When a tracepoint is "off" it has no effect, except for adding a tiny time penalty and space penalty .
When a tracepoint is "on", the function you provide is called each time the tracepoint is executed, in the execution context of the caller.
When the function provided ends its execution, it returns to the caller.
You can put tracepoints at important locations in the code.
Unlike Ftracer , Trace Point can record local variables of the function.
The tracepoint included a function call in the kernel code that, when enabled, would call a callback function passing the parameters of the tracepoint to that function as if the callback function was called with those parameters.
TRACE_EVENT() macro was specifically made to allow a developer to add tracepoints to their subsystem and have Ftrace automatically be able to trace them.
The anatomy of the TRACE_EVENT() macro:
It must create a tracepoint that can be placed in the kernel code.
It must create a callback function that can be hooked to this tracepoint.
The callback function must be able to record the data passed to it into the tracer ring buffer in the fastest way possible.
It must create a function that can parse the data recorded to the ring buffer and translate it to a human readable format that the tracer can display to a user.
Playing with trace events:
cd /sys/kernel/debug/tracing/events
root@amd-PADEMELON:/sys/kernel/debug/tracing/events/drm# ls
drm_vblank_event drm_vblank_event_delivered drm_vblank_event_queued enable filter
echo 1 > ./drm/enable
The enable files are used to enable a tracepoint. The enable file in the events directory can enable or disable all events in the system, the enable file in one of the system's directories can enable or disable all events within the system, and the enable file within the specific event directory can enable or disable that event.
Tracepoint logs can be seen with Ftrace logs :
3) + 20.320 us | dm_crtc_high_irq [amdgpu]();
0) | /* drm_vblank_event_queued: pid=2430, crtc=0, seq=297608 */
0) | send_vblank_event [drm]() {
0) | /* drm_vblank_event_delivered: pid=2430, crtc=0, seq=297608 */
0) 3.858 us | }
3) + 24.783 us | dm_crtc_high_irq [amdgpu]();
3) | dm_pflip_high_irq [amdgpu]() {
3) | drm_send_vblank_event [drm]() {
3) | send_vblank_event [drm]() {
3) | /* drm_vblank_event_delivered: pid=0, crtc=0, seq=297609 */
3) + 33.556 us | }
3) + 35.227 us | }
We can set requitred events using set_event . It is same as enaabing specific event using enable.
[tracing] # echo drm_vblank_event drm_vblank_event_delivered drm_vblank_event_queued > set_event
PERF TOOL:
One of the key secrets for quick use of tracepoints is the perf tool . CONFIG_EVENT_PROFILE configuration option should be set.
perf will be available at ./kernel/tools/perf
$ perf list
-> List of events available in the system
$ perf stat -a -e kmem:kmalloc sleep 10 -> How many kmalloc() calls are happening on a system
The -a option gives whole-system results
$ perf stat -e kmem:kmalloc make -> Monitoring allocations during the building of the perf tool
https://www.kernel.org/doc/Documentation/trace/tracepoints.txt - Kernel TracePoint
http://lwn.net/Articles/379903/
http://lwn.net/Articles/381064/
http://lwn.net/Articles/383362/
ftp://ftp.gnu.org/old-gnu/Manuals/gdb/html_chapter/gdb_10.html - GDB Tracepoints usage
https://lwn.net/Articles/346470/ - PERF TOOL
