GDB Tracepoints

for the Linux kernel

1
 Why can't I use GDB
to debug the Linux kernel?

2
Why can't I use GDB to debug the
kernel?

It is morally wrong
to use a debugger.
Use printk.

3
Why can't I use GDB to debug the
kernel?

Debuggers facilitate
observation.

4
Why can't I use GDB to debug the
kernel?

You need a second machine.

5
What are tracepoints?

6
What are tracepoints?
GDB-based source-level debugging

7
What are tracepoints?
GDB-based source-level debugging
Minimally intrusive

8
What are tracepoints?
GDB-based source-level debugging
Minimally intrusive
Can debug the kernel GDB itself is
running under

9
Breakpoints vs. Tracepoints
Breakpoints stop the program, while
you inspect its state.

10
Breakpoints vs. Tracepoints
Breakpoints stop the program, while
you inspect its state.
Tracepoints pause the program, log
information, and then continue.

11
Breakpoints vs. Tracepoints
Breakpoints stop the program, while
you inspect its state.
Tracepoints pause the program, log
information, and then continue.
In GDB, a selected log hit becomes
“the current state of the program”.

12
Breakpoints vs. Tracepoints
Breakpoints stop the program, while
you inspect its state.
Tracepoints pause the program, log
information, and then continue.
In GDB, a selected log hit becomes
“the current state of the program”.
You choose the information to log
ahead of time.
13
Demo #1

14
How does it work?

15
Tracepoint Implementation
GDB compiles source-language
expressions to bytecode

16
Tracepoint Bytecode
(gdb) maintenance agent file->f_dentry->d_iname
0 reg 0
3 zero_ext 32
5 const8 8
7 add
8 trace_quick 4
10 ref32
11 const8 108
13 add
14 trace_quick 36
16 pop
17 end
(gdb)

17
 Tracepoint probes

kprobes makes it easy to patch

18
 Tracepoint probes

kprobes makes it easy to patch

19
 Tracepoint probes

kprobes makes it easy to patch

20
Tracepoint Hit Log

21
 Tracepoint Hit Log
In kernel memory

22
 Tracepoint Hit Log
In kernel memory
Each entry records:

23
 Tracepoint Hit Log
In kernel memory
Each entry records:
Which tracepoint was hit

24
 Tracepoint Hit Log
In kernel memory
Each entry records:
Which tracepoint was hit
Register values

25
 Tracepoint Hit Log
In kernel memory
Each entry records:
Which tracepoint was hit
Register values
Contents of all memory touched by
tracepoint's bytecode expressions

26
 Tracepoint Hit Log
In kernel memory
Each entry records:
Which tracepoint was hit
Register values
Contents of all memory touched by
tracepoint's bytecode expressions
SMP-safe

27
 Bad /proc interface
Essentially passes GDB remote
protocol packets via write calls,
responses via read calls on /proc/gdb-
tracepoints

28
 Bad /proc interface
Essentially passes GDB remote
protocol packets via write calls,
responses via read calls on /proc/gdb-
tracepoints
Can be controlled by shell scripts
(Python!)

29
 Bad /proc interface
Essentially passes GDB remote
protocol packets via write calls,
responses via read calls on /proc/gdb-
tracepoints
Can be controlled by shell scripts
(Python!)
Ought to be sysfs/kobject-based

30
 Cute Hack #1

(Due to the inimitable

31
 Cute Hack #1
Log holds raw memory, not
expression results

32
 Cute Hack #1
Log holds raw memory, not
expression results
Selecting a hit makes those regs and
memory contents 'current' to GDB

33
 Cute Hack #1
Log holds raw memory, not
expression results
Selecting a hit makes those regs and
memory contents 'current' to GDB
So they can be reinterpreted in more
helpful ways

34
Demo #2

35
 Cute Hack #2

(Also due to the inimitable

36
 Cute Hack #2
struct gtp_hit
{
spinlock_t lock;
int number;
struct gtp_tracepoint *tracepoint;
size_t entries_used;
int error;
struct pt_regs regs;
size_t num_bytes;
unsigned char bytes[];
};

37
 Cute Hack #2

One tracepoint hit structure (with tail)

38
 Cute Hack #2

One tracepoint hit structure (with tail)

39
 Cute Hack #2
struct gtp_hit
{
spinlock_t lock;
int number;
struct gtp_tracepoint *tracepoint;
size_t entries_used;
int error;
struct pt_regs regs;
size_t num_bytes;
unsigned char bytes[];
};

40
 Cute Hack #2
When we log a hit, we log all the
bytes it refers to, traced or not, in the
order the interpreter requests them.

41
 Cute Hack #2
When we log a hit, we log all the
bytes it refers to, traced or not, in the
order the interpreter requests them.
When we query a hit, we re-evaluate
the expression, handing out the next
block of bytes as the interpreter
requests them.

42
 Cute Hack #2
When we log a hit, we log all the
bytes it refers to, traced or not, in the
order the interpreter requests them.
When we query a hit, we re-evaluate
the expression, handing out the next
block of bytes as the interpreter
requests them.
The two interpreters are in sync, so
they ask for the same blocks.
43
 Credits
Michael Snyder
Nicholas McGuire

44
 Thank you!

http://www.red-bean.com/jimb

45