Debugging Ruby
Debugging Ruby
Aman Gupta
@tmm1
AS RUBYISTS, WE'VE SEEN...
nasty bugs
ejpphoto (flickr)
fatboyke (flickr)
code
memory bloat
37prime (flickr)
THIS TALK IS ABOUT...
lsof
strace
ltrace
pleeker (flickr)
TOOLS FOR C CODE.
perftools
gdb
booddin (flickr)
TOOLS FOR NETWORKS.
tcpdump
ngrep
pascal.charest (flickr)
TOOLS FOR CPU USAGE.
perftools
perftools.rb
marksze (flickr)
TOOLS FOR MEMORY USAGE.
bleak_house
gdb.rb
kgrocki (flickr) memprof
IGNORE THE FINE PRINT
-P
Inhibits the conversion of port numbers to names for network files
-p pid
Attach to the process with the process ID pid and begin tracing.
-tt
If given twice, the time printed will include the microseconds.
-T
Show the time spent in system calls.
-o filename
Write the trace output to the file filename rather than to stderr.
read query
response
slow query
stracing ruby: SIGVTALRM
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 2207807 <0.000009>
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 0 <0.000009>
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 140734552062624 <0.000009>
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 140734552066688 <0.000009>
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 11333952 <0.000008>
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 0 <0.000009>
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 1 <0.000010>
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
-s <len>
Snarf len bytes of data from each packet.
-n
Don't convert addresses (host addresses, port numbers) to names.
-q
Quiet output. Print less protocol information.
-A
Print each packet (minus its link level header) in ASCII.
-w <file>
Write the raw packets to file rather than printing them out.
<expr>
libpcap expression, for example:
tcp src port 80
tcp dst port 3306
tcp dst port 80
19:52:20.216294 IP 24.203.197.27.40105 >
174.37.48.236.80: tcp 438
E...*[email protected].%&.....%0....POx..%s.oP.......
GET /poll_images/cld99erh0/logo.png HTTP/1.1
Accept: */*
Referer: http://apps.facebook.com/realpolls/?
_fb_q=1
tcp dst port 3306
19:51:06.501632 IP 10.8.85.66.50443 >
10.8.85.68.3306: tcp 98
E..."K@[email protected]
.UB
.UD.....z....L............
GZ.y3b..[......W....
SELECT * FROM `votes` WHERE (`poll_id` =
72621) LIMIT 1
tcpdump -w <file>
PERFTOOLS
google's cpu profiler
CPUPROFILE=/tmp/myprof ./myapp
pprof ./myapp /tmp/myprof
wget http://google-perftools.googlecode.com/files/google-
perftools-1.6.tar.gz download
tar zxvf google-perftools-1.6.tar.gz
cd google-perftools-1.6
./configure --prefix=/opt
make compile
sudo make install
# for linux
export LD_PRELOAD=/opt/lib/libprofiler.so setup
# for osx
export DYLD_INSERT_LIBRARIES=/opt/lib/libprofiler.dylib
pprof.rb /tmp/myrbprof
github.com/tmm1/perftools.rb
gem install perftools.rb
github.com/ice799/ltrace/tree/libdl
ltrace -F <conf> -b -g -x <sym>
-b
Ignore signals.
-g
Ignore libraries linked at compile time.
-F <conf>
Read prototypes from config file.
-x <sym>
Trace calls to the function sym.
-s <num>
Show first num bytes of string args.
-F ltrace.conf
int mysql_real_query(addr,string,ulong);
void garbage_collect(void);
int memcached_set(addr,string,ulong,string,ulong);
ltrace -x garbage_collect
19:08:06.436926 garbage_collect() = <void> <0.221679>
19:08:15.329311 garbage_collect() = <void> <0.187546>
19:08:17.662149 garbage_collect() = <void> <0.199200>
19:08:20.486655 garbage_collect() = <void> <0.205864>
19:08:25.102302 garbage_collect() = <void> <0.214295>
19:08:35.552337 garbage_collect() = <void> <0.189172>
ltrace -x mysql_real_query
mysql_real_query(0x1c9e0500, "SET NAMES 'UTF8'", 16) = 0 <0.000324>
mysql_real_query(0x1c9e0500, "SET SQL_AUTO_IS_NULL=0", 22) = 0 <0.000322>
mysql_real_query(0x19c7a500, "SELECT * FROM `users`", 21) = 0 <1.206506>
mysql_real_query(0x1c9e0500, "COMMIT", 6) = 0 <0.000181>
ltrace -x memcached_set
memcached_set(0x15d46b80, "Status:33", 21, "\004\b", 366) = 0 <0.01116>
memcached_set(0x15d46b80, "Status:96", 21, "\004\b", 333) = 0 <0.00224>
memcached_set(0x15d46b80, "Status:57", 21, "\004\b", 298) = 0 <0.01850>
memcached_set(0x15d46b80, "Status:10", 21, "\004\b", 302) = 0 <0.00530>
memcached_set(0x15d46b80, "Status:67", 21, "\004\b", 318) = 0 <0.00291>
memcached_set(0x15d46b80, "Status:02", 21, "\004\b", 299) = 0 <0.00658>
memcached_set(0x15d46b80, "Status:34", 21, "\004\b", 264) = 0 <0.00243>
GDB
the GNU debugger
gdb <executable>
gdb attach <pid>
Debugging Ruby Segfaults
test_segv.rb:4: [BUG] Segmentation fault
ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9.7.0]
#include "ruby.h"
VALUE
segv()
{
VALUE array[1];
array[1000000] = NULL;
return Qnil;
}
void
Init_segv()
{
rb_define_method(rb_cObject, "segv", segv, 0);
}
Debugging Ruby Segfaults
test_segv.rb:4: [BUG] Segmentation fault
ruby 1.8.7 (2008-08-11 patchlevel 72) [i686-darwin9.7.0]
def test
#include "ruby.h" require 'segv'
4.times do
VALUE Dir.chdir '/tmp' do
segv() Hash.new{ segv }[0]
{ end
VALUE array[1]; end
array[1000000] = NULL; end
return Qnil;
} sleep 10
test()
void
Init_segv()
{
rb_define_method(rb_cObject, "segv", segv, 0);
}
1. Attach to running process
$ ps aux | grep ruby
joe 23611 0.0 0.1 25424 7540 S Dec01 0:00 ruby test_segv.rb
2. Use a coredump
Process.setrlimit Process::RLIMIT_CORE, 300*1024*1024
$ sudo mkdir /cores
$ sudo chmod 777 /cores
$ sudo sysctl kernel.core_pattern=/cores/%e.core.%s.%p.%t
def run
@listeners.each {|name,s|
s.run
}
ruby-bleak-house myapp.rb
bleak /tmp/bleak.<PID>.*.dump
github.com/fauna/bleak_house
• BleakHouse
• installs a patched version of ruby: ruby-bleak-house
• unlike gdb.rb, see where objects were created
(file:line)
• create multiple dumps over time with `kill -USR2
<pid>` and compare to find leaks
http://www.scribd.com/doc/31772032/memprof
plugging a leak in rails3
• in dev mode, rails3 is leaking 10mb per request
plugging a leak in rails3
• in dev mode, rails3 is leaking 10mb per request
# in environment.rb
require `gem which memprof/signal`.strip
plugging a leak
in rails3
send the app some
requests so it leaks
$ ab -c 1 -n 30
http://localhost:3000/
holding references
to all controllers
find references to object
holding references
to all controllers
• In development mode, Rails reloads all your
application code on every request
• ActionView::Partials::PartialRenderer is caching
partials used by each controller as an optimization
• But.. it ends up holding a reference to every single
reloaded version of those controllers
• In development mode, Rails reloads all your
application code on every request
• ActionView::Partials::PartialRenderer is caching
partials used by each controller as an optimization
• But.. it ends up holding a reference to every single
reloaded version of those controllers
RACK-PERFTOOLS
rack middleware for perftools.rb
$ curl http://localhost:3000/__start__
$ curl http://localhost:3000/home
$ curl http://localhost:3000/about
$ curl http://localhost:3000/__stop__