Linux Performance Tuning

Wednesday, November 10, 2010

1
 Logistics

Tutorial runs from 9 to 5:00pm

Morning break at 10:30am

Lunch at 12:30--1:30 pm

Afternoon break at 3:00-3:30pm

Feel free to ask me questions

But I reserve the right to defer some answers until
later in the session or to the break/end of the class.

Please fill out and return the tutorial evaluation
form!
2
 Agenda

Introduction to Performance Tuning

Filesystem and storage tuning

Network tuning

NFS performance tuning

Memory tuning

Application tuning

3
 Introduction to Performance
Tuning

Complex task that requires in-depth
understanding of hardware, software, and
application

If it were easy the OS would do it automatically (and
the OS does a lot automatically to begin with)

Goals of Performance Tuning

Speed up time to do a single large task (time to do
perform some large matrix calculation)

Graceful degredation of a web/application server as
it is asked to service a larger and larger number of
requests 4
 Stress Testing

What happens when a server is put under a

large amount of stress?

“My web server just got slashdotted!”

Typically the server behaves well until the load
increases beyond a certain critical point; then it
breaks down.

Transaction latencies go through the roof

The server may cease functioning altogether

Measure the system when it is functioning
normally, and then when it is under stress. 5
What changes?
 Finding Bottlenecks

Careful tuning of memory usage won't matter if

the problem is caused by a shortage of disk
bandwidth

Performance measurement tools are hugely
important to diagnose what is placing limits on
the scalability or performance of your
application

Start with large areas, then narrow down

Is your application I/O bound? CPU bound?
Network bound?
6
 Incremental Tuning

Use the scientific method

Establish a baseline

Define testing parameters which are replicated from
test to test.

Measure the performance given a starting
configuration.

Change one parameter at a time

Record everything

Make sure you get the same results when you
repeat a test!
7
 Measurement overhead

Some performance measurement tools may

impact your application's behavior

If you're not familiar with how a particular tool
interacts with your workload, don't assume that a
tool has zero overhead with your application!

Enabling application performance metering or
debugging may also change its baseline
numbers.

8
 A basic performance tuning
methodology

Define your baseline configuration and
measure its performance

[ If appropriate, define a stress test workload, and
measure it. ]

Make a single change to the system
configuration. Measure the results of that
change and record it.

Repeat as necessary

Make sure to test single changes as well as
combination of changes. Sometimes effects are
synergistic 9
 Basic Performance Tools

free

top

iostat

10
 The free(1) command

Basic command which shows memory usage

11
 Questions to ask yourself after
looking at free(1) output

Will adding more memory help?

Often the cheapest way to speed up server

If the system is using paging or swapping,
adding more physical memory may help

Will a larger page cache help?

More sophisticated tools will answer these
questions later...

But asking questions is the beginning of wisdom

12
 The top(1) command

Good general place to start

13
 Questions to ask yourself when
looking at top(1) output

What are the “top” tasks running; should they
be there? Are they running? Waiting for disk?
How much memory are they taking up?

How is the CPU time (overall) being spent?

User time, System time, Niced user time, I/O Wait,
Hardware IRQ, Software IRQ, “Stolen” time

14
 The iostat(1) command

Part of the systat package; shows I/O statistics

Use -k to for kilobytes instead of 512 sectors

15
 Advanced iostat(1)

Many more details with the -x option

rrqm/s, wrqm/s – read/write requests merged per
second

r/s, w/s – read/write request per second

rkB/s, wkB/s – number of kilobytes of read/write
transfers per second

avgrq-sz --- average request size in 512 byte
sectors per second

avgqu-sa – average request queue length

…
16
 Advanced iostat(1), continued

Still more details revealed with the -x option

await – average time (in ms) between when a
requested is issued and when it is completed (time
in queue plus time for device to service the request)

svctm – average sevice time (in ms) for I/O
requests that were issued to the device

%util – Percentage of CPU time during which the
device was servicing requests. (100% means the
device is fully saturated)

17
 Example of iostat -xk 1

Workload “fs_mark -s 10240 -n 1000 -d /mnt”

Creates 1000 files, each 10k, in /mnt, with an fsync
after writing each file

Result: 33.7 files/second

18
 Conclusions we can draw from
the iostat results

Utilization: 98.48%

The system is I/O bound

Adding memory or speeding up the CPU clock
won't help

Solution – attack the I/O bottleneck

Add more I/O bandwidth resources (use a faster
disk or use a RAID array)

Or, do less work!

19
 Speeding up fs_mark

If we mount the (ext4) file system with -o

barrier=0, file/sec becomes 358.3

But this risks fs corruption after a power fail

Is the fsync() really needed? Without it, file/sec
goes up to 17,010.30

Depends on application requirements

Better: use -o journal_async_commit

Using journal checksums, it allows ext4 to safely
use only one barrier per fsync() instead of two.
(Requires Linux 2.6.32)
20
 Using -o journal_async_commit

Using ”fs_mark -s 10240 -n 1000 -d /mnt” again

Result: 48.2 files/sec (a 46% improvement over
33.7 files/sec!)

21
Comparing the two results

33.7
files/sec

2000
barrier
ops

49.2
files/sec

1000
barrier
ops

22
 Before we leave fs_mark...

How does fs_mark fare on other file systems?

ext2 (no barriers) – 574.9

ext3 (no barriers) – 348.8 (w/ barriers) – 30.8

ext4 (no barriers) – 358.3 (w/ barriers) – 49.2

XFS (no barriers) – 337.3 (w/ barriers) – 29.0

reiserfs (no barriers) – 210.0 (w/ barriers) – 31.5

Important note: these numbers are specific to
this workload (small files, fsync heavy) and not
a general figure of merit for these file systems
23
 Lessons Learned So Far

Measure, analyze, and then tweak

Bottleneck analysis is critical

It is very useful to understand how things work
under the covers

Adding more resources is one way to address a
bottleneck

But so is figuring ways of doing less work!

Sometimes you can achieve your goal by working
smarter, not harder.

24
 The snap script

Handy quickie shell script which I and a

colleague developed while working on the
Advanced Linux Response Team at IBM

Collects a lot of statistics: iostat, meminfo,
slabinfo, sar, etc. in a low impact fashion.

Collects system configuration information

Especially useful when I might not have access to
the system for security reasons

Gather information for a day; then analyze for
trends or patterns
25
 Agenda

Introduction to Performance Tuning

Filesystem and storage tuning

Network tuning

NFS performance tuning

Memory tuning

Application tuning

26
 File system and storage tuning

Choosing the right storage devices

Hard Drives

SSD

RAID

NFS appliances

File System Tuning

General Tips

File system specific

27
 Hard Drives

Disks are probably the biggest potential

bottleneck in your system

Punched cards and paper tape having fallen out of
favor...

Critical performance specs you should examine

Sustained Data Transfer Rate

Rotational speed: 5400rpm, 7200rpm, 10,000 rpm

Areal density (max capacity in that product family)

Seek time (actually 3 numbers: average, track-to-
track, full stroke)
28
 Transfer Rates

The important number is the sustained data

transfer rate (aka ”disk to buffer”) rate

Typically around 70-100 Mb/s; slower for laptop
drives

Much less important: The I/O transfer rate

At least for hard drives, whether you are using
SATA I's 1.5 Gb/s or SATA II's 3.0 Gb/s won't
matter except for rare cases when transfering data
out of the track buffer

SDD's might be a different story, of course...
29
 Short stroking hard drives

HDD performance are not uniform across the

platter

Up to 100% performance improvements on the
”outer edge” of the disk

Consider partitioning your disk to take this into
account!

If you don't need the full 1TB of space, partitioning
your disk to only use the first 100GB or 300GB
could speed things up!

Also – when running benchmarks, use the same
partitions for each file system tested. 30
 What about SSD's?

Advantages of SSD

Fast random access reads

Fails usually when writing, not when reading

Less suceptible to mechanical shock/vibration

Most SSD's use less power than HDD's

Disadvantage of SSD's

Cost per Gb much more expensive

Limited number of write cycles

Writes are slower than reads; random writes can be
much slower (up to a ½ sec average, 2 sec worst 31
case for 4k random writes for really bad SSD's!)
Getting the right SSD is important

Really good website for goes into great detail

about this is Anand Tech

http://www.anandtech.com/storage/

Many of the OEM SSD's included laptops are
not the good SSD's, and you the pay the OEM
markup to add insult to injury.

32
 Should you use SSD's?

For laptops and desktops, absolutely!

For servers, it depends...

If you need fast random access reads, yes!

If you care about power consumption, be careful

When idle, SSD's save only 0.2 to 0.4 Watts

When active, SSD's use roughly the same power as
5400rpm 2.5” drive and save 3W or so compared to
high performance 3.5” drives

For certain workloads, the write endurance problem
of SSD's may be a strong concern
33
 PCIe attached flash

Like SSD's, only more so

Speed achieved by writing to large numbers of
flash chips in parallel

Potentially 100k to 1M 4k random reads / seconds

Synchronous 4k random write just as slow as SSD's

Very expensive, but the price is starting to drop

In some cases, they can be cost effective

1 server with PCIe attached flash could replace
several servers with HDD's/SSD's in some cases
34
 RAID

Redundant Array of Inexpensive Disks

RAID 0 – Striping

RAID 1 – Mirroring

RAID 5 – 3 or more disks, with a rotating parity
stripe

RAID 6 – 4 more disks, with two rotating parity
stripes

RAID 10 – Mirroring + striping

35
 RAID tuning considerations

Adding more spindle improves performance

RAID 5/6 requires some special care

Writes smaller than the N*stripe size will require a
read/modify/write cycle in order to update the parity
stripe (where N is the number of non-spare disks)

If the RAID device is going to be broken up using
LVM or partitions, make sure the LV/parition is
aligned on a full stripe boundary

36
 Filesystem Tuning

Most general purpose file systems work quite

well for most workloads

But in some file systems are better for certain
specialized workloads

Reiserfs – small (< 4k) files

XFS – very big RAID arrays, very large files

Ext3 is a good general purpose filesystem that
many people use by default

Ext4 will be better at RAID, larger files, while still
working well on small-medium sized files
37
 Managing Access-time Updates

Posix requires that a file's last access time is

updated each time its contents are accessed.

This means a disk write for every single read

The mount options noatime and relatime can
reduce this overhead.

The relatime option will only update the atime if
mtime and ctime is newer than the last atime.

Only saves approximately half the writes compared
to noatime

Some applications do depend on atime being
updated 38
 Tuning ext3/ext4 journals

Sometimes increasing the journal size can help;

especially if your workload is very metadata-
intensive (lots of small files; lots of file
creates/deletes/renames)



Journal data modes

data=ordered (default) – data is written first before
metadata is committed

data=journal – data is written into the journal

data=writeback – only metadata is logged; after a
crash, uninitialized data can appear in newly
allocated data blocks 39
Using ionice to control read/write
priorities

Like the nice command but affects the priority
of read/write requests issued by the process

Three scheduling classes

Idle – only if there are no other high priority
requests pending

Best-effort – requests served round-robin (default)

Real time – highest priority request always gets
access

For best-effort and real time classes, there are
8 priorities, with 0 being the highest priority and
7 the lowest priority 40
 Agenda

Introduction to Performance Tuning

Filesystem and storage tuning

Network tuning

NFS performance tuning

Memory tuning

Application tuning

41
 Network Tuning

Before you do anything else... check the basic

health of the network

Speed, duplex, errors

Tools: ethtool, ifconfig, ping

Check TCP throughput: ttcp or nttcp

Look for ”wierd stuff” using wireshark / tcpdump

Network is a shared resource

Who else is using it?

What are bottlenecks in the network topology?
42
 Latency vs Throughput

Latency

When applications need maximum responsiveness


Lockstep protocols (i.e., no sliding window
optimizations)

RPC-based protocols

Throughput

When transfering large data sets

Very often tuning efforts will trade off latency for
throughput or vice versa

43
 Interrupt Coalescing

This reduces CPU load by amortizing the cost

of an interrupt over multiple packets; this allos
us to trade off latency for throughput


ethtool -C ethX rx-usecs 80 rx-frames 20”

This will delay a receive interrupt for 80 s or until 20
packets are received, whichever comes first


ethtool -C ethX rx-usecs 0 rx-frames 1”

This will cause an interrupt to be sent for every
packet received

Different NIC's will have different defaults and
may have additional tuning parameters
44
 Enable NIC optimizations

Some device drivers don't enable these

features by default

You can check using “ethtool -k eth0”

TCP segment offload

“ethtool -K tso on”

Checksum off-load

“ethtool -K tx on rx on”

Large Receive offload (for throughput)

“ethtool -K lro on”
45
 The bandwidth-delay product

Very important when optimizing for throughput,

especially for high speed, long distance links

Represents the amount of data that can be “in
flight” at any particular point in time.

BDP = 2 * bandwidth * delay

BDP = bandwidth * Round Trip Time (RTT)

example:

(100 Mbits/sec / 8 bits/byte) * 50 ms ping time =
625kbytes

46
 Why the BDP matters

TCP has to be able to retransmit any dropped

packets; so the kernel has to remember what
data has been sent in case it needs to
retransmit it.

TCP Window

Limits on the size of the TCP window to control
kernel memory consumed by the networking stack

47
 Using the BDP

The BDP in bytes plus some overhead room

should be used as [wmax] below when setting
these parameters in /etc/sysctl.conf:

net.core.rmem_max= [wmax]

Maximum Socket Receive Buffer size

net.core.wmem_max= [wmax]

Maximum Socket Send Buffer size

net.core.rmem_max also known as
/proc/sys/net/core/rmem_max

e.g., set via “echo 2097152 >
/proc/sys/net/core/rmem_max” 48
 Per-socket /etc/sysctl.conf
settings

net.ipv4.tcp_rmem = [wmin] [wstd] [wmax]

receive buffer sizing in bytes (per socket)

net.ipv4.tcp_wmem = [wmin] [wstd] [wmax]

memory reserved for send buffers in bytes (per
socket)

Modern kernels do automatic tuning of the
receive and send buffers; and the defaults are
better; still if your BDP is very high, you may
need to boost [wstd] and [wmax]. Keep [wmin]
small for out-of-memory situations.
49
 For large numbers of TCP
connections

net.ipv4.tcp_mem = [pmin] [pdef] [pmax]

pages allowed to be used by TCP (for all sockets)

For 32-bit x86 systems, kernel text & data
(including TCP buffers) can only be in the low
896MB.

So on 32-bit x86 systems, do not adjust these
numbers, since they are needed to balance
memory usage with other Lowmem users.

If this is a problem, best bet is to switch to a 64-bit
x86 system first.
50
 Increase transmit queue length

The ethernet default of 100 is good for most

networks and where we need to balance
interactive responsiveness with large transfers

However, for high speed networks and bulk
transfer, this needs to be increased to some
value between 1000-50000

“ifconfig eth0 txqueuelength 2000”

Tradeoffs: more kernel memory used;
interactive response may be impacted.

Experiment with ttcp to find the slowest value that
51
works for your network/application.
Optimizing for Low Latency TCP

This can be very painful, because TCP is not

really designed for low latency applications.

TCP is engineered to worry about congestion
control on wide-area networks, and to optimize for
throughput on large data streams.

If you are writing your own application from
scratch, very often basing your own protocol on
UDP is often a better bet.

Do you really need a byte-oriented service?

Do you only need automatic retransmission to deal
with lost packets? 52
 Nagle Algorithm

Goal: To make networking more efficient by

batching small writes into a bigger packet for
efficiency

When the OS gets a small amount of data (a single
keystroke in an telnet connection), delay a very
small amount of time to see if more bytes will be
coming.

This naturally increases latency!

Requires application-level change

int on = 1;

setsockopt (sockfd, SOL_TCP, TCP_NODELAY, 53

&amp;on, sizeof (on));

 Delayed Acknowledgements

On the receiver end, wait a small amount of

time before sending a bare acknowledgement
to see if there's more data coming (or if the
program will send a response upon which you
can piggy-back your response)

This can interact with TCP slow-start to cause
longer latencies when the send window is
initially small.

After congestion or after the TCP connection has
been idle, the send window (maxmimum bytes of
unack'ed data) must be set down the MSS value 54
Solving the Delayed Ack problem

Disable slow-start algorithm on the sender?

Slow-start is a MUST implement (RFC 2581)

Disable delayed acknowledgments on the
receiver?

Delayed acknowledgments is a SHOULD (RFC
2581)

Some OS's have a way of disabling delayed
acknowledgments; Linux does not

There is a hack that works on a per-packet basis,
though...
55
 Enabling QUICKACK

Linux tries to be “clever” and automatically
figure out when to disable delayed
acknowledgments when it believes the other
side is in slow start.

Hack to force “quickack” mode:

int on = 1;

setsockopt (sockfd, SOL_TCP, TCP_QUICKACK,
&amp;on, sizeof (on));
But QUICKACK mode is disabled once other side
is done with slow start. So you have to re-enable
it any time the connection is idle for longer than 56
the retransmission time.
 Agenda

Introduction to Performance Tuning

Filesystem and storage tuning

Network tuning

NFS performance tuning

Memory tuning

Application tuning

57
 NFS Performance tuning

Optimize both your network and your filesystem

In addition, various client and server specific
settings that we'll discuss now

General hint: use dedicated NFS servers

NFS file serving uses all parts of your system: CPU
time, memory, disk bandwidth, network bandwidth,
PCI bus bandwidth

Trying to run applications on your NFS servers will
make both NFS and the apps run slowly

58
 Tuning a NFS Server

If you only export file system mountpoints, use

the no_subtree_check option in /etc/exports

Can burn large amonuts of CPU for metadata
intensive workloads

Bump up the number of NFS threads to a large
number (it doesn't hurt that much to have too
many). Say, 128... (instead of 4 or 8 which is
way too little). How to do this is distro-specific:

/etc/sysconfig/nfs

/etc/defaults/nfs-kernel-server
59
 PCI Bus tuning

NFS serving puts heavy demands on both

networking cards and hard bus adapters

If you have a system with multiple PCI buses,
put the networking and storage cards on
different buses

Network cards tend to use lots of small DMA
transfers, which tends to hog the bus

60
 NFS client tuning

Make sure you use NFSv3 and not NFSv2

Make sure you use TCP and not UDP

Use the largest rsize/wsize that the client/server
kernels support

Modern client/servers can do a megabyte at a time

Use the hard mount option, and not soft

Use intr so you can recover an NFS server is down

All of these are the default except for intr

Remove outdated fstab mount options. Just use
61
“rw,intr”
 Tuning your network config for
NFS

Tune the network for bulk transfers (throughput)

Use the largest MTU size you can

For ethernets, consider using jumbo frames if all of
the intervening switches/routers support it

62
 Agenda

Introduction to Performance Tuning

Filesystem and storage tuning

Network tuning

NFS performance tuning

Memory tuning

Application tuning

63
 Memory Tuning

Memory tuning problems can often look like

other problems

Unneeded I/O caused by excessive paging/swaping

Extra CPU time caused by cache/TLB thrashing

Extra CPU time caused by NUMA-induced memory
access latencies

These subtleties require using more
sophisticated performance measurement tools

64
 To measure swapping activity

The top(1) and free(1) command will both tell

you if any swap space is in use

To a first approximation, if there is any swap in use,
the system can be made faster by adding more
RAM.

To see current swap activity, use the sar(8)
program

First use of a very handy (and rather complicated)
system activity recorder program; reading through
the man page strongly recommended

Part of the systat package 65
 Using sar to obtain swapping
information

Use “sar -W <interval> [<num. of samples>]”

Reports number of pages written (swapped out)
and read (swapped in) from the page device
out per second.

The first output is the average since system was
started.

66
 Optimizing swapping

Use multiple swap devices

Use fast swap devices

Fast devices can be given a higher priority

Add more memory to avoid swapping in the first
place

67
 Swapping vs. Paging

Swap used for anonymous pages

i.e., pages which are not backed by a file

Pages which are backed by a file are subject to
paging

If they have been modified, or made dirty, they are
”cleaned” by being written to their backing store

If a page has not been be used recently, it is
”deactivated” by removing it from processes' page
table

Clean and inactive pages may be purposed for
other uses on an LRU basis 68
 Optimizing Paging

Unlike swapping, some amount of paging is

normal – and unavoidable

So we can't just manage the amount of paging to
zero, like we can with swapping

Goal: to minimize amount of paging in the steady-
state case

Key statistics:

majflts/s – major faults (which result in I/O) / second

pgsteal/s – pages reclaimed from the page and
swap cache / second to satisfy memory demands
69
 Using sar to obtain information
about paging

Use “sar -B <interval> [<num. of samples>]”

Reports many statistics

pgpgin/s, pgpgout/s – ignore, not useful/misleading

fault/s – # of page faults / sec.

majfault/s – # of page faults that result in I/O / sec.

pgfree/s – # of pages placed on the free list / sec.

pgscank/s – # of pages scanned by kswaped / sec.

pgscand/s – # of pages scanned directly / sec.

pgsteal/s – # of pages reclaimed from scache / sec.

%vmeff – pgsteal/s / (pgscank/s + pgscand/s) 70
Other ways of finding information
about memory utilization

cat /proc/meminfo

Something especially important on 32-bit x86
kernels: Low Memory vs. High Memory

Documentation/filesystems/proc.txt

cat /proc/slabinfo

Useful for seeing how the kernel is using memory

ALT-sysrq-m (or 'echo m > /proc/sysrq-trigger')

Different for different kernel versions and
distributions; /proc/slabinfo may not exist if
CONFIG_SLUB used and not CONFIG_SLAB 71
/proc/meminfo

72
 Interesting bits from sysrq-m

Per-zone statistics

73
 About Memory Caches

2GHz processor  2 billion cycles per second

Memory is much slower

Solution: use small amounts of fast cache
memory

Typically 32Kb very fast Level 1 cache

Maybe 4-8MB of somewhat slower Level 2 cache

Can see how much cache you have using
dmidecode and x86info

Not much tuning that can be done except by
improving the C/C++ program code 74
 TLB Caches

The Translation Lookaside Cache speeds up

translation from a virtual address to a physical
address

Normally requires 2-3 lookups in the page tables

TLB cache short circuits this lookup process

The x86info program will show the TLB cache
layout

Hugepages are a way to avoid consuming too
many TLB cache entries
75
 Using hugepages

Build a kernel that avoids using modules

The core kernel text segment uses huge pages;
modules do not

Modify an application to use hugepages (or
configure an application to use it if it already
has provision to use hugepages).

“mount -t hugetlbfs none /hugepages” then mmap
pages in /hugepages

On new qemu/kvm, you can use the option

“-mem-path /hugepages”

Use shmget(2) with the flag SHM_HUGETLB 76
 Configuring hugepages

On most enterprise distro's this must be done at

boot time or shortly after it

Kernel boot option “hugepages=nnn”

/etc/sysctl.conf: “vm.nr_hugepages=nnn”

These pages are reserved for hugepages and can
not be used for anything else

With kernels newer than 2.6.23, things are
more flexible

Kernel boot option “movablecore=nnn[KMG]”

Memory reserved this way can be used for
77
hugepages and other uses
 Agenda

Introduction to Performance Tuning

Filesystem and storage tuning

Network tuning

NFS performance tuning

Memory tuning

Application tuning

78
 Application Tuning

Access to the source code?

Open source vs. Proprietary

Ability/willingness to modify the code?

Even if it's open source, you might not want to
modify the code

Proprietary programs

Read the documentation; find the knobs and find
the application-level statistics you can gather

… but there are still some tricks we can do to figure
out what is going on when you don't have the
79
source...
A quick aside: Java Performance
Tuning

I'm not a Java programmer.... but I've worked
with a lot of Java performance tuning experts

First thing to consider is Garbage Collection

The GC is overhead that burns CPU time

GC can cause unpredictable pauses in the program

Collecting GC stats: JVM command-line option
-verbose:gc

Sizing the heap

Larger heap means less GC's

… but more time spent GC'ing when you do 80
 Generational GC

Observation: objects in Java have a high infant

mortality rate

Temporary objects, etc.

So put them in a separate arenas.

An object starts in the nursery (aka eden) space.
The nursery is GC'ed more frequently.

Objects which survive a certain number of GC
passes get promoted from the nursery to a tenured
space (which is GC'ed less frequently)

Need to configure the size of the nursury and
tenured space 81
Reducing GC's by not creating as
much Garbage

Requires being able to modify the code

Very often, though, Java programmers can
make extra work for the Java Run-time
Environment without realizing it

Two common examples

Using String and Integer class variables to do
calculations (instead of StringBuffer and the
primitive int type)

Using Java.util.Map instead of creating a Class

82
 Back to C/C++ applications

Tools for investigating applications

strace/ltrace

valgrind

gprof

oprofile

perf

Most of these tools work better if you have
source access

But sometimes not source is not absolutely required
83
 strace and ltrace

Useful for seeing what the application is doing

Especially useful when you don't have source

System call tracing: strace

Shared library tracing: ltrace

Run a new command with tracing:

strace /bin/ls /usr

Attach to an already existing process

ltrace -p 12345
84
 Valgrind

Used for finding memory leaks and other

memory access bugs

Best used with source access (compiled with -g);
but not strictly necessary

Works by emulating x86 in x86 and adding
checks to pointer references and malloc/free
calls

Other architectures supported

Commercial alternative: purify (uses object
code insertion)
85
 C/C++ profiling using gprof

To use, compile your code using the -pg option

This will add code to the compiled binary to
track each function call and its caller

In addition the program counter is sampled by
the kernel at some regular interval (i.e., 100Hz
or 1kHz) to find the “hot spots”

Demo time!

86
 System profiling using oprofile

Basic operation very similar to gprof

Sample the program counter at regular intervals

Advantages over gprof

Does not require recompiling application with -pg

Can profile multiple processes and the kernel all at
the same time

Demo time!

87
 Perf: the next generation

Originally intended to be a way to access

performance counters

Added the ability to sample kernel tracepoints

Sampling can be restricted to a process, a
process and its children, or the whole system

With perf record/report/perf annotate
performance events can be tied to specific
C/C++ lines of code (with source files and
object files compiled with -g)

Demo time! 88
 Userspace Locking

One other application issues which can be a

very big deal: userspace locking

Rip out fancy multi-level locking (i.e., user-
space spinlocks, sched_yield() calls, etc.)

Just use pthread mutexes, and be happy

Linux implements pthread mutexes using the
futex(2) system call. Avoids kernel context switch
except in the contended case

The fast path really is fast! (So need for
fancy/complex multi-level locking – just rip it out)
89
 Processor Affinity

Rarely a good idea... but can be used to

improve response time for critical tasks

Set CPU affinity for tasks using taskset(1)

Set CPU affinity for interrupt handlers using
/proc/irq/<nn>/smp_affinity

Strategies

Put producer/consumer processes on the same
CPU

Move interrupt handlers to a different CPU

Use mpstat(1) and /proc/interrupts to get 90

processor-related statistics
 Conclusion

Performance tuning is fractal

There's always more to tweak

“It's more addictive than pistachios!”

Understanding when to stop

Great way of learning more up and down the
technology stack – from the CPU chip up
through to the OS to the application tuning

91