HPC debugging

Victor Eijkhout

2022

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

Profiling and debugging;

optimization and
programming strategies.

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

1 Analysis basics

• Measurements: repeated and controlled

beware of transients, do you know where your data is?
• Document everything
• Script everything

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

2 Compiler options

• Defaults are a starting point

• use reporting options: -opt-report, -vec-report
useful to check if optimization happened / could not happen
• test numerical correctness before/after optimization change
(there are options for numerical corretness)

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

3 Optimization basics

• Use libraries when possible: don’t reinvent the wheel

• Premature optimization is the root of all evil (Knuth)

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

4 Code design for performance

• Keep inner loops simple: no conditionals, function calls, casts

• Avoid small functions: try macros or inlining
• Keep in mind all the cache,TLB, SIMD stuff from before
• SIMD: Fortran array syntax helps

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

5 Multicore / multithread

• Use numactl: prevent process migration

• ‘first touch’ policy: allocate data where it will be used
• Scaling behaviour mostly influenced by bandwidth

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

6 Multinode performance

• Influenced by load balancing

• Use HPCtoolkit, Scalasca, TAU for plotting
• Explore ‘eager’ limit (mvapich2: environment variables)

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

7 Classes of programming errors

Logic errors:
functions behave differently from how you thought,
or interact in ways you didn’t envision

Hard to debug

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

8 More classes of errors

Coding errors:
send without receive
forget to allocate buffer

Debuggers can help

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

Defensive programming

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

9 Defensive programming

• Keep It Simple (‘restrict expressivity’)

• Example: use collective instead of spelling it out
• easier to write / harder to get wrong
the library and runtime are likely to be better at optimizing than
you

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

10 Memory management

Beware of memory leaks:

keep allocation and free in same lexical scope

C++ does this automatically with RAII

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

11 Modular design

Design for debuggability, also easier to optimize

Separation of concerns: try to keep code aspects separate

Premature optimization is the root of all evil (Knuth)

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

12 MPI performance design

Be aware of latencies: bundle messages

(this may go again separation of concerns)

Consider ‘eager limit’

Process placement, reduction in number of processes

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

Debugging

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

13

Debugging is like being the detective in a crime movie

where you are also the murderer. (Filipe Fortes, 2013)

What do you do when your program misbehaves?

• Insert print statements, recompile, run again.

• Run your program in a debugger
• (also: attach a debugger, inspect a core dump)

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

14 Simple example: listing

tutorials/gdb/c/hello.c

#include <stdlib.h>
#include <stdio.h>
int main() {
printf("hello world\n");
return 0;
}

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

15 Simple example: running

%% cc -g -o hello hello.c
# regular invocation:
%% ./hello
hello world
# invocation from gdb:
%% gdb hello
GNU gdb 6.3.50-20050815 # ..... [version info]
Copyright 2004 Free Software Foundation, Inc. .... [copyright info
(gdb) run
Starting program: /home/eijkhout/tutorials/gdb/hello
Reading symbols for shared libraries +. done
hello world

Program exited normally.

(gdb) quit
%%
Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

16 Source listing

%% cc -o hello hello.c
%% gdb hello
GNU gdb 6.3.50-20050815 # ..... version info
(gdb) list

Important to use the -g compile option!

Eijkhout: programming
 Defensive programming
Debugging

17 Run with arguments

Memory debugging
Parallel Debugging

tutorials/gdb/c/say.c

#include <stdlib.h>
#include <stdio.h>
int main(int argc,char **argv) {
int i;
for (i=0; i<atoi(argv[1]); i++)
printf("hello world\n");
return 0;
}

%% gdb say
.... the usual messages ...
(gdb) run 2
Starting program: /home/eijkhout/tutorials/gdb/c/say 2
Reading symbols for shared libraries +. done
hello world
hello world
Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging
18 Memory problems 1
// square.c
int nmax,i;
float *squares,sum;

fscanf(stdin,"%d",nmax);
for (i=1; i<=nmax; i++) {
squares[i] = 1./(i*i); sum += squares[i];
}
printf("Sum: %e\n",sum);

%% cc -g -o square square.c
%% ./square
5000
Segmentation fault

The debugger will stop at the problem.

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

19 Stack trace

Displaying a stack trace

gdb lldb
(gdb) where (lldb) thread backtrace

(gdb) backtrace
#0 0x00007fff824295ca in __svfscanf_l ()
#1 0x00007fff8244011b in fscanf ()
#2 0x0000000100000e89 in main (argc=1, argv=0x7fff5fbfc7c0) at sq

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

20 Inspecting a stack frame

Investigate a specific frame

gdb clang
frame 2 frame select 2

Then print variables and such.

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

21 Out-of-bounds errors

// up.c
int nlocal = 100,i;
double s, *array = (double*) malloc(nlocal*sizeof(double));
for (i=0; i<nlocal; i++) {
double di = (double)i;
array[i] = 1/(di*di);
}
s = 0.;
for (i=nlocal-1; i>=0; i++) {
double di = (double)i;
s += array[i];
}

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

22 Out of bounds in debugger

Program received signal EXC_BAD_ACCESS, Could not access memo

Reason: KERN_INVALID_ADDRESS at address: 0x0000000100200000
0x0000000100000f43 in main (argc=1, argv=0x7fff5fbfe2c0) at u
15 s += array[i];
(gdb) print array
$1 = (double *) 0x100104d00
(gdb) print i
$2 = 128608

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

23 Breakpoints

Set a breakpoint at a line

gdb lldb
break foo.c:12 breakpoint set [ -f foo.c ] -l 12

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

24 Stepping

Stepping through a program

gdb lldb meaning
run start a run
cont continue from breakpoint
next next statement on same level
step next statement, this level or next

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

Memory debugging

Eijkhout: programming
 Defensive programming
Debugging
25 Program with problems
Memory debugging
Parallel Debugging

tutorials/gdb/c/square1.c

#include <stdlib.h>
#include <stdio.h>
//codesnippet gdbsquare1c
int main(int argc,char **argv) {
int nmax,i;
float *squares,sum;

fscanf(stdin,"%d",&nmax);
squares = (float*) malloc(nmax*sizeof(float));
for (i=1; i<=nmax; i++) {
squares[i] = 1./(i*i);
sum += squares[i];
}
printf("Sum: %e\n",sum);
//codesnippet end

return 0;
}
Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

26 Valgrind output

%% valgrind square1
==53695== Memcheck, a memory error detector
==53695== [stuff]
10
==53695== Invalid write of size 4
==53695== at 0x100000EB0: main (square1.c:10)
==53695== Address 0x10027e148 is 0 bytes after a block of si
==53695== at 0x1000101EF: malloc (vg_replace_malloc.c:236)
==53695== by 0x100000E77: main (square1.c:8)
==53695==

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

Parallel Debugging

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

27 Debugging

I assume you know about gdb and valgrind. . .

• Interactive use of gdb, starting up multiple xterms

feasible on small scale
• Use gdb to inspect dump:
can be useful, often a program crashes hard and leaves no dump

Note: compile options -g -O0

Eijkhout: programming
 Defensive programming
Debugging
28 Parallel debuggers
Memory debugging
Parallel Debugging

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

29 Buggy code

for (it=0; ; it++) {

double randomnumber = ntids * ( rand() / (double)RAND_MAX )
printf("[%d] iteration %d, random %e\n",mytid,it,randomnumb
if (randomnumber>mytid && randomnumber<mytid+1./(ntids+1))
MPI_Finalize();
MPI_Barrier(comm);
}

Eijkhout: programming
 Defensive programming
Debugging
30 Parallel inspection
Memory debugging
Parallel Debugging

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

31 Stack trace

Eijkhout: programming
 Defensive programming
Debugging
Memory debugging
Parallel Debugging

32 Variable inspection

Eijkhout: programming