Debugging

SWE 795, Fall 2019

Software Engineering Environments
 Today
• Part 1 (Lecture)(~80 mins)
• Debugging

• Break!

• Part 3 (Discussion)(45 mins)

• Discussion of readings

LaToza GMU SWE 795 Fall 2019 2

 Example

LaToza GMU SWE 795 Fall 2019 3

 Steps in fixing bugs
• Reproduce the problem
• Find cause of defect
• Investigate fix
• Implement fix
• Test fix

• Will focus on finding cause of defect today

LaToza GMU SWE 795 Fall 2019 4

Edit / Debug Cycle

Investigate
14%
28% 28%
50% 12% 86% 11%
Reproduce 5% 3%
Debug 40% Compile 20% 22%Test
6% 6% Edit 5% 4%
50% 33% 16% 29%
55% 11%
32% 22% 67%
20%
18%

86%
11%
3%
Reuse
11%

Circle size: % of %me Edge thickness: % of transi%ons observed

For tasks in code in your own codebase that you haven’t seen recently

5
LaToza and Myers. Developers ask reachability ques%ons. ICSE 2010.
 Debugging process model

L. Layman, M. Diep, M. Nagappan, J. Singer, R. Deline and G. Venolia, "Debugging Revisited: Toward Understanding the Debugging Needs of Contemporary Software Developers," 2013 ACM / IEEE
International Symposium on Empirical Software Engineering and Measurement, Baltimore, MD, 2013, pp. 383-392.
6
 Formulate & test hypotheses
• Use knowledge & data so far to formulate hypothesis
about why bug happened
cogitation, meditation, observation, inspection,
contemplation, hand-simulation,
gestation, rumination, dedication, inspiration,
articulation

• Recognize cliche
seen a similar bug before

• Controlled experiments - test hypotheses by gathering

data

Eisenstadt, M. Tales of Debugging from the Front Lines. Proc. Empirical Studies of Programmers, Ablex Publishing, Norwood, NJ,
1993, 86-112.

LaToza GMU SWE 795 Fall 2019 7

 Resources for testing hypotheses

L. Layman, M. Diep, M. Nagappan, J. Singer, R. Deline and G. Venolia, "Debugging Revisited: Toward Understanding the Debugging Needs of Contemporary Software Developers," 2013 ACM / IEEE
International Symposium on Empirical Software Engineering and Measurement, Baltimore, MD, 2013, pp. 383-392.

LaToza GMU SWE 795 Fall 2019 8

 Definitions
• Error - discrepancy between actual behavior of system
and intended behavior

• Failure - incorrect output value, exception, etc.; an error

that has become observable

• Fault - lines in code which are incorrect

• Debugging: determining the cause of a failure

• May involve finding location (fault localization) as well
as explanation.

LaToza GMU SWE 795 Fall 2019 9

 Resources used in debugging

L. Layman, M. Diep, M. Nagappan, J. Singer, R. Deline and G. Venolia, "Debugging Revisited: Toward Understanding the Debugging Needs of Contemporary Software Developers," 2013 ACM / IEEE
International Symposium on Empirical Software Engineering and Measurement, Baltimore, MD, 2013, pp. 383-392.

LaToza GMU SWE 795 Fall 2019 10

 Information needs in debugging
* How did this runtime state occur? (12)
data, memory corruption, race *
Where was this variable
last changed? (1)
conditions, hangs, crashes, failed API Why didn’t this
calls, test failures, null pointers * happen? (3)

Record execution history

omniscient debuggers
Provide interactions for browsing or searching

WhyLine
directly supports all 3 questions
in some situations

LaToza and Myers. Hard-to-answer ques%ons about code. PLATEAU 2010.

LaToza GMU SWE 795 Fall 2019 11
 In what
How do I debug
circumstances
* this bug in this
environment?(3)
* does this bug
occur? (3)

statistical debugging [1] No need to

Examine
reproduce
-Sample execution traces correlations
environment on
on user computers between crashes
developer
-Find correlations between and predicates
computer
crashes and predicates

[1] Liblit, B., Aiken, A., Zheng, A. X., and Jordan, M. I. 2003. Bug isolation via remote program sampling. In
Proceedings of the ACM SIGPLAN 2003 Conference on Programming Language Design and Implementation.

LaToza GMU SWE 795 Fall 2019 12

 How is this object different
✖ from that object? (1)

Which team’s component

✖ caused this bug? (1)
Which team should I
assign this bug to?
What runtime state changed
✖ when this executed? (2)

LaToza GMU SWE 795 Fall 2019 13

 Informa=on needs in debugging

• What code could have caused this behavior?

• What's statically related to this code?

• What code cause this program state?

Andrew J. Ko, Robert DeLine, and Gina Venolia. 2007. Information Needs in Collocated Software Development Teams. In Proceedings of the 29th international conference on
Software Engineering (ICSE '07). IEEE Computer Society, Washington, DC, USA, 344-353. DOI: https://doi.org/10.1109/ICSE.2007.45
14
 Activity
• What's the hardest debugging bug you've ever
debugged?
• What made it hard?

LaToza GMU SWE 795 Fall 2019 15

 What makes debugging hard?

L. Layman, M. Diep, M. Nagappan, J. Singer, R. Deline and G. Venolia, "Debugging Revisited: Toward Understanding the Debugging Needs of Contemporary Software Developers," 2013 ACM / IEEE
International Symposium on Empirical Software Engineering and Measurement, Baltimore, MD, 2013, pp. 383-392.

LaToza GMU SWE 795 Fall 2019 16

 What makes hard bugs hard to debug?
• Cause / effect chasm - symptom far removed from the root cause (15
instances)
timing / synchronization problems
intermittent / inconsistent / infrequent bugs
materialize many iterations after root cause
uncertain connection to hardware / compiler / configuration
• Inapplicable tools (12 instances)
Heisenbugs - bug disappears when using debugging tool
long run to replicate - debugging tool slows down long run even more
stealth bug - bug consumes evidence to detect bug
context - configuration / memory makes it impossible to use tool
• What you see if probably illusory (7 instances)
misreads something in code or in runtime observations
• Faulty assumption (6)
• Spaghetti code (3)

Eisenstadt, M. Tales of Debugging from the Front Lines. Proc. Empirical Studies of Programmers, Ablex Publishing, Norwood, NJ,
1993, 86-112.

LaToza GMU SWE 795 Fall 2019 17

 What makes hard bugs hard to debug?

LaToza GMU SWE 795 Fall 2019 18

 Activity

• What is a strategy you've used to debug a defect?

LaToza GMU SWE 795 Fall 2019 19

 Some debugging strategies
• Backwards: Find statement that generated incorrect
output, follow data and control dependencies
backwards to find incorrect line of code
• Forwards: Find event that triggered incorrect
behavior, follow control flow forward until incorrect
state reached
• Input manipulation: Edit inputs, observe differences
in output
• Blackbox debugging: Find documentation, code
examples to understand correct use of API

LaToza GMU SWE 795 Fall 2019 20

 Traditional debugging techniques
• Stepping in debugger
• Logging - insert print statements or wrap particular
suspect functions
• Dump & diff - use diff tool to compare logging data
between executions
• Conditional breakpoints
• Profiling tool - detect memory leaks, illegal memory
references

Eisenstadt, M. Tales of Debugging from the Front Lines. Proc. Empirical Studies of Programmers, Ablex Publishing, Norwood, NJ,
1993, 86-112.

LaToza GMU SWE 795 Fall 2019 21

 Debugging tools
• Make breakpoint debuggers better
• Support stepping backwards (omniscient debuggers)

• Support finding statement that generated incorrect output

• Find part of program that generated incorrect output (slicing)
• Output: subset of program
• Help developers follow data and control dependencies backwards
• Support for navigating control and data dependencies (WhyLine)
• Compare execution across different runs to guess locations that might be related
(automatic debugging)
• Output: list of potential fault locations
• Simplify input to find a simpler input that still generates failure (delta debugging)
• Output: simplified input
• Enable developers to directly answer questions about causality
• Support searching across control and data flow (Reacher)

• Help developers understand execution state

• Show visualizations which depict the current contents of memory
• Let developers experiment, editing program till they find the right one (live
programming)[editing code]
LaToza GMU SWE 795 Fall 2019 22
 Program analysis building blocks
• Many tools rely on gathering an execution trace
• Record the value of every expression as it
executes (or sometimes at function boundaries)
• Challenge: scalability
• Other tools use log data
• Gives developer control over what is being logged
• More easily scalable, requires developer to control
what is logged
• Other tools use test coverage data
• Which statement executes on each test, test
passing or succeeding

LaToza GMU SWE 795 Fall 2019 23

 Make breakpoint debugging better

• Debugging in a debugger is hard

• Forces developer to guess which methods to
step into
• Forces developers to guess which values to
instrument
• Changing guess requires reproing failure again
• Can be time consuming

• What if developers could debug forwards and

backwards?

LaToza GMU SWE 795 Fall 2019 24

 ZStep94
• Forwards / backwards stepping
through execution events See value of selected variables

• Select graphical output, find code that drew it

•

Demo: http://web.media.mit.edu/
~lieber/Lieberary/ZStep/ZStep.mov
Henry Lieberman and Christopher Fry. 1995. Bridging the gulf between code and behavior in programming. In Proceedings of the
SIGCHI conference on Human factors in computing systems (CHI '95), 480-486.

LaToza GMU SWE 795 Fall 2019 25

 Omniscient debugger

Demo / talk: http://video.google.com/videoplay?

docid=3897010229726822034#

Bill Lewis. Debugging backwards in time. In Proceedings of the Fifth International Workshop on Automated Debugging
(AADEBUG 2003), October 2003.

LaToza GMU SWE 795 Fall 2019 26

 Associating incorrect output with
responsible code

Brian Burg, Andrew J. Ko, and Michael D. Ernst. 2015. Explaining Visual Changes in Web Interfaces. In Proceedings of the 28th Annual ACM Symposium on User Interface Software &
Technology (UIST '15). ACM, New York, NY, USA, 259-268. DOI: https://doi.org/10.1145/2807442.2807473
LaToza GMU SWE 795 Fall 2019 27
 Find part of the program that caused
incorrect output
• Slice
• Subset of the program that is responsible for
computing the value of a variable at a program
point
• Backwards slice
• Transitive closure of all statements that have a
control or data dependency

• Originally formulated as subset of program

LaToza GMU SWE 795 Fall 2019 28

 Early evidence for slicing
• BEGIN Participants performed 3 debugging tasks on
READ(X, Y) short code snippets
TOTAL := 0.0
Asked to recognize code snippets afterwards
SUM := 0.0
IF X <= 1
THEN SUM := Y
ELSE BEGIN
READ(Z)
TOTAL := X * Y
END
WRITE(TOTAL, SUM)
END
• (Static) slice - subset of the program that produces the same variable
values at a program point
• Slice on variable Z at 12

Mark Weiser. 1982. Programmers use slices when debugging. Commun. ACM 25, 7 (July 1982), 446-452.
LaToza GMU SWE 795 Fall 2019 29
 Slicers debug faster
• Students debugging 100 LOC C++ programs
• Students given
Programming environment
Hardcopy input, wrong output, correct output
Files with program & input
• Compared students instructed to slice against everyone
else
Excluding students who naturally use slicing strategy

• Slicers debug significantly faster (65.29 minutes vs.

30.16 minutes)

Francel M. A. and S. Rugaber (2001). The Value of Slicing While Debugging, Science of Computer Programming, 40(2-3), 151-
169.

LaToza GMU SWE 795 Fall 2019 30

Dynamic
slicing

Hiralal Agrawal, Richard A. Demillo, and Eugene H.

Spafford. 1993. Debugging with dynamic slicing and
backtracking. Softw. Pract. Exper. 23, 6 (June 1993),
589-616.

LaToza GMU SWE 795 Fall 2019 31

 Compare faulty & unfaulty execution
traces

Ben Liblit. (2005). Cooperative bug isolation. Dissertation, UC Berkeley.

LaToza GMU SWE 795 Fall 2019 32

 Compare faulty & unfaulty execution
traces
• Program runs on user computer
Crashes or exhibits bug (failure)
Exits without exhibiting bug (success)
• Counters count # times predicates hit
Counters sent back to developer for failing and
successful runs
• Statistical debugging finds predicates that predict bugs
100,000s to millions of predicates for small applications
Finds the best bug predicting predicates amongst
these
• Problems to solve
Reports shouldn’t overuse network bandwidth (esp
~2003)
Logging shouldn’t kill performance
Interesting predicates need to be logged (fair
sampling)
Find good bug predictors from runs
Handle multiple bugs in failure runs
Ben Liblit. (2005). Cooperative bug isolation. Dissertation, UC Berkeley.

LaToza GMU SWE 795 Fall 2019 33

 Compare faulty & unfaulty execution
traces
• Predictor of what statements are related to a bug:
Fail(P) - Context(P)
Pr(Crash | P observed to be true) - Pr(Crash | P observed
at all)

• Example of a “likelihood ratio test”

• Comparing two hypotheses

• 1. Null Hypothesis: Fail(P) <= Context(P)
Alpha <= Beta
• 2. Alternative Hypothesis: Fail(P) > Context(P)
Alpha > Beta

Ben Liblit. (2005). Cooperative bug isolation. Dissertation, UC Berkeley.

LaToza GMU SWE 795 Fall 2019 34

 Simplify failure inducing input
• Long sequence of steps uncovered by tester
triggers a bug.
• Which of these steps are causing the bug
• Complex input - which part of input is
responsible for bug?
• Example - 10,700 Mozilla bugs (11/20/2000)

Andreas Zeller and Ralf Hildebrandt. Simplifying and Isolating Failure-Inducing Input. IEEE Transactions on Software Engineering
28(2), February 2002, pp. 183-200.
LaToza GMU SWE 795 Fall 2019 35
 Find shortest repro steps
• ddmin algorithm sketch:
• 1. Decompose input into pieces
2. Run tests on pieces
3. If there’s a piece that still fails, go back to 1 on
piece
Otherwise, found locally minimal smallest input

Andreas Zeller and Ralf Hildebrandt. Simplifying and Isolating Failure-Inducing Input. IEEE Transactions on Software Engineering
28(2), February 2002, pp. 183-200.

LaToza GMU SWE 795 Fall 2019 36