CPEN 522

Software Verification and

Testing

Ali Mesbah
ECE, UBC
 Today’s Objectives

– Getting to know each other

– What is this course about?
– How to successfully pass this course

2
 Contact Info
Lecturer: Dr. Ali Mesbah
Email: [email protected]
Office: KAISER 4044
Office hours: by appointment

3
 Bio

• Professor @ UBC (2021-now)

• Visiting Scientist @ Google (2017-2018)
• Associate Professor @ UBC (2016-2021)
• Assistant Professor @ UBC (2011-2016)
• Visiting Researcher @ Fujitsu Labs (2010)
• PhD in CS @ TU Delft (2005-2009)
• Software Engineer (2001-2005)

4
 Software AnaLysis and Testing (SALT)
http://salt.ece.ubc.ca

Research Group: Research:

• 3 PhD, 3 MSc • Software engineering
• Undergrad Interns • AI4SE: Generative AI
• Software testing and analysis
• Web and mobile engineering

Teaching:
• Software construction Partners:
• Software verification • Collaboration with ~10
different companies
• AI4SE
5
6
Our Testing Research

7
8
 Google
• Sabbatical (every 6 years)
• Visiting Scientist @ Google (8 months)
– Mountain View office

9
What I liked about my visit @ Google

10
 What I liked about my visit @ Google
• The free food and snacks (of course)!

11
 What I liked about my visit @ Google
• The colorful offices!

12
 What I liked about my visit @ Google
• The amazing software dev. Infrastructure!

13
 My research @ Google
• Learning to fix compilation errors automatically
– Machine learning, deep learning
– Neural Machine Translation

• Large-scale Refactoring for Java Type Migration

– 300 millions lines of Java code

14
 TA

• Sijia Gu, [email protected]

15
 Resources
• UBC Canvas
– See Syllabus
• GitHub (Lab work)
– https://github.com
– Create an account on GitHub if you don’t have one!
– Submit your account (see Canvas for
announcements).

16
 Assessment

• Active Participation: 5%
• Seminar Presentations: 15%
• Midterm (February 28): 20%
• Lab Work: 30%
• Final Exam (April 10): 30%

17
 Canvas
– Syllabus (read it!)
– Lecture notes after each lecture
– Announcements
– Q&A, discussions -> participation marks!

18
 Active Participation
Active participation means
• class attendance,
• participation in the discussions in the class and
online
• asking good questions, responding to questions
asked

Negative points will result from:

• not show up, not paying attention, not answering
questions, not asking (good) questions.

19
 Student Presentations
• Each student presents once
• A topic on software verification and testing
(more details will follow soon)

20
 Student Presentations
Topics:

1. A related technique
2. A tool, + demo
3. A paper/article
4. Industrial case studies

A list of potential topics will be provided soon.

21
 Lab Work
• Work in Groups
• Online lab meetings with TA
• Assignments: 3 parts
– Web Testing
– REST API
– Performance Testing
• Project: open-ended

22
 Project

• Project
– A project proposed by you!

23
 Lecture Style

– This will vary!

– Sometimes the lectures will be very typical, where I talk on

a certain topic

– Mostly mini-lectures plus in-class exercises -- here’s where

participation comes in!

– Some will be async (watch a video/lecture)

– Presentations by you on hot testing related topics and

tools

24
Questions?

25
– What is software verification and testing?

– Why is it important?

– What do you want to learn in this course?

26
Part II: Preliminaries

Pezze & Young

Chapters 1-4
 Bug

1946

30
“Our civilization runs on software”

31
 “Software is eating the world”

Discuss with your neighbors and write down

– What does this sentence mean?
– Do you agree?
– Can you provide some examples?

32
 “Software is eating the world”
Marc Andreessen (Netscape)

• Digital docs (PDF) ate libraries

• E-mail ate mail services
• Netflix ate video stores
• Expedia ate travel agents
• Amazon is eating retail stores
• Uber is eating taxis
• AirBnB is eating hotels
• …

33
 “AI is eating the world”

Discuss with your neighbors and write down

– What does this sentence mean?
– Do you agree?
– Can you provide some examples?

34
 Software Dependability
More and more we depend on software

Which in turn means software needs to be more

and more dependable!

35
 Boeing 787 (Dreamliner) bug

A software bug that

causes a complete
electric shutdown
of Boeing’s 787 and
potentially “loss of
control” of the
aircraft.
http://www.theguardian.com/business/2015/may/01/us-aviation-authority-boeing-787-dreamliner-bug-could-cause-loss-of-control

36
 More examples of
software defects

Toyota's electronic throttle control system (ETCS) had

software bugs that could cause unintended acceleration. At
least 89 people were killed as a result

In 2013, US court rules against Toyota:

• Toyota’s ETCS source code is of unreasonable quality.
• Toyota’s source code is defective and contains bugs,
including bugs that can cause unintended acceleration.
• code-quality metrics predict presence of additional bugs.

37
 More infamous examples
• NASA Mars Polar Lander destroyed
– its flight software mistook vibrations due to
atmospheric turbulence for evidence that the vehicle
had landed and shut off the engines 40 meters from
the Martian surface!
• Apple’s “goto fail” bug: a duplicated line of code
which caused a (SSL) public key certificate check
to pass a test incorrectly.
• Heartbleed, an OpenSSL vulnerability
– 900 SINs stolen from Canada Revenue 2014

38
39
“It's all done. I just need to test it.”

40
“It's all done. I just need to test it.”

“It's much easier to develop

software when it doesn't have to
be correct. :)”

41
 CPEN 522:
Software Verification and Testing

– What do you want to learn in this course?

42
 This Course
• Techniques
– The basics to be effective

• Embedding in development process

– Testing as a communication vehicle

• Early testing as a design activity

– Testing shapes the system

• Testing as an engineering discipline

– Tradeoff in costs and quality
– Scientific basis
43
 Skills You’ll Learn

• Test design:
– Which test cases are likely to reveal errors?
• Test assessment:
– Have I tested enough? Are my tests good?
• Analysis techniques
– Control-flow analysis
– Data-flow analysis
– Static bug checkers
44
 Textbook

You can now request a free digital copy of the book:

https://ix.cs.uoregon.edu/ michal/book/free.php
~
45
 Pezzè & Young
• Part I: Intro
• Part II: Basic techniques
– “finite abstractions of behavior”
• Part III: Problems & Methods
– Test case selection, test adequacy
– Functional & model-based testing
– Testing object-oriented software
• Part IV: Process
46
 xvii
The Book: Pezzè & Young
• Goal is to
– achieve balance of costs, schedule, and quality

• Testing and analysis are

– integral to modern software engineering practice
– equally important and technically demanding as
other aspects of development

47
 Learning Objectives
• Knowledge level
– Raw facts: essential software analysis methods,
tools, techniques, ...
• Application level:
– Practical: actually use selected techniques
• Evaluation level:
– Analytical: Decide what’s useful in your project
– Criticize, analyze, investigate, reflect, innovate, ...

Be a better Software Analyzer -> Developer -> Manager -> 48

Engineer
 Topics Covered
• principles and challenges of software verification and testing,
• static/dynamic analysis,
• requirements-based testing and acceptance testing,
• different levels of testing including unit, integration, system testing,
• regression testing, test selection and prioritization,
• test oracles,
• adequacy analysis and coverage,
• fault-based analysis and mutation testing,
• testing domain-specific systems: web application testing, GUI
testing, etc
• control and data flow analysis
• program analysis, symbolic execution, concolic testing
• problem tracking, debugging, and fault localization,
• program repair

49
Questions?

50
 Testing a SaaS

• You are hired as a test engineer.

• The goal is to improve the quality and find any
current and potential future bugs.
• In pairs, devise a test plan in which you outline
how you would go about achieving the goal.

51