Exploratory Testing

Yulai de Meer Fjeld

20.04.2021
Agenda
• Intro
• Testing taxonomy and practices
• Testing is context-dependent
• Exploratory testing
• What | How | When
• Pros / Cons
• Exercise: Basic calculator
• Summary
About
• Yulai de Meer Fjeld
• IFI graduate
• M.Sc. Programming and networks
• Software developer @ Bouvet
• Scrum master
• Full stack (three years)
• Java, Spring, Angular
• Large, agile context (10+ teams)
Testing Taxonomy
Testing Practices
• Pair programming
• Code review
• Static analysis
• Etc.

All contribute to making you a better developer and tester.

Testing is Context-Dependent
Testing is Context-Dependent
• Software systems are different
• Intended for different users
• Used in different ways
• Pose different risks
Testing is Context-Dependent
• Software systems are different
• Intended for different users
• Used in different ways
• Pose different risks

• Require different focus in testing

• We cannot test all systems the same way
• We need information about the system under test
• Contextual knowledge and information should guide the test effort
What is Exploratory Testing?
What is Exploratory Testing?
• Dynamic
• Execution of software / components / system
• Focus on functionality / expected behaviour
What is Exploratory Testing?
• Dynamic
• Execution of software / components / system
• Focus on functionality / expected behaviour

• Experience-based
• Based on experience / knowledge / intuition
• Often requires in-depth domain knowledge
Characteristics
Characteristics
• "Thinking" activity
• Informal test design technique
Characteristics
• "Thinking" activity
• Informal test design technique

• Test cases are not created in advance

• Check the system on the fly
Characteristics
• "Thinking" activity
• Informal test design technique

• Test cases are not created in advance

• Check the system on the fly

• Iterative process, widely used in agile

• Discover | Investigate | Learn
• Tester controls the design of the tests as they are performed, and uses
information gained during testing to design new and better tests
How to conduct Exploratory Testing?
How to conduct Exploratory Testing?
• Create a bug taxonomy
• Develop a bug classification
• Classify bugs found in past / similar projects
How to conduct Exploratory Testing?
• Create a bug taxonomy
• Develop a bug classification
• Classify bugs found in past / similar projects

• Understand the system under test

• Define risks (in light of "typical" bugs)
• Suggest ideas for testing
How to conduct Exploratory Testing?
• Create a bug taxonomy
• Develop a bug classification
• Classify bugs found in past / similar projects

• Understand the system under test

• Define risks (in light of "typical" bugs)
• Suggest ideas for testing

• Create a test charter to guide the test effort

• What to test | How to test | What to consider
How to conduct Exploratory Testing?
• Create a bug taxonomy
• Develop a bug classification
• Classify bugs found in past / similar projects

• Understand the system under test

• Define risks (in light of "typical" bugs)
• Suggest ideas for testing

• Create a test charter to guide the test effort

• What to test | How to test | What to consider

• Assess findings
• Refine new tests
Exercise: Basic Calculator
• Buggy calculator developed by "Mike Talks" (agile coach and software
tester)
• URL: https://testsheepnz.github.io/BasicCalculator.html#main-body

• Use the principles of exploratory testing to test the various

prototypes
Context: Simple Calculator
Context: Simple Calculator
• Basic arithmetic operations
Context: Simple Calculator
• Basic arithmetic operations

• Multiple input fields

Context: Simple Calculator
• Basic arithmetic operations

• Multiple input fields

• Dropdown list
Context: Simple Calculator
• Basic arithmetic operations

• Multiple input fields

• Dropdown list

• Multiple buttons
Context: Simple Calculator
• Basic arithmetic operations

• Multiple input fields

• Dropdown list

• Multiple buttons

• Checkbox
Bug Taxonomy: Classifying Bugs
• Basic arithmetic operations
• ?

• Multiple input fields

• ?

• Dropdown list
• ?

• Multiple buttons
• ?

• Checkbox
• ?
Bug Taxonomy: Classifying Bugs
• Basic arithmetic operations
• Divide by zero
• Large numbers / small number

• Multiple input fields

• Expected input, null/empty string vs. 0, SQL injection
• Combination of numbers and strings

• Dropdown list
• Available options, selecting

• Multiple buttons
• Button press should perform expected functions
• Responsive

• Checkbox
• Check / uncheck, availability
Understanding the System Under Test
Understanding the System Under Test
• Who are the customers and who are the users?
• Is the customer the same as the user?
• How can we describe the typical user?
Understanding the System Under Test
• Who are the customers and who are the users?
• Is the customer the same as the user?
• How can we describe the typical user?

• What will the product be used for?

• How important is feature X to the customers?
• How important is feature X to the users?
Understanding the System Under Test
• Who are the customers and who are the users?
• Is the customer the same as the user?
• How can we describe the typical user?

• What will the product be used for?

• How important is feature X to the customers?
• How important is feature X to the users?

• What is the time to market?

• How much is invested in the development of the product?
• How much is invested in the testing?
Understanding the System Under Test
• Who are the customers and who are the users?
• Is the customer the same as the user?
• How can we describe the typical user?

• What will the product be used for?

• How important is feature X to the customers?
• How important is feature X to the users?

• What is the time to market?

• How much is invested in the development of the product?
• How much is invested in the testing?

• What are the customers’ and users’ expectations?

Creating a Test Charter
• What to test?
• Arithmetic operations
• Input validation

• How to test?
• Compare various inputs to expected outputs

• Other considerations
• ??
Assessing Exploratory Testing
• Exploratory testing can prove hugely beneficial…
• Detects "different" kinds of bugs

• However, must be applied in the right context

• When lacking specifications and documentation
• When under severe time pressure: Limited preparation necessary

• Quality of the test effort depends on…

• Experience, skill, creativity of the tester
• Limited by the domain knowledge of the tester

• Difficult to reproduce failure scenarios

• Little to no documentation

• Should be used as a complement to other test techniques

Resources
• Mike Talks' "Basic Calculator":
https://testsheepnz.github.io/BasicCalculator.html#main-body
• OWASP Juice Shop. Famous site to test security vulnerabilities:
https://juice-shop.herokuapp.com/#/
• Heuristics Cheat Sheet: https://testobsessed.com/wp-
content/uploads/2011/04/testheuristicscheatsheetv1.pdf
References
• Previous lecture slides by Raluca Florea (2016)
• Black, R., van Veenendal, E., Graham, D. (2012). Foundations of Software
• Testing: ISTQB Certification 3E. Cengage Learning.
• IEEE 829: Standard for Software and System Documentation
• Heuristics cheat sheet, [online]: https://testobsessed.com/wp-
content/uploads/2011/04/testheuristicscheatsheetv1.pdf (last accessed 20.04.2021)
• Vasylyna, N. (23.03.2011). "Exploratory Testing – Advantages and Disadvantages", [online]:
https://blog.qatestlab.com/2011/02/23/exploratory-testing-advantages-and-disadvantages/ (last accessed
20.04.2021)
• Stahl, M. (16.10.2017). "Using Bug Taxonomy to Design Better Software Tests", [online]:
https://www.stickyminds.com/article/using-bug-taxonomy-design-better-software-tests (last accessed
20.04.2021)
• Mike Talks. Basic calculator. [online]: https://testsheepnz.github.io/BasicCalculator.html#main-body (last
accessed 20.04.2021)
• Bug Hunter Sam. (11.12.2019). "Mindmaps and heuristics for testing", [online]:
https://bughuntersam.com/mindmaps-and-heuristics-for-testing/ (last accessed 20.04.2021)