EXN QA Internship

Testing types – lesson 1

Belgium Session name: Testing types

Matenstraat 214 (a)
2845 Niel
+32 32 130 383 (t) Main objective: interns learn about different testing levels, types and techniques. Also learn about
+32 32 131 454 (f) “Exploratory testing” and “State transition testing”
Romania Secondary objectives:
Barbu Lautaru 11 (a)
700399 Iasi
+40 332 808 044 (t) Pre-requisites:
+40 332 811 504 (f)
Logistics requirements:
[email protected]
Information sources:
www.expertnetwork.eu

Training time:
Working time:
 page 2 of 6

Testing levels
In SCRUM (Agile) Quality Assurance (QA) is the responsibility of everyone, not only the QA engineers. QA is all the
activities we do to ensure correct quality during development of new products. That’s why tests are frequently group by
where they are added in the software development process. The main levels during the development process are:
- unit testing (currently not done)
- integration testing
- system testing

Unit testing refers to tests that verify the functionality of a specific section of the code. Developers usually write these
types of tests as they work on code (white-box style), to ensure that the specific function is working as expected. One
function might have multiple tests, to catch corner cases or other branches in the code. Unit testing aims to eliminate
construction errors before code is promoted to additional testing; this strategy is intended to increase the quality of the
resulting software as well as the efficiency of the overall development process.

Integration testing refers to tests that aim to verify and expose defects in the interfaces and interaction between integrated
components/modules.

System testing is conducted on a completely integrated system and aims to verify that the system meets its requirements.
System testing is an extremely broad area, but can be broken down quickly into its main ingredients as described on the
next chapter.

Testing types
Bellow are the testing types performed by the QA engineers during the different phases of the project (some of them
require specific knowledge or higher seniority):
- documentation testing
- functional testing
- usability testing
- compatibility testing
- sanity and smoke testing
- regression testing
- non-functional testing (security, performance, load, scalability, stress, recoverability)
- acceptance testing
- beta testing

Testing techniques
- exploratory testing
- state transition testing
- equivalence partitioning
- boundary value analysis

Exploratory testing
 page 3 of 6

Exploratory testing is all about discovery, investigation and learning. It emphasizes on personal freedom and responsibility
of the individual tester. Test cases are not created in advance but QA engineers check system on the fly. They may note
down ideas about what to test before test execution. The focus of exploratory testing is more on testing as a "thinking"
activity.

Under scripted testing, you design test cases first and later proceed with test execution. On the contrary, exploratory testing
is simultaneous process of test design and test execution all done at the same time.

Scripted test execution is usually a non-thinking activity where QA engineers execute the test steps and compare the actual
results with expected results. Such test execution activity can be automated.

Here are some major differences between scripted and exploratory testing.
Scripted testing
- directed from requirements Exploratory testing
- directed from requirements and exploring during testing
- determination of test cases well in advance - determination of test cases during testing
- confirmation of testing with the requirements - investigation of system or application
- emphasizes on prediction and decision making - emphasizes on adaptability and learning
- involves confirmed testing - involves investigation
- is about controlling tests - is about improvement of test design
- like making a speech - you read from a draft - like making a conversion - its spontaneous
- the script is in control - the tester's mind is in control
Also related to exploratory testing:
- is not random testing but it is ad-hoc testing with purpose of finding bugs
- is structured and rigorous
- is cognitively structured as compared to procedural structure of scripted testing
- is highly teachable and manageable

Exploratory test preparation goes through following 5 stages detailed below and it is also called session based test
management:
1. Create a bug taxonomy (classification)
 categorize common types of faults found in the past projects
 analyze the root cause analysis of the problems or faults
 find the risks and develop ideas to test the application.
2. Test charter
 test charter should suggest: what to test, how it can be tested and what needs to be looked
 test ideas are the starting point of exploration testing
 test charter helps determine how the end user could use the system
3. Time box
 this method includes pair of QA engineers working together not less than 90 minutes
 there should not be any interrupted time in those 90 minutes session
 time box can be extended or reduced by 45 minutes
 this session encourages QA engineers to react on the response from the system and prepare for the correct
outcome
4. Review results
 evaluation of the defects
 learning from the testing
 analysis of coverage areas
5. Debriefing
 compilation of the output results
 compare the results with the charter
 check whether any additional testing is needed
 page 4 of 6

 
During exploratory execution, following needs to be done:
- mission of testing should be very clear
- keep notes on what needs to be tested, why it needs to be tested and the assessment of the product quality
- tracking of questions and issues raised during exploratory testing
- better to pair up the QA engineers for effective testing
- the more we test, more likely to execute right test cases for the required scenarios
 
It is very important to document and monitor the following:
- test coverage - whether we have taken notes on the coverage of test cases and improve the quality of the software
- risks - which risks needs to be covered and which are all important ones?
- test execution log - recordings on the test execution
- issues / queries - take notes on the question and issues on the system
 
Pros of exploratory testing:
- this testing is useful when requirement documents are not available or partially available
- it involves investigation process which helps find more bugs than normal testing
- uncover bugs which are normally ignored by other testing technique
- helps to expand the imagination of QA engineers by executing more and more test cases which finally improves
productivity as well
- this testing drill down to smallest part of application and covers all the requirements
- this testing covers all the types of testing and it covers various scenarios and cases
- encourages creativity and intuition
- generation of new ideas during test execution

Cons of exploratory testing:

- this testing purely depends on the QA engineer skills
- limited by domain knowledge of the QA engineer
- not suitable for long execution time

Challenges of exploratory testing

- learning to use the application or software system is a challenge
- replication of failure is difficult
- determining whether tools need to be used can be challenging
- determine the best test cases to execute can be difficult
- reporting of the test results is challenge as report doesn't have planned scripts or cases to compare with the actual result
or outcome
- documentation of all events during execution is difficult to record
- don't know when to stop the testing

When to use exploratory testing?

- when testing team has experienced QA engineers
- when early iteration is required
- when there is a critical application
- when new QA engineers join the team
 
Conclusion
Exploratory testing is performed to overcome the limitations of scripted testing. It helps in improving the test case suite. It
empathizes on learning and adaptability. Smarter exploratory testing finds more errors in less time.
State transition testing
State transition testing is a black box testing technique and is used where some aspect of the system can be described in
what is called a “finite state machine”. This simply means that the system can be in a (finite) number of different states,
and the transitions from one state to another are determined by the rules of the “machine”.
 page 5 of 6

This is the model on which the system and the tests are based. Any system where you get a different output for the same
input, depending on what has happened before, is a finite state system.

A state transition model has four basic parts:

- the states that the software may occupy (open/closed or funded/insufficient funds)
- the transitions from one state to another (not all transitions are allowed)
- the events that cause a transition (withdrawing money, closing a file)
- the actions that result from a transition (an error message, or being given your cash)

Figure below, shows a state transition example of entering a Personal Identity Number (PIN) to a bank account. The
states are shown as circles, the transitions as lines with arrows and the events as the text near the transitions. (We have not
shown the actions explicitly on this diagram, but they would be a message to the customer saying things such as ‘Please
enter your PIN’.)

The state diagram shows seven states but only four possible events (Card inserted, Enter PIN, PIN OK and PIN not OK).
We have not specified all of the possible transitions here – there would also be a time-out from ‘wait for PIN’ and from the
three tries which would go back to the start state after the time had elapsed and would probably eject the card. There would
also be a transition from the ‘eat card’ state back to the start state. We have not specified all the possible events either –
there would be a ‘cancel’ option from ‘wait for PIN’ and from the three tries, which would also go back to the start state
and eject the card. 

In deriving test cases, we may start with the following:

- First test case here would be the normal situation, where the correct PIN is entered the first time.
- A second test (to visit every state) would be to enter an incorrect PIN each time, so that the system eats the card.
- A third test we can do where the PIN was incorrect the first time but OK the second time, and another test where the
PIN was correct on the third try. These tests are probably less important than the first two.
- Note that a transition does not need to change to a different state (although all of the transitions shown above do go to a
different state). So there could be a transition from ‘access account’ which just goes back to ‘access account’ for an
action such as ‘request balance’.
 page 6 of 6

Test conditions can be derived from the state graph in various ways. Each state can be noted as a test condition, as can each
transition. However this state diagram, even though it is incomplete, still gives us information on which to design some
useful tests and to explain the state transition technique.

We need to be able to identify the coverage of a set of tests in terms of transitions. We can also consider transition pairs
and triples and so on. Coverage of all individual transitions is also known as 0-switch coverage, coverage of transition pairs
is 1-switch coverage, coverage of transition triples is 2-switch coverage, etc.