TEST APPROACHES

TEST LEVELS
TEST TYPES
 SoftServe Confidential

Agenda
• Test Approaches
• Test Types by Test Levels
• Test Types by Test Objectives
• Testing Order
 SoftServe Confidential

Test Types
Acceptance Black Box Conformance
Security
Grey Box
Internationalization
Performance Load
Usability
System GUI Beta
Stress
Functional White Box Ad-hoc
Compatibility
Sanity
Exploratory Integration
Unit
Recovery Smoke Capacity
Error-Handling
Regression Alpha
Install/uninstall
Negative
Localization Confirmation
 SoftServe Confidential

Test Type Definition

Test Type it’s a group of test activities aimed at testing a component or
system focused on a specific test objective.

Test activities can be grouped by:

• Test Approaches
• Test Levels
• Test Objectives
 SoftServe Confidential

TEST APPROACHES
 SoftServe Confidential

Test Approaches
• Proactive and Reactive
• Manual and Automated
• Verification and Validation
• Positive and Negative
• Black-box, White-box and Grey-box
• Scripted and Unscripted
• Collaboration-based Test Approaches
 SoftServe Confidential

Proactive and Reactive testing

Reactive behavior is reacting to problems Proactive behavior involves acting in
when they occur instead of doing advance of a future situation, rather than
something to prevent them. just reacting.

Test design process is initiated as early as

Testing is not started until design and
possible in order to find and fix the
coding are completed.
defects before the build is created.
 SoftServe Confidential

Manual and Automated

Manual testing is the process through which Automated testing is the process through which
software developers run tests manually, comparing automated tools run tests that repeat predefined
program expectations and actual outcomes in order VS actions, comparing a developing program’s
to find software defects expected and actual outcomes. More info:

Manual Testing Automated Testing

Time consuming and tedious: Since test cases are executed by Fast: Automation runs test cases significantly faster than
human resources so it is very slow and tedious. human resources.

Less reliable: Manual testing is less reliable as tests may not be More reliable: Automation tests perform precisely same
performed with precision each time because of human errors. operation each time they are run.

Self-contained: Manual testing can be performed and completed Not self-contained: Automation can’t be done without manual
manually and provide self-contained results. testing. You have to manually check the automated test results.
Implicit: Implicit knowledge are used to judge whether or not Explicit: Automated tests execute consistently as they don’t get
something is working as expected. This enables engineer to find tired and/or lazy like us humans.
extra bugs that automated tests would never find.
 SoftServe Confidential

Verification and Validation

Are we building Are we building

the product right? the right product?

To ensure that the product

To ensure that work products
meets the user’s needs, and that
meet their specified
the specifications were correct in
requirements.
the first place.
 SoftServe Confidential

Positive and Negative

In positive testing In negative testing
our intention is our intention is

to prove that an application will

work on giving valid input data. i.e. to prove that an application will not
testing a system by giving its work on giving invalid inputs.
corresponding valid inputs.
 SoftServe Confidential

Black-box, White-box, Grey-box

Black-box Testing is a software testing White-box Testing is a software testing
method in which the internal structure/ method in which the internal structure/
design/ implementation of the item being design/ implementation of the item being
tested is NOT known to the tester. tested is known to the tester.

Black-box Testing White-box Testing

Levels Applicable To: System, Acceptance Test Levels Levels Applicable To: Component, Integration Test
Level
Responsibility: Quality Control Engineers Responsibility: Software Developers

Grey-box Testing is a software testing method

which is a combination of Black-box and White-box
Testing methods.
 SoftServe Confidential

Scripted and Unscripted

Scripted testing Unscripted testing

Test execution carried out by Test execution carried out without

following a previously documented previously documented sequence of
sequence of tests. tests.
 SoftServe Confidential

Unscripted testing
Exploratory testing Ad-hoc testing

Testing carried out informally; no formal test

An informal test design technique where the
preparation takes place, no recognized test
tester actively controls the design of the
design technique is used, there are no
tests as those tests are performed and uses
expectations for results and
information gained while testing to design
arbitrariness guides the test execution
new and better tests
activity

Exploratory Testing Ad-hoc Testing

Aim: to get the information to design new and Aim: to find defects
better tests
Result: defects are found and registered; new tests Result: defects are found and registered
are designed and documented for further usage
 SoftServe Confidential

Collaboration-based Test Approaches

Collaboration-based test approaches focus on defect avoidance by collaboration and

communication
• Collaborative User Story Writing
• Acceptance Criteria
• Acceptance Test-driven Development (ATDD)
 SoftServe Confidential

TEST TYPES BY TEST

LEVELS
 SoftServe Confidential

Test Levels
 SoftServe Confidential

Test Levels Acceptance

Test levels are groups of test activities that

are organized and managed together. System integration

According to ISTQB there are next test

levels: System
• Component testing
• Component integration testing
• System testing Component Integration
• System integration testing
• Acceptance testing
Component (Unit)
 SoftServe Confidential

Test Levels Acceptance

Test levels are characterized by the following

attributes: System integration
• Specific objectives
• Test basis, referenced to derive test
cases System
• Test object (i.e., what is being tested)
• Typical defects and failures
• Specific approaches and responsibilities Component Integration

Component (Unit)
 SoftServe Confidential

Entry Criteria and Exit Criteria

• Entry criteria define the preconditions for undertaking a given activity.

• Exit criteria define what must be achieved in order to declare an activity completed.
• Entry criteria and exit criteria should be defined for each test level and will differ based
on the test objectives.
• In Agile software development, exit criteria are often called Definition of Done
• Entry criteria that a user story must fulfill to start the development and/or testing
activities are called Definition of Ready
• Typical entry criteria: availability of resources, testware, initial quality level of a test
object.
• Typical exit criteria: measures of thoroughness and completion criteria.
 SoftServe Confidential

Component level
Testing on the Component Test Level is called
Acceptance Component (Unit, Module) testing

System integration Component (Unit) Test Level

Who DEV
When Component is developed
System Why To validate that each unit of the
software performs as designed
How White-box testing
Component Integration

Component (Unit)
 SoftServe Confidential

Component testing
Examples of a test basis: Typical test objects for component testing include:
• Detailed design • Components, units or modules
• Code • Code and data structures
• Data model • Classes
• Component specifications • Database modules

Typical defects and failures:

• Incorrect functionality (e.g., not as described in design
specifications)
• Data flow problems
• Incorrect code and logic
 SoftServe Confidential

Component Integration level

Component integration testing focuses on the interactions
Acceptance and interfaces between integrated components

Integration Test Level

System integration
Who DEV
When Units to be integrated are developed
System Why To expose faults in the interaction
between integrated units
How White-box
Component Integration Depends on definite units

Component (Unit)
 SoftServe Confidential

Component Integration testing

Test objects: Typical defects for component integration testing
• Subsystems • Incorrect data, missing data, or incorrect data encoding
• Databases • Incorrect sequencing or timing of interface calls
• Infrastructure • Interface mismatch
• Interfaces • Failures in communication between components
• APIs
• Microservices

Examples of a test basis:

• Software and system design
• Sequence diagrams
• Use cases
• Architecture at component or
system level
• Workflows
 SoftServe Confidential

System level
Testing on the System Test Level
Acceptance is called System testing

System Test Level

System integration
Who QC
When Separate units are integrated into
System System
Why To evaluate the system’s compliance
with the specified requirements
Component Integration How Black-box testing

Component (Unit)
 SoftServe Confidential

System testing
Examples of a test basis: Typical defects
• System and SRS • Incorrect calculations
• Risk analysis reports • Incorrect or unexpected system functional or non-
• Use cases functional behavior
• Epics and user stories • Incorrect control and/or data flows within the system
• State diagrams • Failure of the system to work properly in the production
environment(s)
• Failure of the system to work as described in system and
Test objects: user manuals
• Applications
• Hardware/software systems
• Operating systems
• System under test (SUT)
• System configuration and
configuration data
 SoftServe Confidential

System Integration testing

Test objects: Typical defects for system integration testing
• Subsystems • Inconsistent message structures between systems
• Infrastructure • Incorrect data, missing data, or incorrect data encoding
• Interfaces • Interface mismatch
• APIs • Failures in communication between systems
• Unhandled or improperly handled communication
failures between systems
Examples of a test basis:
• Sequence diagrams
• Use cases
• Architecture at component or
system level
• Workflows
 SoftServe Confidential

System Integration level

Testing on the Integration Test Level
Acceptance is called Integration testing

Integration Test Level

System integration
Who DEV, QC
When Units to be integrated are
System developed
Why To expose faults in the interaction
between integrated units
Component Integration How White-box/ Black-box/ Grey-box
Depends on definite units

Component (Unit)
 SoftServe Confidential

Acceptance level
Testing on the Acceptance Test Level is called
Acceptance Acceptance testing

Acceptance Test Level

Who People who have not been involved
System integration
into development
When Component is developed
System Why To evaluate the system’s compliance
with the business requirements and
assess whether it is acceptable for
Component Integration delivery
How Black-box testing

Component (Unit)
 SoftServe Confidential

Acceptance testing
Examples of a test basis: Test objects:
• Business processes • System under test
• User or business • System configuration and configuration data
requirements • Business processes for a fully integrated system
• Regulations, legal • Recovery systems and hot sites
contracts and • Operational and maintenance processes
standards • Forms
• Use cases • Reports
• Installation procedures • Existing and converted production data
• Risk analysis reports

Typical defects
• System workflows do not meet business or user requirements
• Business rules are not implemented correctly
• System does not satisfy contractual or regulatory requirements
• Non-functional failures such as security vulnerabilities, inadequate performance
efficiency under high loads, or improper operation on a supported platform
 SoftServe Confidential

Acceptance Testing
Common forms of acceptance testing include the following:
• User acceptance testing
• Operational acceptance testing
• Contractual and regulatory acceptance testing
• Alpha and beta testing

Alpha & Beta

Internal Acceptance External Acceptance

Testing (Alpha Testing) Testing (Beta Testing)
 SoftServe Confidential

TEST TYPES BY
TEST OBJECTIVES
 SoftServe Confidential

Test objectives

Depending on its objectives, testing will be organized differently:

• Testing of a function to be performed by the component or system;
• Testing of a nonfunctional quality characteristic, such as reliability or
usability;
• Testing related to changes.
 SoftServe Confidential

Test Types: Functional testing

• Functional testing evaluates the functions that a component or system should perform.
The functions are “what” the test object should do.
• Testing based on an analysis of the specification of the functionality of a component or
system.

Testing of function
(Functional testing) According to ISO 25010 the main objective of
Functional testing is:
Testing of software product • Functional completeness
characteristics • Functional correctness
(Non-functional testing)
• Functional appropriateness

Confirmation and
Regression testing
 SoftServe Confidential

Functional testing. Attributes.

Functional suitability: the degree to which a component or system provides functions
that meet stated and implied needs when used under specified conditions
• Functional completeness: degree to which the set of functions covers all the specified
tasks and user objectives
• Functional correctness: degree to which a product or system provides the correct
results with the needed degree of precision
• Functional appropriateness: degree to which the functions facilitate the
accomplishment of specified tasks and objectives
 SoftServe Confidential

Functional testing: Smoke

A subset of all defined/planned test cases that cover the
New Build
main functionality of a component or system, to
ascertaining that the most crucial functions of a program
work, but not bothering with finer details. Smoke
Testing
Purposes:
• is done before accepting a build for further
testing; NO
Test
Pass?
• is intended to reveal simple but critical failures
YES
to reject a software build\release;
Other
• determines whether the application is so badly Test Types
broken that further testing is unnecessary.
 SoftServe Confidential

Smoke testing Example

Task: Test new version of a Notepad application.
Smoke Testing Procedure: quickly check the main
Notepad features (run application, type text, open file,
edit file, save file).
Defect: There is no ability to Open a file. Button
“Open” does nothing.
Summary: build is not accepted, critical bug is logged
to a Bug Tracking system, developers' team and
project manager are informed by QC engineer about
that fact.
 SoftServe Confidential

Functional testing Example #1

• Verify adding of two numbers (5+3 should be 8);
• Verify subtraction of two numbers (5-2 should be 3);
• Verify multiplication of two number (5*3 should be 15);
• Verify division of two numbers (10/2 should be 5);
• Verify getting radical of some number (√25 should be 5);
• Verify multiplication of some number by zero (5*0
should be 0);
• Etc.
 SoftServe Confidential

Functional testing Example #2

Task: Test Save feature of Notepad application.

Functional Testing Procedure: test different flows
of Save functionality (Save new file, save updated file,
test Save As, save to protected folder, save with
incorrect name, re-write existed document, cancel
saving, etc.)
Defect: While trying to save file using Save As
command, still default file name can only be used.
User cannot change the filename because the edit-
box is disabled.
 SoftServe Confidential

Test Types: Non-functional testing

• Testing the attributes of a component or
system that do not relate to functionality.
Testing of function • Non-functional testing is the testing of
(Functional testing) “how well the system behaves”.

According to ISO 25010 Non-functional

Testing of software product
characteristics characteristics are:
(Non-functional testing) • Performance efficiency
• Compatibility
• Usability
Confirmation and
Regression testing
• Reliability
(Testing related to changes) • Security
• Maintainability
• Portability
39
 SoftServe Confidential

Non-functional testing
• Performance efficiency: Time behavior, Resource utilization, Capacity.
• Compatibility: Co-existence, Interoperability.
• Usability: Appropriateness recognizability, Learnability, Operability, User error
protection, User interface aesthetics, Accessibility.
• Reliability: maturity (robustness), fault-tolerance, recoverability and availability.
• Security: Confidentiality, Integrity, Non-repudiation, Accountability, Authenticity.
• Maintainability: Modularity, Reusability, Analysability, Modifiability and Testability.
• Portability: Adaptability, Installability and Replaceability.
 SoftServe Confidential

Non-functional testing: UI
UI Testing: The testing a product's graphical user interface to ensure it
meets its written specifications

Check if any UI recommendations exist for the application type your team
develop. Make sure dialogs you test comply with these recommendations.
 SoftServe Confidential

Non-functional testing: Performance

Performance Testing: Testing with the intent of determining how
efficiently a product handles a variety of events.

Purposes:
• demonstrate that the system meets performance criteria;

• compare two systems to find which performs better;

• measure what parts of the system or workload cause the system to

perform badly.
 SoftServe Confidential

Performance testing Example

Task: Server should respond in less than 2 sec when up to 100 users access it
concurrently. Server should respond in less than 5 sec when up to 300 users access it
concurrently.

Performance Testing Procedure: check response time of the server with 100 and
300 users at the same time.

Defect: starting from 200

concurrent requests respond time is 10-15 seconds.
 SoftServe Confidential

Non-functional testing: Load

Load testing is a type of performance testing conducted to evaluate the
behavior of a component or system with increasing load, e.g., numbers of
parallel users and/or numbers of transactions, to determine what load can
be handled by the component or system.
Purposes
• evaluation of performance and efficiency of software.
• performance optimization (code optimization, server configuration).
• selection of appropriate hardware and software platforms for the
application.
 SoftServe Confidential

Load testing Example

Task: Server should allow up to 500 concurrent connections.

Load Testing Procedure: emulate different amount of requests to server close to pick
value, for instance, measure time for 400, 450, 500 concurrent users. Verify that the server
is able to respond correctly with the maximum number of users.

Defect: Server returns “Request

Time Out” starting from 490
concurrent requests.
 SoftServe Confidential

Non-functional testing: Stress

Stress testing: A type of performance testing conducted to evaluate a
system or component at or beyond the limits of its anticipated or
specified work loads, or with reduced availability of resources such as
access to memory or servers
Purposes:
• the general study of the behavior of the system under extreme loads.
• examination of handling of errors and exceptions under extreme load.
• examination of certain areas of the system or its components under the
disproportionate load.
• testing the system capacity.
 SoftServe Confidential

Stress testing Example

Task: Server should allow up to 500 concurrent connections.

Stress Testing Procedure: emulate amount of requests to server greater than pick value,
for instance, check system behavior for 501, 510, and 550 concurrent users. Verify that
`500+` users get appropriate error message from server. (e.g. `an error occurred, please
try again later`) The server should not crash due to overload.

Defect: Server crashes starting from 500 concurrent requests and user’s data is lost.
Data should not be lost even in stress situations. If possible, system crash also should be
avoided.
 SoftServe Confidential

Non-functional testing: L10N, I18N

Internationalization is the process of
Localization is the process of adapting a
designing and coding a product so it can
globalized application to a particular
perform properly when it is modified for
culture/locale.
use in different languages and locales.

Localization (L10N) testing checks how Internationalization (I18N) testing

checks if all data/time/number/
well the application under test has been
currency formats are displayed according
Localized into a particular target to selected locale and if all language
language. specific characters are displayed.
 SoftServe Confidential

Localization testing Example

Task: Verify that ‘Login’ page is translated to German.

Localization Testing Procedure:

Test all labels and captions on the page whether they are translated to German; force
appearance of different messages (e.g.: when password or login does not exist) to check
whether they are localized and not truncated.

Defects:
• “Password you have entered does not exist in the system” message is truncated on
German Locale;
• “Login” label is not translated and still appears in English under German locale.
 SoftServe Confidential

Internationalization testing Example

Task:
Verify that list of users with German special characters (e.g.: “ü”, “ß” etc) in names are
sorted correctly by ‘First Name’ column

Functional Testing Procedure:

Create enough different users with special characters in First Name. Sort them via the
table to ensure that special characters are sorted correctly

Defect:
Sorting performs incorrectly: all names which start from special characters (e.g.: “ü”, “ß”
etc) are always listed at the end of the sorted column. Instead they should be sorted with
all other characters (e.g.: “ß” at once after word with “ss”, but on at the end of the list)
 SoftServe Confidential

Test Types: Confirmation

Confirmation testing confirms that an original defect has been successfully fixed.
Depending on the risk, one can test the fixed version of the software in several
ways, including:
• executing all test cases that previously have failed due to the defect,
• adding new tests to cover any changes that were needed to fix the defect
However, when time or money is short when fixing defects, confirmation testing
might be restricted to simply exercising the steps that should reproduce the failure
caused by the defect and checking that the failure does not occur
 SoftServe Confidential

Test Types: Confirmation

Confirmation testing or re-testing is a testing type that runs test cases
that failed the last time they were run, in order to verify the success of
corrective actions.

1. Build 1.0.0 – Test for Function A - Passed, test for Function B - Failed

2. In the next build 1.0.1 changes are introduced to Function B and

Common Library by developers

3. Now we need to re-run test for Function B to ensure, that Function B

was changed correctly.
 SoftServe Confidential

Test Types: Regression

Regression testing is a testing of a previously tested program following
modification to ensure that defects have not been introduced or
uncovered in unchanged areas of the software, as a result of the changes
made. It is performed when the software or its environment is changed.

Purpose:
• verifies that the system still meets its requirements.

May be any type of software testing (functional, GUI, etc…).

 SoftServe Confidential

TESTING ORDER
 SoftServe Confidential

Testing Order
• Executed every time the new build ready

Smoke

• Only if Smoke testing passed

Functional
Non-functional

• After any changes to already tested functionality

Regression
 SoftServe Confidential

Testing Order
Some factors to consider in prioritizing test cases:
• Mission-critical components
• Complex features
• Where failures would be most visible
• Features that undergo frequent changes
• Areas with past histories of problems
• Areas with complex coding
• Areas of most frequent use
 SoftServe Confidential

Summary
Test activities can be grouped using different classification:
• By the degree of automation (Manual and Automated);
• By the level of awareness about the system and its internal structure (Black-, White-,
Grey-box);
• By the basis of positive scenario (Positive and Negative);
• By the degree of preparedness to be tested (Scripted and Unscripted);
• By the degree of component isolation (by Test levels);
• By the Test Objectives.

All mentioned Test Types are not mutually exclusive but are complementary.
 SoftServe Confidential

Thank you!