Chapter 6

Software Test Automation

Frameworks

1
Lesson Outlines

Part I Testing Framework

Part II Model Based Testing: MBT

2
 What is Automation Testing?

 Automation testing is a software testing

technique to test and compare the actual
outcome with the expected outcome.
 This can be achieved by writing test scripts or
using any automation testing tool.
 Test automation is used to automate repetitive
tasks and other testing tasks which are difficult
to perform manually.

Source: https://www.softwaretestinghelp.com/automation-testing-tutorial-1/ 3
 Part-I Lesson Objectives of Testing Framework

 By the end of the session, students will able to:

 Overview automation testing
 Define what testing framework is.
 Identify types of testing framework
 State pros and cons of types of testing frameworks
 Hands on Practice using open source software
testing framework tools – pytest, robot test
framework, selenium…

4
When To Automate?

 Frequent regression testing

 Repeated test case Execution is required
 User Acceptance Tests
 Faster Feedback to the developers
 Reduce the Human Effort

Softsmith Infotech 5
Test Automation Framework(TAF): Intro.
 What is Framework?
 A framework is considered to be a combination of set
protocols, rules, standards and guidelines that can be
incorporated or followed as a whole.
 Test Automation Framework (TAF):
 is scaffolding that is laid to provide an execution
environment for the automation test scripts.
 is a constructive blend of various guidelines, coding
standards, concepts, processes, practices, project
hierarchies, modularity, reporting mechanism, test data
injections etc. to pillar automation testing.

6
TAF: Advantages

Reusability of code
Maximum coverage
Recovery scenario
Low-cost maintenance
Minimal manual intervention
Easy Reporting

7
TAF: Types of Test Frameworks
There are various TAF types of framework depending on
the factors (re-usability, maintainability, ease of use,..) to do
in automation. The TAF types include:
 Linear (Record and Play) Testing Framework
 Module Based Testing Framework (MBTF)
 Library Architecture Testing Framework (LATF)
 Data Driven Testing Framework (DDTF)
 Keyword Driven Testing Framework (KDTF)
 Hybrid Testing Framework (HTF)
 Behavior Driven Development Framework (BDDF)
 ….

8
TAF: Linear scripting
In a linear automation framework, users target a specific program
functionality, create test scripts in sequential order and run them
individually. This process includes capturing all the tests like navigation,
inputs, etc., and playing them back repeatedly to conduct the test. This
also called record and playback testing framework.
Pros:
 Better for small projects, faster delivery to customer
Cons:
 No reusability at all.
 Hard coding the data doesn't allow us to run with multiple data
sets. Maintenance is high – It requires a lot of effort to do even
small changes.

9
TAF: Module Based
 The framework divides the entire “Application Under
Test” into a number of logical and isolated modules.
 For each module, we create a separate and
independent test script. Thus, when these test scripts
took together builds a larger test script representing more
than one modules.
 Pros: easy to reuse and modify
 Cons: Need to modify test script while changing test data

10
TAF: Library Arch
 The basic fundamental behind the framework is to determine
the common steps and group them into functions under a
library and call those functions in the test scripts whenever
required.
 Example: The login steps can be combined into a function
and kept into a library. Thus all the test scripts those require to
login the application can call that function instead of writing the
code all over again.

 Pros: great degree of re-usability like MBFT

 Cons: Need to modify test script while changing test data plus
adding library might complicate to some extent
11
TAF: Data Driven
 Unlike MBTF and LATF, Data Driven Testing Framework
helps the user segregate the test script logic and the test data
from each other. It lets the user store the test data into an
external database or other sources like csv, txt file, xml, etc..
 The data is conventionally stored in “Key-Value” pairs. Thus,
the key can be used to access and populate the data within
the test scripts.

12
TAF: Data Driven
 Example: Let us consider the “e-mail – Login” Functionality.
 The first step is to create an external file that stores the test
data (Input data and Expected Data). Let us consider an
excel sheet for instance
 The second step is to populate the test data into Automation
test Script. For this purpose, several API’s can be used to
read the test data. GUI might be developed to load test data
for test execution.

13
 TAF: Data Driven
 Pros:
 The most important feature of this framework is that it
considerably reduces the total number of scripts required to
cover all the possible combinations of test scenarios. Thus lesser
amount of code is required to test a complete set of scenarios.
 Increases flexibility and maintainability as any change in the
test data matrix would not hamper the test script code.
 Cons:
 The process is complex and requires an extra effort to come up
with the test data sources and reading mechanisms.
 Requires proficiency in a programming language that is being
used to develop test scripts

14
TAF: Keyword Driven
 KDTF is an extension to DDTF in a sense that it not only
segregates the test data from the scripts, it also keeps the
certain set of code belonging to the test script into an external
data file.
 These set of code are known as Keywords and hence the
framework is so named. Keywords are self-guiding as to what
actions need to be performed on the application.
 The keywords and the test data are stored in a tabular like
structure and thus it is also popularly regarded as Table driven
Framework. Take a notice that keywords and test data are
entities independent of the automation tool being used.

15
TAF: Keyword Driven...
 Example: Let us consider the “e-mail – Login” Functionality.
 The first step is to create a tabular specification of test case
for testing login. All keywords can be re-used many times in a
test case whenever needed. All the required keywords are
designed and placed in the base code of the framework.
 The locator column contains the locator value that is used to
identify the web elements on the screen or the test data that
needs to be supplied.

16
TAF: Keyword Driven...
 Pros:
 Possessing scripting knowledge is not required, unlike Data
Driven Testing
 A single keyword can be used across multiple test scripts.
 Cons:
 The user should be well versed with the Keyword creation
mechanism to be able to efficiently leverage the benefits
provided by the framework.
 The framework becomes complicated gradually as it grows
and a number of new keywords are introduced.


17
TAF: Hybrid
 HTF is as the name suggests, it is a combination of more
than one above mentioned frameworks. The best thing about
such a setup is that it leverages the benefits of all kinds of
associated frameworks.

18
TAF: Hybrid...
 Example of Hybrid Testing Framework: the Test sheet would
contain both the keywords and the Data for testing login page.

 In the above example, keyword column contains all the

required keywords used in the particular test case and data
column drives all the data required in the test scenario.
 If any step does not need any input then it can be left empty.

19
TAF: Behavior Driven Development
 BDDF allows automation of functional validations in easily readable
and understandable format to Business Analysts, Developers,
Testers, etc. Such frameworks do not necessarily require the user to
be acquainted with the programming language.
 There are different tools available for BDD like cucumber, Jbehave
etc.
 Gherkin language is used to write test cases in Cucumber.

20
TAF: Behavior Driven Development...
 In the diagram, some of components are described below.
 Object Repository: Object Repository acronym as OR is constituted of the set
of locators types associated with web elements.

 Test Data: The input data with which the scenario would be tested and it can
be the expected values with which the actual results would be compared.

 Configuration File/Constants/ Environment Settings: The file stores the

information regarding the application URL, browser-specific information etc. It is
generally the information that remains static throughout the framework.

 Generics/ Program logics/ Readers: These are the classes that store the
functions which can be commonly used across the entire framework.

 Build tools and Continuous Integration: These are the tools that aids to the
capabilities of the framework to generate test reports, email notifications and
logging information.

21
Levels of Test Automation

FRAMEWORK:
Test / Process / Hybrid:
Methodologies, Libraries,
Level 3 Re-usable components

ADVANCED
METHODOLOGIES:
Data-Driven, Level 2
Keyword-Driven,
Hybrid

BASIC:
Level 1 Record & Playback,
Functional
Decomposition

22
Levels of Test Automation: Statistics

Level Type of Applications Effort spent on Tools Used Methodology

Automation

Level 1 Client Server, 200 person- Rational Robot, Record &

Web months WinRunner, Playback,
SilkTest, Functional
QTP Decomposition

Level 2 Client Server, Web, 700 person- Rational Robot, Data Driven,
Mainframe, Devices, months WinRunner, Keyword Driven,
ERP Packages SilkTest, Hybrid
QTP, Test
Complete

Level 3 Multilayered, Complex 300 person- QC, BPT, QTP, Framework,

Applications months SilkTest, Perl, Libraries,
Shell Scripting Reusable
Components

23
 Part-II
Model Based Testing (MBT)

24
 Part-II Lesson Objectives of MBT

 By the end of the session, students will able to:

 Discuss the evolution of software testing
 Define what Model based testing is.
 State pros and cons of types of MBT testing.

25
Evolution of Software Testing
 Evolution of Software Testing...

There Is Too Much to Test & Not Enough Time

Then…
 One computer platform
 One OS
 One app
 One browser
 One release per year

Now…
 Many platforms (PC, tablet, phone, cloud, voice
…)
 Multiple OS (Win, iOS, Android, Chrome, legacy,
…)
 More connected apps
 Multiple browsers (Chrome, IE, Firefox …
 Multiple releases (every month or sooner)
 Evolution of Software Testing...

Model Based Testing (MBT)

Automated Test Design ATD

Test Models

Keyword Driven
Testing Efficiency
Testing Frameworks

Test Scripts
Capture/Replay

Manual Testing
User Conference on
Advanced Automated Testing
 Evolution of Software Testing...

Manual Testing
• “It is cheap”! Anybody can do it!
• It is time consuming, ineffective,
and error prone
• It is still not (yet) replaceable
 When tests are hard (or too

expensive) to automate
 Exploratory testing (with data)
 Evolution of Software Testing...

Test Scripts and Capture Replay

• Automatic execution!
• Interactions have to be manually
performed or coded
• But capture/replay is still driven by
manual testing
• Scripting requires special skills
• One small change in SUT requires
re-recording or re-coding
 Evolution of Software Testing...

(Keyword Driven) Testing Frameworks

• Simpler to use, understand &

maintain
• Allows reduction of required skills for
test specification down to Excel!
• Keywords are reusable & tests
become retargetable!
• Still requires (highly) skilled
automation team for scripting
keywords

Example:
Login(username,password)
StartVehicle()
 Evolution of Software Testing...

“BDD”: Gherkin as Input to MBT

Feature: Shopping with esd.conformiq.com

Scenario: Successful shopping

Given a user has logged in successfully

When user adds 1 item(s) of "CQ0003"
Then shopping basket contains 1 item(s) Import & Refine
Scenario: Bad product id

Given a user has logged in successfully

When user adds 3 item(s) of "123"
Then application displays invalid entry
Generate Tests
Scenario: Remove item

Given a user has logged in successfully

When user adds 1 item(s) of "CQ0003"
Then shopping basket contains 1 item(s)
When user clicks Remove all button for
item "CQ0003"
Then shopping basket contains 0 item(s)

Execute!
 Evolution of Software Testing...

AI Techniques Have Been Leveraged for (Many) Years!

• Fuzzy and Probabilistic Logic  λ

λ

• Classification  λ λ λ λ
λ

• Search & Optimization 

λ
λ
λ λ λ
λ
• Expert Systems  λ λ

•
λ
(Deep) Machine Learning (ML) ?
λ λ λ

Remember this?

Today the term “AI” is generally reduced to (Deep) Machine Learning …

 Evolution of Software Testing...

Introducing Intelligent Test Automation

• From manual to automatically executable test in two steps

 Feed manual tests and train intelligent test automation (if any),

e.g., resolve unknown text to automation rules

 Execute and fix unrecognized object identifiers (if any)

• The more intelligent test automation is trained - the less it needs to

learn!
 Evolution of Software Testing...

Intelligent Test Automation Naturally Extends to MBT!

Requirements Test Design Test Creation Test Execution

Includes BAs & Testers Includes optimization Includes scripts, data & validations Includes harnesses

Test Files
(fully generated)

Custom Libraries

Exec Tool(s)

Intelligent Test Automation

 Traditional Automated Testing

Imagine that this projector is the software you are testing.

Traditional Automated Testing...

Typically, testers automate by creating static scripts.

 Traditional Automated Testing...

Given enough time, these scripts will cover the behavior.

 Traditional Automated Testing...

But what happens when the software’s behavior changes?

 Model Based Testing (MBT): Overview

So What’s a Model?

• A model is a description of a system’s behavior.

• Models are simpler than the systems they describe.

• Models help us understand and predict the system’s behavior.

MBT: Overview

Now, imagine that the top projector is your model.

MBT: Overview

The model generates tests to cover the behavior.

MBT: Overview

… and when the behavior changes…

MBT: Overview

… so do the tests.
 MBT: Overview

● “Model-based testing is black box testing process

for derivation of test cases from a model that
describe functional aspects of the system under test
and executing those test cases.”
is a partial description of

Model System

can be derived from

can be run against

Abstract Tests Executable Tests

see: Practical Model Based Testing- A Tools
Approach, Mark Utting & Bruno Legeard

are abstract versions of

 MBT: What is model-based Testing?
 Model-based testing is a software testing technique where the
run time behavior of the software under test is checked
against predictions made by a model.
 A model is a description of a system’s behavior.
 Behavior can be described in terms of input sequences,
actions, conditions, output, and flow of data from input to
output.
 It should be practically understandable and can be reusable;
shareable must have a precise description of the system
under test.

46
 MBT: What is model-based Testing?...
 There are numerous models available, and it describes
different aspects of the system behavior.
 Examples of the model are:
 Data Flow,
 Control Flow, Dependency Graphs
 Decision Tables,
 State transition machines
 Model-Based Testing describes how a system behaves in
response to an action (determined by a model). Supply action,
and see if the system responds as per the expectation.
 It is a lightweight formal method to validate a system. This
testing can be applied to both hardware and software testing.
47
 MBT: Note Pad Writing Testing Example
The model below explains the simplified approach of writing poetry in
notepad and possible actions related to each step. For each and every
action (like starting, Entering a poem, Saving),Test Case can be
generated, and the output can be verified.

48
 MBT: Finite State Machine
 Finite State Machines: This model helps testers to assess
the result depending on the input selected. Various
combinations of the inputs can result in a corresponding
state of the system. The system will have a specific state
and current state, which is governed by a set of inputs
given by the testers.
 Example: Consider a system that allows employees to
log- into the application. Now, the current state of the
employee is ”Out,” and it became ”In” once he signs- into
the system. Under the ”in” state, an employee can view,
print, and scan documents in the system.

49
 MBT: State Chart
State Charts: It is an extension of the Finite state machine and
can be used for complex and real time systems. Statecharts are
used to describe various behaviors of the system. It has a definite
number of states. The behavior of the system is analyzed and
represented in the form of events for each state.
Example: Defects are raised in the defect management tool with
the status as New. Once it is fixed by developers, it has to be
changed to status Fixed. If a defect is not fixed, change status to
Re-open. State charts should be designed in such a way that it
should call for an event for each state.

50
 MBT: Steps

Five Main Steps

1) Model the SUT
2) Generate abstract tests from the
model.
3) Concretize the abstract
tests to make them
executable
4) Execute the tests on
the SUT and assign
verdicts.
5) Analyze the test results.
 MBT: Challenges

 Deployment of MBT in every organization obviously

requires a high amount of investment and effort.

Following are drawbacks of MBT in Software Engineering.

 Necessary Skills required in testers
 Learning curve time will be more
 Difficult to understand the model itself

52
 MBT: Advantages

Model Testing: The following are benefits of MBT:

 Easy test case/suite maintenance

 Reduction in Cost

 Improved Test Coverage

 Can run different tests on n number of machines

 Early defect detection

 Increase in defect count

 Time savings

 Improved tester job satisfaction

53
Lab 1 Pytest Framework in python
 Presentation slide
Hands on Practice using Pytest Framework
 Practice 1: Testing simple functions
 Source: https://docs.pytest.org/en/7.1.x/how-to/assert.html
 Practice 2: Testing Bank Account
 Source: https://www.sitepoint.com/python-unit-testing-unittest-
pytest/

54
Lab 2 Robot Framework in python
Presentation slide
Hands on Practice using Robot Framework in python
 Practice 1: Testing Hello Robot demo application
 Source:https://www.geeksforgeeks.org/robot-framework-in-python/
 Practice 2: Testing desk calculator demo application
 Source: https://github.com/robotframework/RobotDemo
 Practice 3: Testing Google Search
 Source:https://blog.devgenius.io/robot-framework-how-to-guide-
for-web-testing-6a5bfa877064
 Practice 4: Testing Ecommerce Search
 Source: https://blog.devgenius.io/robot-framework-how-to-guide-
for-web-testing-6a5bfa877064

55