TESTING FRAMEWORKS

PRESENTATION BY ANIL SHEORAN

Agenda
01 What is a Framework ?

02 Benefits of Test Automation Framework

03 Types of Automated Testing Frameworks

04 Create a testing framework

 Welcome!!
These slide’s will provide a high-level over-
view of each type of framework and how
they can contribute to the success of any
automated testing process
 What is a Framework?

What is a Framework ?
A framework is the basic structure of something. It’s a set of ideas or facts that provide support for
01 something. In the case of business problems, a framework creates the basic structure that gives
focus and support to the problem you’re trying to solve.

What is a Test Automation Framework ?

Test Automation Framework is a collection of various tools used to set the rules and guidelines for
02 developing the test cases for the test professionals. These rules can be coding standards, handling
test data, store the test results, and other resources. The test framework helps in developing the
test scripts and recording them.
04
03
 What is a Framework?

What goes in a Framework ?

03 When you install a software framework, the first thing that you need to
take care of is the system requirements. Once a framework is installed
and configured, it creates a directory structure.

For example, fig. (i) illustrates the directory structure of the Framework.
Each of these folders could have additional directories. Directories can
further have files, classes, templates, and more.

04
 What is a Framework?
Difference between a Library and a Framework

04 Framework : A collection of libraries which provide unique properties and behaviour to your application. (All in-
gredients)
Library : A well defined set of instructions which provide unique properties and behaviour to your data. (Oil on
Fish)

Example: Let's say you want to cook a fish curry. Then you need ingredients like oil, spices and
other utilities. You also need fish which is your base to prepare your dish on (This is data of your
application). all ingredients together called a framework. Now you goanna use them one by one or
in combination to make your fish curry which is your final product.

Now, if you consider only one ingredient, like say oil. Now what brand of oil you want to use is up
to you. This will only change taste of your application. Their use depends on the developer's pref-
erence and they are easily replaceable.
04
 Benefits & Disadvantages
Benefits

• It helps in avoiding the duplication of test cases as well that is used to automate across the platforms.
• It improves test efficiency.
• It maximizes the test coverage and functionality of the application.
• It also defines the reusability of code.
• The multiple data sets can be executed easily.
• It helps in avoiding the hard code data.
• It lowers the cost of maintenance and easily scalable.
• Testing and debugging the code is a lot easier and can be done even by testers who do not own the code

04
• It helps the team to organize the test suites and helps in improving the test efficiency.

Disadvantages :
• The scripts developed using a particular framework are not reusable for other frameworks.
• The individual should have good knowledge of frameworks as well.
• Sometimes the data and complexity make the automating test not working in favour of the tester.
 Types of Framework
There are five common types of test automation frameworks, each with their own architecture and differing benefits and disadvantages.

Linear Automation Framework

Record and playback
feature

Modular Based Testing Framework Data-Driven Framework

Built on the concept of abstraction Use for testing bunch of data from ex-
ternal file

Keyword-Driven Framework Hybrid Testing Framework

Keywords are also stored in an ex- Combination of multiple frame-
ternal data table work
Types of Framework
Linear Automation Framework
01
Linear Automation Framework

With a linear test automation framework, also referred to

as a record-and-playback framework, testers don’t need
to write code to create functions and the steps are written
in a sequential order. In this process, the tester records
each step such as navigation, user input, or checkpoints,
and then plays the script back automatically to conduct
the test.

Example : Consider logging into Flight Reservation

Application and checking whether the application
has loaded on successful log-on. Here , the tester
will simply record the steps and add validation
steps.
 01. Linear Automation Framework
Advantages Disadvantages

• There is no need to write custom code, so expertise in • The scripts developed using this framework aren’t reusable.
test automation is not necessary. The data is hardcoded into the test script, meaning the test
• This is one of the fastest ways to generate test scripts cases cannot be re-run with multiple sets and will need to be
since they can be easily recorded in a minimal amount of modified if the data is altered.
time. • Maintenance is considered a hassle because any changes
• The test workflow is easier to understand for any party to the application will require a lot of rework. This model is
involved in testing since the scripts are laid out in a se- not particularly scalable as the scope of testing expands.
quential manner.
• This is also the easiest way to get up and running with
automated testing, especially with a new tool.
Types of Framework
Modular Based Testing Framework
02
Modular Based Testing Framework

Implementing a modular framework will require testers to di-

vide the application under test into separate units, functions,
or sections, each of which will be tested in isolation. After
breaking down the application into individual modules, a test
script is created for each part and then combined to build
larger tests. These larger sets of tests will begin to repres-
ent various test cases.

A key strategy in using the modular framework is to build an

abstraction layer, so that any changes made in individual
sections won’t affect the overarching module.

Example : Using the same example as above, re-

corded steps will be placed inside a function.
Now, you will call this function in the main script
 02. Modular Based Testing Framework
Advantages Disadvantages

• If any changes are made to the application, only the • Data is still hard-coded into the test script since the tests are
module and it’s associated individual test script will executed separately, so you can’t use multiple data sets.
needs to be fixed, meaning you won’t have to tinker with • Programming knowledge is required to set up the framework
the rest of the application and can leave it untouched.
• Creating test cases takes less effort because test scripts
for different modules can be reused.
Types of Framework
Data-Driven Framework
03
Data-Driven Framework

In this Framework , while Test Case logic resides in Test

Scripts, the Test Data is separated and kept outside the Test
Scripts. Test Data is read from the external files (Excel Files,
Text Files, CSV Files, ODBC Sources, DAO Objects, ADO
Objects) and are loaded into the variables inside the Test
Script. Variables are used both for Input values and for Veri-
fication values. Test Scripts themselves are prepared either
using Linear Scripting or Modular Framework.

Example : Developing the Flight Reservation Login

script using this method will involve two steps.
Step 1) Create a Test - Data file which could be Ex-
cel , CSV , or any other database source.
Step 2) Develop Test Script and make references to
your Test- Data source.
 03. Data-Driven Framework
Advantages Disadvantages

• Tests can be executed with multiple data sets. • You’ll need a highly-experienced tester who is proficient in
• Multiple scenarios can be tested quickly by varying the various programming languages to properly utilize this
data, thereby reducing the number of scripts needed. framework design. They will need to identify and format the
• Hard-coding data can be avoided so any changes to the external data sources and to write code (create functions)
test scripts do not affect the data being used that connect the tests to those external data sources seam-
• You’ll save time by executing more tests faster. lessly.
• Setting up a data-driven framework takes a significant
amount of time.
Types of Framework
Keyword Driven Testing Framework
04
Keyword Driven Testing Framework
In a keyword-driven framework, each function of the applic-
ation under test is laid out in a table with a series of instruc-
tions in consecutive order for each test that needs to be run.
In a similar fashion to the data-driven framework, the test
data and script logic are separated in a keyword-driven
framework, but this approach takes it a step further.

With this approach, keywords are also stored in an external

data table (hence the name), making them independent
from the automated testing tool being used to execute the
tests. Keywords are the part of a script representing the
various actions being performed to test the GUI of an ap-
plication. These can be labeled as simply as ‘click,’ or
‘login,’ or with complex labels like ‘clicklink,’ or ‘verifylink.’
 04. Keyword Driven Testing Framework
Example : To understand this consider you want to verify login op-
Advantages

eration for a Website say YAHOO MAIL. The table will look like this • Minimal scripting knowledge is needed.
Step 1: Creating Data Table (Different from Test-Data Table created in Data • A single keyword can be used across multiple test
Driven Framework). This Data Table contains Action to be performed on scripts, so the code is reusable.
GUI Objects and corresponding arguments if any. Each row repres- • Test scripts can be built independent of the application
ents one Test Step. under test.
Step 2: Writing Code in the form of Component Functions. Disadvantages

• The initial cost of setting up the framework is high. It is time-

consuming and complex. The keywords need to be defined
and the object repositories / libraries need to be set up.
• You need an employee with good test automation skills.
• Keywords can be a hassle to maintain when scaling a test
operation. You will need to continue building out the reposit-
ories and keyword tables.
Types of Framework
Hybrid Test Automation Framework
05
Hybrid Test Automation Framework

As the name suggests this framework is the combination of

one or more frameworks discussed above pulling from their
strengths and trying to mitigate their weaknesses. This hy-
brid test automation framework is what most frameworks
evolve into over time and multiple projects.
 Create a framework
Outline of the major steps in building Selenium framework

Java TestNG Architecture Component ExtentReport Build tools

Choose a program- Choose a unit test Design the framework Build the compo- Choose a reporting Integrate framework with
ming language framework architecture nent mechanism other tools
Choose a programming language
In what programming language do I want to write my tests?

The most popular languages among the Selenium com-

munity are Java, C#, Python and JavaScript. To decide
which programming language you should pick, consider the
below factors.
• What programming language is being used to develop
the web apps you need to test?
• Does your company have an in-house framework that
you can reuse?
Java is the safest choice if you start a new project from
scratch since it is widely adopted by the community due to
the fact that it works across platforms.
Moreover, you can easily find code examples or
troubleshooting tips if you get stuck. Java is also the top pri-
ority for each new release of Selenium.
Choose a programming language
In what programming language do I want to write my tests?

You can also write Selenium tests using the famous Beha-
vior-Driven Development (BDD) method.
In brief, BDD helps boost the readability of your tests by
structuring a test flow into Given, When, and Then (GWT)
statements.
As a result, not only test automation engineers with pro-
gramming skills but also domain experts and business test-
ers can understand the tests and contribute meaningfully to
the process of test creation, test result debugging, and test
maintenance.
Some tools that you can leverage if you choose BDD:
• Cucumber (support for most major languages)
• SpecFlow (for C#)
 Choose a unit test framework
Now we’ve selected the most suitable programming Its recommend TestNG since it offers several important be-
language, we now need to pick a unit test framework nefits, such as:
that we will build our framework upon • TestNG is similar to JUnit, but it is much more powerful
than JUnit—especially in terms of testing integrated
classes. And better yet, TestNG inherits all of the bene-
fits that JUnit has to offer.
Java • TestNG gives you the ability to write more flexible and

Python Junit, TestNG, JBehave powerful tests. Some of the highlight features are: easy
annotations, grouping, sequencing, and parameterizing.

Python

Java C#
PyTest, PyUnit

C#
MSTest, xUnit, NUnit
JS
JavaScript
JEST, Jasmine
Design the
Framework
architecture
Now, it’s time to take a look at our
framework’s architecture. After re-
ferring to many big and small Sel-
enium projects, come up with a
sustainable, maintainable, and
scalable architecture shown in the
diagram. Its highly recommend that
you follow this architecture or at
least the core principles behind it.

The beauty of this architecture comes

from the fact that there are two separate
components called
[1] SeleniumCore
[2] SeleniumTest
 Build the SeleniumCore component
SeleniumCore is designed to manage the browser instances as well as element interactions. This component helps you to create,
and destroy WebDriver objects.

One WebDriver object, as its name suggests, “drives” a

browser instance such as moving from web page to web
page. Ideally, the test writers should not care about how the
browser instances are created or destroyed. They just need
a WebDriver object to execute a given test step in their test
flow.

To achieve this kind of abstraction, we normally follow a best

practice called the Factory design pattern. Here is a class
diagram explaining how we use the Factory design pattern in
our framework.

In the diagram, LoginTest, LogoutTest and OrderTest are the

test classes that “use” the DriverManagerFactory to “manu-
facture” DriverManager objects for them.
 Build the SeleniumTest component
SeleniumTest component contains all test cases that use the classes provided by SeleniumCore. The design pattern we’ll apply here is called
PageObject pattern (POM).

SeleniumCore
(POM)has become the de-facto pattern used A simple Page object
in test automation frameworks because it re- • A constructor that receives a WebDriver object
duces duplication of code thus reduces the and sets its internal WebDriver object to that
test maintenance cost. object.
• The element locators that help the WebDriver
Applying POM means we’ll organize the UI object find the web elements you want to in-
elements into pages. A page can also include teract with. E.g. userNameTextBox
“actions” or business flows that you can per- • Methods to perform on the Login page such
form on the page. For instance, if your web as setUserName(), setPassword(),
app includes several pages called the Login clickLogin(), and most importantly–login()
page, Home page, Register page, etc., we’ll method that combines all of the three methods
create the corresponding PageObjects for above.
them such as LoginPage, HomePage, Re-
gisterPage, etc.

If the UI of any page changes, we will only

need to update the PageObject in question
once, instead of refactoring all tests that inter-
act with that page. SeleniumTest
 Overview
At this point, our Test Automation framework finally has a concrete foundation. We can now release it to the
team so that everybody will contribute to the test development and test execution efforts. We will now discuss
how to add some more utilities to the framework to increase our productivity.

Automation
Framework
SeleniumCore SeleniumTest
 Choose a reporting mechanism
At the end of the day, test automation will be useless if we cannot
Hopefully we now scale up our volume of automated tests get useful insights from the test results to take meaningful correct-
quickly and run them frequently enough to justify the upfront ive actions. There are a lot of options available out there for log-
investment. As you run more and more tests, you’ll soon find ging your automated tests.
that understanding test results will be difficult without a good Reporting mechanisms provided by testing frameworks such as
reporting mechanism. Junit and TestNG are often generated in XML format, which can
easily be interpreted by other software like CI/CD tools (Jenkins).
Unfortunately, those XMLs are not so easy to read for us human
beings.
10% Third party libraries such as ExtentReport and Allure can help
Passed you create test result reports that are human-readable. They also
include visuals like pie charts and screenshots.
20% Failed
There is an open-source Java reporting library called ReportNG.
It’s a simple HTML plug-in for the TestNG unit-testing framework
70% Skipped that provides a simple, color-coded view of the test results. The
sweet spot is: setting up ReportNG is very easy.

A good report should provide detailed information such as: the

amount of passed or failed test cases, pass rate, the execution
time, and the reasons why test cases failed.
 Decide how to implement CI/CD
Version control
To complete your Selenium framework, there are a few other All Automation teams must collaborate and share source code
areas of concern that you might want to tackle. with each other. Just like a software development project, source
code of the tests and test utilities are stored in a source control
Build tools and dependency managers:
system, also known as a version control system. Popular source
Dependency managers help you manage the dependencies control systems are GitHub, Bitbucket, and TFS. However, its
and libraries that the framework is using. Examples of these recommend that your team set up an in-house source control
tools include Maven, Gradle, Ant, NPM, and NuGet. system using Git if you don’t want to share your source code with
Invest in a dependency manager to avoid missing dependen- the public.
cies when you build your framework CI/CD integration
Build tools assist you in building the source code and depend- Popular CI systems include Jenkins, Bamboo, and TFS. In
ent libraries, as well as in running tests. the world of ever-increasing demand on agility, you will soon
find it useful to integrate your automated tests into DevOps
pipelines so that your organization can speed up delivery and
stay competitive. Its recommend Jenkins since it’s free and
very powerful.
Integrate your framework with other tools

AutoIt is a freeware BASIC-like scripting language de-

Test Planning
Analyze

signed for automating the Windows GUI and general

scripting. It will help you in case you want to work with
desktop GUI, like the download dialog of the browser.

Jira is a famous eco-system for software development

Test case devel-
opment and testing. Thus, consider integrating with Jira in some
common scenarios such as automatically posting and
closing Jira bugs according to Selenium test results.
Test Execution
 Queries
Anything I can assist you with !!
Thank You
PRESENTATION BY ANIL SHEORAN