SOFTWARE TESTING

Module Number: 05

Module Name: Test Automation

Version Code:ST1
Released Date:31-Jan-2019 1
Test Automation

AIM:
To familiarise students about the automation of Software Testing.

2
Test Automation

Objectives:
The Objectives of this module are to understand:
• The key concepts of software test automation.
• The design and architecture for automation.
• Requirements for a test tool.
• The challenges in automation.
• Test metrics and measurements.
• The progress and productivity metrics.

3
Test Automation

Outcomes:
At the end of this module, you are expected to:
• Explain the need for automation.
• Describe the design and architecture for automation.
• Explain how to deal with challenges in automation.
• Elaborate Test metrics and measurements.
• Describe about the progress and productivity metrics.

4
Test Automation

Content
• Software Test Automation
• Design and architecture for automation
• Requirements for a test tool
• Challenges in automation
• Test metrics and measurements.
• Progress and productivity metrics

5
Test Automation

Software Test Automation

6
Test Automation

Software Test Automation

What is Automation Testing?
• Manual Testing is performed by a human sitting in front of a computer carefully executing the
test steps.
• Automation Testing means using an automation tool to execute your test case suite.

7
Test Automation

Software Test Automation

• The automation software can also enter test data into the System Under Test, compare expected
and actual results and generate detailed test reports.

• Test Automation demands considerable investments of money and resources.

8
Test Automation

Software Test Automation

• Using a test automation tool, it is possible to record this test suite and re-play it as required.
• Once the test suite is automated, no human intervention is required.
• The goal of Automation is to reduce the number of test cases to be run manually and not to
eliminate Manual Testing altogether.

9
Test Automation

Software Test Automation

Why Automated Testing?

Automated software testing is important due to following reasons:

• Manual Testing of all workflows, all fields, all negative scenarios is time and money consuming.

• It is difficult to test for multilingual sites manually.

10
Test Automation

Software Test Automation

• Automation does not require Human intervention. You can run automated test unattended (overnight).

• Automation increases the speed of test execution.

• Automation helps increase Test Coverage.

• Manual Testing can become boring and hence error-prone.

11
Test Automation

Software Test Automation

Automated Testing Process:

12
Test Automation

Software Test Automation

Following steps are followed in an Automation Process:
• Test tool selection:
Test Tool selection largely depends on the technology the Application Under Test is built on.
For instance, Quick Test Professional (QTP) does not support Informatica. So, QTP cannot
be used for testing Informatica applications.

13
Test Automation

Software Test Automation

• Define the scope of Automation:

The scope of automation is the area of your Application Under Test which will be automated.

Following points helps to determine scope:

• The features that are important for the business.

• Scenarios which have a large amount of data.

• Common functionalities across applications.

14
Test Automation

Software Test Automation

• Technical feasibility.
• The extent to which business components are reused.
• The complexity of test cases.
• Ability to use the same test cases for cross-browser testing.

15
Test Automation

Software Test Automation

• Planning, Design, and Development

During this phase, you create Automation strategy and plan, which contains the following details:

• Automation tools selected.

• Framework design and its features.

• In-Scope and Out-of-scope items of automation.

16
Test Automation

Software Test Automation

• Automation tested preparation.
• Schedule and Timeline of scripting and execution.
• Deliverables of Automation Testing.

17
Test Automation

Software Test Automation

• Test Execution

• Automation Scripts are executed during this phase. The scripts need input test data before
there are set to run. Once, executed they provide detailed test reports.

• Execution can be performed using the automation tool directly or through the Test
Management tool which will invoke the automation tool.

18
Test Automation

Software Test Automation

• Example: Quality centre is the Test Management tool which in turn will invoke QTP for execution
of automation scripts. Scripts can be executed in a single machine or a group of machines. The
execution can be done during the night, to save time.

19
Test Automation

Software Test Automation

• Maintenance
As new functionalities are added to the System Under Test with successive cycles, Automation
Scripts needs to be added, reviewed and maintained for each release cycle. Maintenance becomes
necessary to improve the effectiveness of Automation Scripts.

20
Test Automation

Software Test Automation

Benefits of Automation Testing
Following are benefits of automated testing:
• 70% faster than the manual testing.
• Wider test coverage of application features.
• Reliable in results.

21
Test Automation

Software Test Automation

• Ensure Consistency.
• Saves Time and Cost.
• Improves accuracy.
• Human Intervention is not required while execution.

22
Test Automation

Software Test Automation

• Increases Efficiency.
• Better speed in executing tests.
• Re-usable test scripts.

23
Test Automation

Software Test Automation

• Test Frequently and thoroughly.

• More cycle of execution can be achieved through automation.

• Early time to market.

24
Test Automation

Self Assessment Question

1. There are __________ phases in Automated Testing Process.

a. 1

b. 2

c. 4

d. 5

Answer: 5

25
Test Automation

Software Test Automation

Which Test Cases to Automate?
Test cases to be automated can be selected using the following criterion to increase the automation.
• High Risk - Business Critical test cases.
• Test cases that are repeatedly executed.

26
Test Automation

Software Test Automation

• Test Cases that are very tedious or difficult to perform manually.

• Test Cases which are time-consuming.

27
Test Automation

Software Test Automation

The following category of test cases are not suitable for automation:
• Test Cases that are newly designed and not executed manually at least once.
• Test Cases for which the requirements are frequently changing.
• Test cases which are executed on an ad-hoc basis.

28
Test Automation

Software Test Automation

Different types of software testing that can be automated are:
• Smoke Testing
• Unit Testing
• Integration Testing

29
Test Automation

Software Test Automation

• Functional Testing
• Regression Testing

30
Test Automation

Software Test Automation

• Data Driven Testing
• Black Box Testing

31
Test Automation

Skill needed for automation and types of tools

What are the skills needed for Automated Testing?
The skills required depends on what generation of automation the company is in.
1. Capture / playback and test harness tools (first generation).
2. Data driven tools (second generation).
3. Action driven (third generation).

32
Test Automation

Skill needed for automation

1. Capture / Playback and Test Harness Tools:
One of the most boring and time-consuming activity during testing life cycle,
is to rerun manual tests number of times.
Here, capture/playback tools are of great help to the testers.
These tools do this by recording and replaying the test input scripts.
As a result, tests can be replayed without attendant for long hours specially
during regression testing.
.
33
• Also, these recorded test scripts can be edited as per need i.e., whenever
changes are made to the software.
• These tools can even capture human operations e.g., mouse activity,
keystrokes etc.
• A capture / playback tool can be either intrusive or non-intrusive.
• Intrusive capture / playback tools are also called as native tools, as they
along with Software-under-test (SUT), resides on the same machine

34
Test Automation

Skill needed for automation

Non-intrusive capture / playback tools on the other hand, reside on the separate machine and is
connected to the machine containing software to be tested using special hardware.
One advantage of this tool, is to capture errors which users frequently make and which
developers cannot reproduce.
Test harness tools are the category of capture / playback tools used for capturing and replaying
sequence of tests.
These tools enable running large volume of tests unattended and help in generating reports by
using comparators. These tools are very important at CMM level – 2 and above.

35
Test Automation

Skill needed for automation

2. Data-driven Tools:
This method helps in developing test scripts that generate the set of input conditions and
corresponding expected output.
The approach takes as much time and effort as the product.
However, changes to application do not require the automated test cases to be changed as long as
the input conditions and expected output are still valid.
This generation of automation focuses on input and output conditions using the black box testing
approach.

36
Test Automation

Skill needed for automation

3. Action-driven Tools:
This technique enables a layman to create automated tests.
There are no input and expected output conditions required for running the tests.
All actions that appear on the application are automatically tested, based on a generic set of controls
defined for automation.
From the above approaches / generations of automation, it is clear that different levels of skills are
needed based on the generation of automation selected.

37
Test Automation

Scope of automation
• Automation is the process of evaluating the AUT (Application Under Test) against the specification
with the help of a tool.
Depending on the nature of testing, there are two main branches under automation.
• Functional testing with automation.
• Performance testing with automation.

38
Test Automation

Scope of automation
• Functional Testing with the help of Automation
Functional automated testing has emerged as a key area in most of the testing processes.
The main area, where the functional testing tools are used is for regression test case
execution.
Mostly in the agile scrum methodology where frequent releases are happening, it is almost
impossible to execute all the regression test cases manually within short span of time.

39
Test Automation

Scope of automation
Automation gives a high ROI (Return of Investment) in this area since, it is a one time effort for
generating the scripts.

Maintenance to the existing scripts will be considerably less, if the automation follows a good
framework which exactly suits the nature of application.

It is advisable to keep at least 60 to 70% of regression cases to be automated for being adherent to
the timelines of test case execution.

Automating test cases is not desirable if it is a onetime testing.

40
Test Automation

Scope of automation
Performance Testing with the help of Automation

• Performance testing is the process of evaluating the application performance which is being
a critical requirement of every application nowadays.

• Performance testing is almost impossible by manual means.

• There are different tools used across organisations for evaluating application performance.

• There are different divisions under performance testing based on the nature of testing.

41
Test Automation

Scope of automation
Load and Stress testing

• Load testing is performed to confirm that application can withstand the maximum load which
is mentioned in the SRS.

• Here, the application will be tested with the specified load without overloading the application.

• But in stress testing, the intention is to find out the maximum load the application can
accommodate.

• Application will be tested with more load than the specified limit in order to find out the break
point.

42
Test Automation

Scope of automation
• For load and stress testing, large number of concurrent users is needed and multiple computers are
required.
• In manual testing it is difficult and almost impossible to run the severe load for several days to identify
problems in the application.
• While using automation, the testing software (tool) monitors the CPU and memory usage of each test
computer.
• The tool generates artificial loads by creating virtual users and which will produce a guaranteed test
accuracy.
• The tools can generate number of virtual users on a single computer.
• Automation helps us to find out defects as we can run the test using multiple users, multiple environments
on several days.

43
Test Automation

Scope of automation
• Automation is an effective means of eliminating/reducing manual effort during regression and
functional test case execution.

• Also, the chances of defect escape will be reduced considerably, since human mistakes will not
occur once the script is fully developed.

• Hence, the automation is more economical considering the fact of time and quality.

44
Test Automation

Self Assessment Question

2. White Box testing is developer friendly. State whether True or False.

a. True
b. False

Answer: True

45
Test Automation

Design and architecture for automation

Test Automation Framework Architecture
Test automation framework architecture efforts are often complete failures.
It is true.
There are many companies, who have given up on creating a good test automation framework
architecture, because after investing a large amount of time and money and resources in doing it
the wrong way, they have incorrectly assumed the entire effort is not cost effective.

46
Test Automation

Design and architecture for automation

• Test automation framework architecture basics
Let us start off by talking about what the goals of a successful test automation framework architecture
should be.
There are two goals creating a test automation framework:
• Having the ability to easily create simple automated tests that use the framework.
• Decoupling cosmetic application changes from the tests, so that when the application changes,
the tests do not all have to be updated.

47
Test Automation

Design and architecture for automation

• A good test automation framework architecture should at the very least provide these two
important services.
• We create a test automation framework to allow us to support the creation of tests and to
keep those tests from being dependent on the actual UI of our application—as much as
possible.

48
Test Automation

Design and architecture for automation

• A simple test automation framework architecture
Once, you understand the two very important goals and why they are important, it is easier
to think about how you should design a test automation framework architecture.

49
Test Automation

Design and architecture for automation

50
Test Automation

Design and architecture for automation

Here, you can see that there are four layers to test automation framework architecture.
First, we have the browser layer or the web app itself.
This just represents your actual application.
Next, we have the Selenium or web driver layer.
This layer represents your browser automation tool.
Selenium is basically just a framework for automating a browser.
It does not know anything about testing, it is an API that lets you interact with the browser
programmatically.

51
Test Automation

Design and architecture for automation

• After that, we have the framework layer.
• This is the actual framework you create which uses Selenium, or whatever web driver you want,
to actually automate your application.
• The framework is acting as an abstraction between your application and the tests.
• The framework knows about the UI of your application and how to interact with it using
Selenium.
• It is your job to create this layer.

52
Test Automation

Design and architecture for automation

• Finally, we have the actual tests.
• The tests use the framework to manipulate your application and check the state of the application.
• These tests should be simple to understand and should not have to know anything about the
actual implementation of the UI of your application.
• These tests should rely on the framework to give them the ability to do whatever they need to do
in your application.

53
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
Step 1: Understand your project requirements thoroughly:
• Maintaining the quality of an application by delivering a bug-free product is crucial for the
success of any project.
• Automated testing can help improve the quality of a project and increase scope and depth
of the tests.

54
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
(Continued) Step 1: Understand your project requirements thoroughly:
• For this, get a deep understanding of your project requirements such as:
• Project type (web/desktop/mobile).
• Scope of the project.
• Existing team’s strength on code language before you start the process.
• There is no such tool that is good or bad, but the ROI of any tool relies on the need, i.e., what
precisely does one need to automate and what amount of test cases needs automation?

55
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
Step 2: Consider your existing test automation tool as a benchmark :
• It is considered that Selenium Test Automation Tool as a benchmark to evaluate and
determine the best automation tool for their project.
• The advantages and drawbacks of Selenium Web Driver framework need to be understood
before evaluating other tools.
• Selenium is a free, open source tool available to test web applications and websites.

56
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
(Continued) Step 2: Consider your existing test automation tool as a benchmark :
• It provides record-playback tool to produce and record playback test scripts. It provides
language support for C# Ruby, Java, JavaScript, Python, and Node.js but lacks proper
customer support services.
• Testing teams leverage Selenium, if they are comfortable with coding techniques and
they test web applications through the UI.

57
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
(Continued) Step 3: Identify the key criteria suitable for a project
There are many key points one should consider before deciding the best-fit automation tool for a
project.
Here, is the list of few key criteria that was used to evaluate the best tool in our case which is shown
in the next slide.

58
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
(Continued) Step 3: Identify the key criteria suitable for a project
• Ease of Developing and Maintaining the Scripts: Development and maintenance of test
scripts should be as simple as possible to decrease the human and time resource utilisation.
• Ease of Test Execution for Non-Technical user: The test suite execution should be
simple for any project member to run them easily as and when required. Also, it should be
easy for manual testers who have very little or no technical knowledge.

59
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
(Continued) Step 3: Identify the key criteria suitable for a project
• Support to Web, Desktop and Mobile application: Leveraging 3 different tools for 3 types
of platforms for test automation is a complicated task to handle. It is better to select a tool
that supports all the three platforms.
• Intuitive Test Report: Test reports build confidence and so the reports need to be intuitive
and simple for the management team to understand easily.

60
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
(Continued) Step 3: Identify the key criteria suitable for a project
• Cross Browser Testing: Support to Cross browser testing is a must when there are multiple
end-users and no particular browser restriction.
• Support to Keyword and Data Driven Testing: Keyword driven testing acts as an
extension to the data driven testing framework. When a project becomes complex, test
framework needs to be extended.

61
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
(Continued) Step 3: Identify the key criteria suitable for a project
• Technical Support and Assistance: Automation Engineers definitely need help while
handling critical problems of a project. The tool that provides technical support and
assistance would be of great help.
• Language Support like C#, Java, Python and others: Every test scenario cannot be
recorded. In some cases, the tester must write the code. So, the tool that supports the required
language to write the customised scripts would be helpful.

62
Test Automation

Requirements for a test tool

Simple steps to select the right test automation tool for your project:
(Continued) Step 3: Identify the key criteria suitable for a project
• Integration with builds: Support to integrate with Continuous Integration tools for
automated builds and deployments is necessary.
• Pricing: Depending on the above qualities and project cost estimates, evaluate the cost
difference between the other available automation tools.

63
Test Automation

Self Assessment Question

3. Which of the following is the step to Select the Right Test Automation tool for your Project?
i. Identify the key criteria suitable for a project.
ii. Consider your existing test automation tool as a benchmark.
iii. Understand your project requirements thoroughly.

a. Only i
b. Only ii
c. Only i and ii
d. All i, ii and iii

Answer: All i, ii and iii 64

Test Automation

Challenges in automation
Here, we have listed out some of the most common challenges that a team faces, as they pursue
automation efforts—and also how to overcome these challenges.
1. Effective Team Collaboration
2. Deciding on the best testing approach
3. Evaluating and choosing good tools
4. Training or acquiring skilled testers
5. Significant capital investment

65
Test Automation

Challenges in automation
1. Effective Team Collaboration:
• Collaboration is a challenge for any product development group—developers, test automation
engineers, and conventional testers alike.
• The problem is more acute with test automation because, it requires considerably more
planning, foresight, communication, and collaboration, as it fully depends on historical testing
data, tester experience, and at least one proof-of-concept.
• Testers need to work well with developers, customers, and one another to get all these pieces
together.

66
Test Automation

Challenges in automation
1. Effective Team Collaboration:
• How to overcome it?
• Since, testers will be central in communicating the value of automation testing to senior
management, each automation tester should be in close contact with manual testers, business
stakeholders, developers, and system architects.
• To be successful, it is vital that most of the team commits to active and continuous
involvement because, of all the moving pieces coming in from the different departments. You
will also need management buy-in prior to automating your tests.

67
Test Automation

Challenges in automation
2. Deciding on the best testing approach:
• When possible, it is always best to prioritise meeting this challenge before buying new test
automation tools. Think carefully on this one. You might acquire good tools—but which
tests should you automate first, which one should you leave as manual tests, and how will
you go about doing it all? The tools will not give you insight into what your team should
automate.
• The tools may also be focused on one area when you need help with another. Beware that
you will not be able to automate all of your test cases, particularly in areas such as
compatibility or recovery.

68
Test Automation

Challenges in automation
2. Deciding on the best testing approach:
How to overcome it?
• Which test cases are important to you? Before you begin to think about test
automation, it’s important to define a set of criteria that each test case must satisfy to
be a candidate for automatisation.
• It is very important to focus on areas that contain unknowns that can mean higher risk
to product success. It is also important to give serious attention to complex functional
areas that are especially useful or interesting to customers.

69
Test Automation

Challenges in automation
2. Deciding on the best testing approach:
How to overcome it?
• Risk-focused test automation ascribes a higher priority to testing the elements that are
most likely to fail—and hold the potential for the most serious negative consequences in
the event of a failure in production.
As you assess the level of risk, it is critical to keep the following in mind:
• Financial impact of a potential failure.
• Probability of failure for each vulnerability (get developer input on these).
• Service-level agreements with clients and customers.
• Human and equipment safety issues.
70
Test Automation

Challenges in automation
3. Evaluating and choosing good tools:
• Many record and play testing tools are available, all trying to alleviate the same problem
of automating manual tests, but have proven to be harmful to team's productivity levels.
• This makes it difficult to sift through the chaff and find tools that will actually improve
software testing and meet real needs.
• It is important to steer clear of hype, do sufficient research, and carefully evaluate
fact-based proposals of reputable tool vendors.

71
Test Automation

Challenges in automation
3. Evaluating and choosing good tools:
• Tool evaluation can seem unachievable for various reasons:
• Lack of expertise needed to make the most of the capabilities of a specific tool.
• Indefinite knowledge on whether a tool exists to meet specific needs.
• Misconceptions lead to desire for a tool that is not available.
• A favorite tool does not seem to provide full test coverage.
• Tool costs exceed the team budget.
• The challenge of choosing good automation tools is a serious problem since, according to
the 2018 Gemini World Quality Report.

72
Test Automation

Challenges in automation
3. Evaluating and choosing good tools
• How to overcome it?
• Though it requires effort from the most of the team, a good assessment begins with
clearly outlining a set of tool requirements criteria.
• These definitions should correspond well with user acceptance criteria for each of
your products, and also reflect the input from professionals who have considerable
experience in using the tools. If they are not on your team, you may be able to find
these experts at conferences or smaller-scale networking events, making it worth
investing in sending your teams to them.

73
Test Automation

Challenges in automation
3. Evaluating and choosing good tools:
• How to overcome it?
• Free webinars are often hosted by product vendors to demo their offerings without the
pressure of being in a sales pipeline.
• Encourage your testing team to register to attend these webinars live, as the live Q&A
portion can be very valuable to take part in.
• If the tool that your team wants is beyond the reach of your budget, it may be worth the
effort to prepare a cost-benefit analysis and make your case to management.
• Demonstrate how implementing this tool will minimise risk and improve release quality,
using a pilot scenario that scales.
74
Test Automation

Challenges in automation
4. Training or acquiring skilled testers:
• It is a misconception that test automation can be easily done by any tester.
• Likewise, many falsely believe that any developer can automate tests.
• Focusing, designing, configuring, and implementing test automation requires specific
skills and time to build experience.

75
Test Automation

Challenges in automation
4. Training or acquiring skilled testers:
• How to overcome it?
• When filling this role, you need to look for a specific skill set beyond technical chops.
• Testers sit at the intersection between your developers, managers, and customers, so
having the ability to articulate ideas from one group to another is a must.
• Being in the tech field, testers must be willing to stay up to date with the latest
technologies and development methodologies to understand what the development team
may be doing or moving towards.
• At the same time, they must also understand how to keep a holistic view of what the
software in development is trying to achieve from a business perspective, and keep test
cases relevant to those goals. 76
Test Automation

Challenges in automation
5. Significant capital investment :
• In nearly all cases, the initial phase of test automation comes with some costs, since it
requires analysis, design, and planning for the implementation.
• You will need to account for all licensing and operating costs—including hardware and
software.
• Even if the choice is made to go with free or low-cost open-source automation tools, the time
spent on training and maintenance can have significant costs in terms of employee focus.

77
Test Automation

Challenges in automation
5. Significant capital investment :
• How to overcome it?
• Bite the bullet and then plan it out.
• Automating is not a movement that should be approached haphazardly. When done
properly, the effort and cost can pay steady dividends well into the future.
• Test automation can deliver significant gains in quality, accuracy, and productivity.
• The business case can be built solidly when the team comes to realise that preventing all
the negative impact and cost of a single major defect in production can directly
compensate for all of the tangible and intangible costs of pursuing test automation.

78
Test Automation

Self Assessment Question

4. Which of the following falls under challenges that teams face as they pursue automation efforts?
i. Effective Team Collaboration.
ii. Deciding on the best testing approach.
iii. Evaluating and choosing good tools.

a. Only i
b. Only ii
c. Only i and ii
d. All i, ii and iii

Answer: All i, ii and iii 79

Test Automation

Test metrics and measurements

Software Testing Metrics:
• A metric is a quantitative measure of the degree to which a system, system component,
or process possesses a given attribute.
• Metrics can be defined as “STANDARDS OF MEASUREMENT”
• Software Metrics are used to measure the quality of the project.
• It is used for describing an attribute.
• Metric is a scale for measurement.

80
Test Automation

Test metrics and measurements

Software Testing Metrics:
• Suppose, in general, “Kilogram” is a metric for measuring the attribute “Weight”. Similarly, in
software, “How many issues are found in thousand lines of code?”, Here, No. of issues is one
measurement and No. of lines of code is another measurement. Metric is defined from these two
measurements.
• Test metrics example:
• How many defects exist within the module?
• How many test cases are executed per person?
• What is the Test coverage %?

81
Test Automation

Test metrics and measurements

Software Test Measurement
• Measurement is the quantitative indication of extent, amount, dimension, capacity, or size
of some attribute of a product or process.
• Test measurement example:
• Total number of defects.

82
Test Automation

Test metrics and measurements

Software Test Measurement
• Measurement vs. Metrics with proper diagram

Measure

Kilometer Hours Meters Seconds

Metrics

Km/hr m/s

83
Test Automation

Test metrics and measurements

Metrics Life Cycle:
• Analysis
• Communicate
• Evaluation
• Report

84
Test Automation

Test metrics and measurements

Metrics Life Cycle:
• Analysis:
1. Identification of the metrics.
2. Define the identified metrics.

85
Test Automation

Test metrics and measurements

Metrics Life Cycle:
• Communicate:
1. Explain the need of metric to stakeholder and testing team.
2. Educate the testing team about the data points need to be captured for processing the metric.

86
Test Automation

Test metrics and measurements

Metrics Life Cycle:
• Evaluation:
1. Capture and verify the data.
2. Calculating the metric(s) value using the data captured.

87
Test Automation

Test metrics and measurements

Metrics Life Cycle:
• Report:
1. Develop the report with effective conclusion.
2. Distribute report to the stakeholder and respective representatives.
3. Take feedback from stakeholders.

88
Test Automation

Test metrics and measurements

Types of Manual Test Metrics:
• Testing Metrics are mainly divided into 2 categories:
• Base Metrics
• Calculated Metrics

89
Test Automation

Test metrics and measurements

Types of Manual Test Metrics:
• Base Metrics:
• Base Metrics are the Metrics which are derived from the data gathered by the Test Analyst
during the test case development and execution.
• This data will be tracked throughout the Test Lifecycle. i.e., collecting the data like Total no. of
test cases developed for a project (or) no. of test cases need to be executed (or) no. of test cases
passed/failed/blocked etc.

90
 Examples
• 1. Number of test cases
• 2. Number of passed, failed, and blocked test cases.
• 3. Total number of defects and critical issues reported, accepted,
rejected, and deferred.
• 4. Number of planned and actual test hours.
• 5. Number of bugs discovered after shipping.
• 6. Number of critical issues reported
• 7. Number of defects accepted.
• 8. Number of defects rejected
• 9. Number of defects deferred.
91
Test Automation

Test metrics and measurements

Types of Manual Test Metrics:
• Calculated Metrics:
• Calculated Metrics are derived from the data gathered in Base Metrics.
• These Metrics are generally tracked by the test lead/manager for Test Reporting purpose.

92
Exaamples

93
94
95
96
Test Automation

Project metrics:
• These are metrics that pertain to Project Quality.
• They are used to quantify defects, cost, schedule, productivity and estimation of various project
resources and deliverables.
Example:
• Schedule: Activities, Milestones, Reviews, Commitments, and Completion dates.
• Risk: Schedule, Cost, Resources, and Deliverables.
• Quality: Issues, defects and rework.
• Resource: Response times, Capacity / load , Forecast / Actual Usage.

97
Test Automation

Productivity metrics
• A quality metric should be a predictor of product quality.
• Classes of product metrics:
• Static metrics:
• Are collected by measurements made of the system representations.
• Help assess complexity, understandability and maintainability.
• Dynamic metrics:
• Dynamic metrics which are collected by measurements made of a program in execution.
• Dynamic metrics help assess efficiency and reliability.

98
Test Automation

Productivity metrics
• Static metrics:
• Have an indirect relationship with quality attributes.
• You need to try and derive a relationship between these metrics and properties such as
complexity, understandability and maintainability.
• Dynamic metrics:
• They are closely related to software quality attributes.
• It is relatively easy to measure the response time of a system (performance attribute) or
the number of failures (reliability attribute).

99
Test Automation

Self Assessment Question

5. Which of the following falls under metric life cycle?
i. Analysis
ii. Communicate
iii. Evaluation

a. Only i
b. Only ii
c. Only i and ii
d. All i, ii and iii

Answer: All i, ii and iii 100

Test Automation

Summary
• Project Metrics are used to quantify defects, cost, schedule, productivity and estimation of various
project resources and deliverables.
• Manual Testing is performed by a human sitting in front of a computer carefully executing the test
steps.
• The automation software can also enter test data into the System Under Test, compare expected and
actual results and generate detailed test reports.
• The goal of Automation is to reduce the number of test cases to be run manually and not to eliminate
Manual Testing altogether.
• Performance testing is the process of evaluating the application performance which is being a critical
requirement of every application nowadays. Performance testing is almost impossible by manual
means. 101
Test Automation

Assignment
1. What skills does every software tester should have?
2. Present a real life scenario on scope of automation.
3. Case study on the design and architecture for automation.
4. Discuss the steps to select the right test automation tool for your project in detail.
5. Discuss the most common challenges that teams face as they pursue automation efforts and also how
to overcome these challenges in detail.
6. Explain software test metrics in detail with proper diagram.
7. Give a comparative study on measurement vs. Metrics with proper diagram.
8. Discuss metrics life cycle in detail.
9. Discuss the types of manual test metrics in detail.
10. Discuss project and productivity metrics in detail. 102
Test Automation

Document Links

Topics URL Notes

Software test https://www.guru99.com/automation-testin This link gives an introduction to
automation g.html Software Test Automation.
Skills needed for https://www.softwaretestinggenius.com/wh This link explains skills needed for
automation at-skills-are-needed-for-automated-testing/ automation.

https://www.zyxware.com/articles/3533/sco This link explains about scope of

Scope of automation
pe-of-automation-in-testing automation.

Design and architecture http://softwaretestingfundamentals.com/sof This link explains about design and
for automation tware-testing-basics architecture for automation.

103
Test Automation

Document Links (Cont…)

Topics URL Notes

Requirements for a test https://www.saviantconsulting.com/blog/4 This link gives the requirements for a
tool -steps-select-test-automation-tool.aspx test tool.
This link explains challenges in
Challenges in https://www.mabl.com/blog/top-5-challen
automation.
automation ges-of-automation-testing

Test metrics and https://www.softwaretestinghelp.com/soft This link will explains Test metrics and
measurements ware-test-metrics-and-measurements/ measurements.

Project metrics https://www.simplilearn.com/project-and- This link explains Project metrics.

process-metrics-article

104
Test Automation

Document Links (Cont…)

Topics URL Notes

https://www.slideshare.net/anandsubraman This link explains about the
Project metrics
iam/project-metrics-measures Project metrics
This link explains Productivity
Productivity https://www.cs.ucy.ac.cy/~cs00pe/epl603/l
metrics
metrics ectures/Lect19-20.pdf

105
Test Automation

Video Links
Topics URL Notes
Software test This video explains about
https://www.youtube.com/watch?v=RbSlW8jZFe8
automation Software Test Automation.
Skill needed for This video explains about the skill
https://www.youtube.com/watch?v=0zXs6EXccvo
automation needed for automation.
Scope of This video describes the scope of
https://www.youtube.com/watch?v=kv-JZAbRess
automation automation.
Design and
This video is about Design and
architecture for https://www.youtube.com/watch?v=y30Rz-w1fPU
architecture for automation.
automation

10
6
Test Automation

E-Book Links

Topics URL
Srinivasan Desikan and
Gopalaswamy Ramesh, https://www.oreilly.com/library/view/software-testing-principles/978
“Software Testing – 8177581218/xhtml/chapter010.xhtml#ch10.1
Principles and Practices”,
Pearson Education, 2006
Naresh Chauhan, Software
https://access.redhat.com/documentation/en-us/red_hat_enterprise_l
Testing Principle and
inux/7/html/storage_administration_guide/index
Practices, Oxford
University Press.
Ron Patton, “Software
https://shekharsk.files.wordpress.com/2016/01/ron-patton-software-t
Testing”, Second Edition,
esting.pdf
Sams Publishing, Pearson
Education, 2007.

107