UNIT V

TEST MANAGEMENT AND AUTOMATION

1. Test Planning

2. Management

3. Execution and Reporting

Software Test Automation:

4. Scope of automation,

5. Design and Architecture for automation

6. Generic requirements for test tool framework

7. Test tool selection

 TEST MANAGEMENT AND AUTOMATION
 Test Management is a series of planning, execution, monitoring and control activities that help
to achieve project goals.
 Automation Testing or Test Automation is a software testing technique that performs using
special automated testing software tools to execute a test case suite.
 On the contrary, Manual Testing is performed by a human sitting in front of a computer
carefully executing the test steps.
TEST PLANNING
 A Test Plan is a detailed document that describes the test strategy, objectives, schedule,
estimation, deliverables, and resources required to perform testing for a software product.
 Test Plan helps us determine the effort needed to validate the quality of the application under
test.
 The test plan serves as a blueprint to conduct software testing activities as a defined process,
which is minutely monitored and controlled by the test manager.
“Test Plan is A document describing the scope, approach, resources, and schedule of intended test
activities.”
A test plan in software testing is a document which outlines the what, when, how, who, and more of a
testing project.
It contains the details of what the scope of testing is, what the test items are, who will do which testing
task, what the items test/pass criteria will be, and what is needed to set up the test environment and
much more.

Test Planning

 The test plan acts as the anchor for the execution, tracking and reporting of the entire testing
project and covers
1. What needs to be tested.
2. How the testing is going to be performed.
3. What resources are needed for testing
4. The time lines by which the testing activities will be performed.
5. Risks that may be faced in all of the above, with appropriate mitigation and contingency plans.
  Scope Management: Deciding features to be Tested / Not Tested.
Scope management pertains to specifying the scope of a project.
For testing, scope management entails.
1. Understanding what constitutes a release of a product;
2. Breaking down the release into features;
3. Prioritizing the features for testing;
4. Deciding which features will be tested and which will not be; and
5. Gathering details to prepare for estimation of resources for testing.
Features to be tested:
• Features that are new and critical for the release.
• Features whose failures can be catastrophic.
• Features that are expected to be complex to test.
• Features which are extensions of earlier features that have been defect prone.
• Given the limitations on resources and time, it is likely that it will not be possible to test all the
combinations exhaustively.

• During planning time a test manager should also consciously identify the features or
combinations that will not be tested.

Deciding Test Approach/ Strategy:

This test approach or strategy should result in identifying the right type of test for each of the
features or combinations.
This includes identifying
1. What type of testing would you use for testing the functionality?
2. What are the configurations or scenarios for testing the features?
3. What integration testing would you do to ensure these features work together?
4. What localization validations would be needed?
5. What non- functional tests would you need to do?

Setting up criteria for testing:

There must be entry and exit criteria for different phases of testing.
 The entry criteria for a test specify threshold criteria for each phase or type of test.
 There may also be entry criteria for the entire testing activity to start.
 The completion/exit criteria specify when a test cycle or a testing activity can be deemed
complete.
Identifying responsibilities, Staffing and Training needs:
 A testing project requires different people to play different roles.
 There are the roles of test engineers, test leads, and test managers.
 Staffing is done based on estimation of effort involved and the availability of time for release.
 Trainings are required to find the perfect fit between the requirements and the skills available.
 Identifying Test Deliverables:

The deliverables include the following:

1. The test plan itself ( master test plan, and various other test plans for
the project)
2. Test case design specifications.
3. Test cases, including any automation that is specified in the plan
4. Test logs produced by running the tests.
5. Test summery Reports.

Testing Tasks: Size, Effort and Schedule Estimation:

 Size estimation quantifies the actual amount of testing that needs to be

done.
 Effort estimate is given in person days, person months, or person years.
 Schedule estimation entail translating the effort required into specific time
frames.

Communications Management:

Communications management consists of evolving and following procedures for communication

that ensure that everyone is kept in sync with the right level of detail.
Since this is intimately connected with the test execution and progress of the testing project.

Risk Management:

Risk: Risk is the probability of occurrence of an undesirable event.

Like other projects, testing projects also face risks.

What is Risk Management?
Risk management is the process of identifying, assessing, and prioritizing the risks to minimize, monitor,
and control the probability of unfortunate events.
Risk management entails:
1. Identifying the possible risks
2. Quantifying the risks
3. Planning how to mitigate the risks
4. Responding to risks when they become a reality
 Risk Identification: Consists of identifying the possible risks that ay hit a project.
Common ways to identifying risks.
1. Use of checklists
2. Use of organizational history and metrics
3. Informal networking across the industry
Risk Quantification: Deals with the expressing the risk in numerical terms.
Two components to the quantification of risk.
 One is the Probability of the risk happening.
 Impact of the risk.
Risk mitigation planning: Deals with identifying alternative strategies to combat a risk event, should
that risk materialize.
When the above 3 steps are carried out systematically and in a timely manner, the organization would
be in a better position to respond to the risks, should the risks become a reality.
2. TEST MANAGEMENT
• Choice of standards.
• Test Infrastructure Management.
• Test People Management.
• Integrating with Product Release.

3. TEST EXECUTION AND REPORTING

Test Execution Phase

Test Execution Phase is carried out by the testers in which testing of the software build is done based on
test plans and test cases prepared.

 The process consists of test script execution, test script maintenance and bug reporting.
 If bugs are reported then it is reverted back to development team for correction and retesting
will be performed.

Test Execution Activities

• Execute tests as per plan.

• Document test results, and log defects for failed cases.
• Map defects to test cases in RTM.
• Retest the Defect fixes.
• Track the defects to closure.
Deliverables of Test Execution:
• Completed RTM with the execution status.
• Test cases updated with results.
• Defect reports.
Test Report

Test Report is a document which contains a summary of all test activities and final test results of a
testing project.
• Test report is an assessment of how well the Testing is performed.
• Based on the test report, stakeholders can evaluate the quality of the tested product and make
a decision on the software release.

The typical benefits of a test report include:

What does a test report contain?

 Project Information
All information of the project such as the project name, product name, and version should be described
in the test report.
 Test Objective
As mentioned in Test Planning tutorial, Test Report should include the objective of each round of
testing, such as Unit Test, Performance Test, System Test …Etc.
 Test Summary

This section includes the summary of testing activity in general. Information detailed here includes
The number of test cases executed
The numbers of test cases pass
The numbers of test cases fail
 Pass percentage
Fail percentage
Comments
This information should be displayed visually by using color indicator, graph, and highlighted table.
Defect
One of the most important information in Test Report is defect. The report should contain following
information.
• Total number of bugs.
• Status of bugs (open, closed, responding).
• Number of bugs open, resolved, closed.
• Breakdown by severity and priority.
Tips to write a good test report:
• Test report is a communication tool between the Test Manager and the stakeholder.
• Through the test report, the stakeholder can understand the project situation, the quality of
product and other things.
• The following scenario shows you why we need a good Test Report.
• You co-operate with outsourcing company, its tester after having performed Performance
Testing of a website, sends you a test report like this
Test Report:

• The information of that report is too abstract. It does not have any detailed information.
• The stakeholder who will read it might be slightly puzzled when they get it.
They might ask or have following sets of questions: -
 Why did they not execute 30 TCs that remains?
 What are these failed Test Cases?
 Doesn't have any bugs description.
To solve that problem, a good Test Report should be:
Detail: You should provide a detailed description of the testing activity, show which testing you have
performed. Do not put the abstract information into the report, because the reader will not understand
what you said.
Clear: All information in the test report should be short and clearly understandable.
Standard: The Test Report should follow the standard template. It is easy for stakeholder to review and
ensure the consistency between test reports in many projects.
Specific: Do not write an essay about the project activity. Describe and summarize the test result
specification and focus on the main point.
For example, to correct the above Test Report, the tester should provide more information such as:
• Project information
• Test cycle: (System Test, Integration Test...etc.)
• Which functions have already tested (% TCs executed, % TCs passed or fail…)
• Defect report (Defect description, Priority or status...)
Automation testing points
 Terms used in Automation.
 Skills needed for automation.
 What to automate, Scope of automate.
 Design and Architecture for Automation.
 Generic requirements for test tool /framework.
 Process model for automation.
 Selecting a test tool.
 Automation for extreme programming model.
 Challenges in Automation.

Software test Automation

What is Automation Testing?
“Automation Testing or Test Automation is a software testing technique that performs using special
automated testing software tools to execute a test case suite.”
 On the contrary, Manual Testing is performed by a human sitting in front of a computer
carefully executing the test steps.
Test Automation is the best way to increase the effectiveness, test coverage, and execution speed in
software testing.

Automated software testing is important due to the following reasons:

• Manual Testing of all workflows, all fields, all negative scenarios is time and money consuming.
• It is difficult to test for multilingual sites manually
• Test Automation does not require Human intervention. You can run automated test unattended
(overnight).
• Test Automation increases the speed of test execution.
• Automation helps increase Test Coverage.
• Manual Testing can become boring and hence error-prone.
 Automated Testing Process:

• Following steps are followed in an Automation Process

• Step 1) Test Tool Selection
• Step 2) Define scope of Automation
• Step 3) Planning, Design and Development
• Step 4) Test Execution
• Step 5) Maintenance

4. SCOPE OF AUTOMATION
 Software Testers and Developers were now on the lookout for a testing method.
 The testing method that could provide the same throughput as Manual Testing, but with
minimal labor and cost.
 In addition to this, they also wanted to reduce the repetitive tasks of Manual Testing.
 The scope of automation or automation testing future scope is huge in the market.
 This was the time when Automation Testing works towards the technical world. It had 2 primary
goals, to be a reliable Software Testing Method and meet the requirements in less time, cost,
and labor.
“So how exactly is Automation Testing successful in saving and money???”
Define the scope of Automation:

The scope of automation is the area of your Application Under Test which will be automated. Following
points help determine scope:
• The features that are important for the business
• Scenarios which have a large amount of data
• Common functionalities across applications
• Technical feasibility
• The extent to which business components are reused
• The complexity of test cases
• Ability to use the same test cases for cross-browser testing
Benefits of Automation Testing
• Reduces Repetitive Tasks.
• Increases Coverage.
• Test Automation helps in Performance Testing.
• Automation Testing equals Reusability.

Which Test Cases to Automate?

Test cases to be automated can be selected using the following criterion to increase the automation ROI
• High Risk - Business Critical test cases.
• Test cases that are repeatedly executed.
• Test Cases that are very tedious or difficult to perform manually.
• Test Cases which are time-consuming.

What do not need to be automated?

The following category of test cases are not suitable for automation:
• Test Cases that are newly designed and not executed manually at least once.
• Test Cases for which the requirements are frequently changing.
• Test cases which are executed on an ad-hoc basis.
Who will Automate ?
 Test Engineers
 Developers

5. DESIGN & ARCHITECTURE FOR AUTOMATION

  Design and architecture is an important aspect of automation.
 The design has to represent all requirements in modules and in the interactions between
modules.(Integration)
 In Integration testing both internal interfaces and external interfaces have to be captured by
design and architecture.
 From fig the thin arrows represent the internal interfaces and the direction of flow and the thick
arrows show the external interfaces. Represent the interaction between the TCDB and the
automation framework only foe automated test cases.
 Architecture involves two major heads:
Test infrastructure– that covers a test case data base.
Defect database or Defect Repository—Contains details of all the defects that are found in various
products that are tested in a particular organization.
 External Modules( TCDB and Defect DB)
 Scenario and Configuration File Modules: Scenarios represent the information on “How to
execute a particular test case”. Configuration file contains set of variables that are used in
automation.
 Test cases and Test Framework Modules: Test cases means the automated test cases taken
from TCDB and executed by the framework. Test framework is a module that combines “What
to execute” and “how they have to be executed.”
 Tools and Results Modules: Tools are build tools and runtime tools etc..Results modules store
the results for each of the test case along with scenarios and variable values have to be stored
for future analysis and action.
 Report Generator and Reports/ Metrics Modules: The module that takes the necessary inputs
and prepares a formatted report is called a report generator. Once the results are available, the
report generator can generate metrics.

6. GENERIC REQUIREMENTS FOR TEST TOOL FRAMEWORK

What is a TEST Automation Framework?
A set of guidelines like coding standards, test-data handling , object repository treatment etc... which
when followed during automation scripting produce beneficial outcomes like increase code re-usage ,
higher portability , reduced script maintenance cost etc.
Mind you these are just guidelines and not rules; they are not mandatory and you can still script without
following the guidelines. But you will miss out on the advantages of having a Framework.

Types of Automation Framework

1) Linear Scripting
2) The Test Library Architecture Framework.
3) The Data-Driven Testing Framework.
4) The Keyword-Driven or Table-Driven Testing Framework.
5) The Hybrid Test Automation Framework.
Some key features /requirements:

• Rapid script development.

• Require minimal knowledge of the tool.
• Test Data – Segregation and Manipulation.
• Readability.
• Reusability.
• Scheduler 24X7 script execution & monitoring.
• Ease of maintenance.
• Agile process fitment.
• Automation scripts developed using this framework can be portable to both QTP and Selenium
IDE. This can be extended or customized based on user requirement.

7. TEST TOOL SELECTION

The importance
 Success in any test automation depends on identifying the right tool for automation.
 Selecting the “correct” Testing Tool for your project is one of the best ways to achieve the
project target.
 The following example will show you the benefit of the testing tool selection.

Type of test tools

 There’re many types of test tool, which Test Manager can consider when selecting the test
tools.
 Open-Source Tools
Open source tools are the program wherein the source code is openly published for use and/or
modification from its original design, free of charge.
Open-source tools are available for almost any phase of the testing process, from Test
Case management to Defect tracking. Compared to commercial tools Open source tools may have
fewer features.
 Commercial Tools
Commercial tools are the software which are produced for sale or to serve commercial purposes.
Commercial tools have more support and more features from a vendor than open-source tools.
 Custom Tools
In some Testing project, the testing environment, and the testing process has special characteristics.
No open-source or commercial tool can meet the requirement. Therefore, the Test Manager has to
consider the development of the custom tool.

Tool selection process

To select the most suitable testing tool for the project, the Test Manager should follow the below
tools selection process

Step 1) Identify the requirement for tools;

You to precisely identify your test tool requirements. All the requirement must
be documented and reviewed by project teams and the management board.
You should consider the following:
A) The tool can generate the test cases automatically.
B) The tool can generate the test result in the desired format.
C) The tester can select which test cases to execute with given set of test data.
D) The tool can execute the test case automatically.
E) The tool can judge and perform test output validation and mark test cases pass or fail.
Step 2) Evaluate the tools and vendors:

After base lining the requirement of the tool, the Test Manager should
• Analyze the commercial and open source tools that are available in the market, based on the
project requirement.
• Create a tool shortlist which best meets your criteria.
• One factor you should consider is vendors. You should consider the vendor’s reputation, after
sale support, tool update frequency, etc. while taking your decision.
• Evaluate the quality of the tool by taking the trial usage & launching a pilot. Many vendors
often make trial versions of their software available for download.
Step 3) Estimate cost and benefit:
• To ensure the test tool is beneficial for business, the Test Manager have to balance the
following factors:
 A cost-benefit analysis should be performed before acquiring or building a tool
Example: After spending considerable time to investigate testing tools, the project team found the
perfect testing tool for the project Guru99 Bank website. The evaluation results concluded that this
tool could
• Double the current productivity of test execution
• Reduce the management effort by 30%
However, after discussing with the software vendor, you found that the cost of this tool is too high
compare to the value and benefit that it can bring to the teamwork.
In such a case, the balance between cost & benefit of the tool may affect the final decision.

Step 4) Make the final decision:

To make the final decision, the Test Manager must have:

• Have a strong awareness of the tool. It means you must understand which is the strong points and
the weak points of the tool
• Balance cost and benefit.
 Even with hours spent reading software manual and vendor information, you may still need to try the tool
in your actual working environment before buying the license.
 You should have the meeting with the project team, consultants to get the deeper knowledge of the tool.
 Your decision may adversely impact the project, the testing process, and the business goals; you should
spend a good time to think hard about it.

oOo