Test software for performance measure using automation

CO5 :
testing tools.
1) Software Testing
Manual Testing
• Manual testing is a testing process that is carried out manually in order to find
defectswithout the usage of tools or automation scripting.

• A test plan document is prepared that acts as a guide to the testing process in order
to havethe complete test coverage.

How to Do Manual Testing

• Requirement Analysis

• Test Plan Creation

• Test case Creation

• Test case Execution

• Defect Logging

• Defect Fix & Re-Verification

Limitations of Manual Testing

Limitation of Manual Testing and Need for Automated Testing Tool. (Question: Explain the
need of Automation testing over manual testing.
i. Manual testing is slow and costly.
ii. It is very labor intensive, it takes a long time to complete tests.
iii. Manual tests don’t scale well. As the complexity of the software increases the
complexity of the testing problem grows exponentially. This leads to an increase in
total time devoted to testing as well as total cost of testing.
iv. Manual testing is not consistent or repeatable. Variations in how the tests are performed
as inevitable, for various reasons. One tester may approach and perform a certain
test differently from another, resulting in different results on the same test, because
the tests are not being performed identically.
v. Lack of training is the common problem, although not unique to manual software
testing.
vi. GUI objects size difference and color combinations are not easy to find in manual
 testing.
vii. Not suitable for large scale projects and time bound projects.
viii. Batch testing is not possible, for each and every test execution Human user interaction is
mandatory.
ix. Comparing large amount of data is impractical.
x. Processing change requests during software maintenance takes more time.
NEED of automated testing
i. An automated testing tool is able to playback pre-recorded and predefined
actions, compare the results to the expected behavior and report the success
or failure of these manual tests to a test engineer.
ii. Once automated tests are created they can easily be repeated and they can be
extended to perform tasks impossible with manual testing.
iii. Because of this, savvy managers have found that automated software testing
is an essential component of successful development projects

Automation Testing
• Automation testing is a technique uses an application to implement entire life
cycle of thesoftware in less time and provides efficiency and effectiveness to the
testing software.

• Automation testing is an Automatic technique where the tester writes scripts by

own anduses suitable software to test the software

• It is basically an automation process of a manual process.

• The main goal of Automation testing is to increase the test efficiency and develop
software value.

Test automation is the use of special software to control the execution of tests and the
comparison of actual outcomes with predicted outcomes. The objective of automated testing
is to simplify as much of the testing effort as possible with a minimum set of scripts. Test
automation can automate some repetitive but necessary tasks in a formalized testing process
already in place, or addadditional testing that would be difficult to perform manually.

1. Automated Software Testing Saves Time and Money

i. Software tests have to be repeated often during development cycles to ensure quality.
Every time source code is modified software tests should be repeated.

ii. For each release of the software it may be tested on all supported operating systems
and hardware configurations.

iii. Manually repeating these tests is costly and time consuming. Once created, automated
tests can be run over and over again at no additional cost and they are much faster than
manual tests.
 iv. Automated software testing can reduce the time to run repetitive tests from days to
hours. v. A time savings that translates directly into cost savings

2. Testing Improves Accuracy

i. Even the most conscientious tester will make mistakes during monotonous manual
testing.

ii. Automated tests perform the same steps precisely every time they are executed and
never forget to record detailed results.

3. Increase Test Coverage

i. Automated software testing can increase the depth and scope of tests to help improve
software quality.

ii. Lengthy tests that are often avoided during manual testing can be run unattended.

iii. They can even be run on multiple computers with different configurations.

iv. Automated software testing can look inside an application and see memory contents,
data tables, file contents, and internal program states to determine if the product is
behaving as expected.

v. Automated software tests can easily execute thousands of different complex test cases
during every test run providing coverage that is impossible with manual tests.

vi. Testers freed from repetitive manual tests have more time to create new automated
software tests and deal with complex features.

4. Automation Does What Manual Testing Cannot

i. Even the largest software departments cannot perform a controlled web application test
with thousands of users.

ii. Automated testing can simulate tens, hundreds or thousands of virtual users interacting
with network or web software and applications.

5. Automated QA Testing Helps Developers and Testers

i. Shared automated tests can be used by developers to catch problems quickly before
sending to QA.

ii. Tests can run automatically whenever source code changes are checked in and notify
the team or the developer if they fail.

iii. Features like these save developers time and increase their confidence.

6. Team Morale Improves

 i. This is hard to measure but we’ve experienced it first hand, automated software testing
can improve team morale.

ii. Automating repetitive tasks with automated software testing gives your team time to
spend on more challenging and rewarding projects. iii. Team members improve their skill
sets and confidence and, in turn, pass those gains on to their organization.

7. Test Complete is a Powerful and Affordable Automated Software Testing Tool

i. Test Complete addresses a full range of software testing challenges facing corporate IT
departments, product developers, QA engineers, and consultants.

ii. Test Complete enhances the software testing process by increasing efficiency,
removing complexity and lowering costs

Types of test automation tools:

• Static automation tools:

These tools are used throughout a software development lifecycle, e.g. tools used
for verification purposes.

There are many varieties of static testing tools used by different people as per the
type of system being developed.

These tools do notinvolve actual input and output. Rather, they take a symbolic
approach to testing, i.e. they do not test the actual execution of the software. e.g.
Flow analyzers, Coverage analyzers, Interface analyzer Code complexity
measurement tools can be used to measure the complexity of a given code.

Similarly, data-profiling tools can be used to optimize a database. Code-profiling

tools can be used to optimize code. Test-generators are used for generating a test
plan form code. Syntax-checking tools are used to verify correctness of code.

• Dynamic automation tools:

These tools test the software system with live data.

They are used at different levels of testing starting from unit testing & which may go
up to system testing & performance testing.

These tools are generally used by tester.

These tools test the software system with live data. e.g. Test driver, Test beds,
Emulators

There are many different tools used for dynamic testing.

 Some of the areas covered by testing tools are:
1. Regression testing using automated tools.
2. Defect tracking and communication systems used by tracking &
communication.Performance, Load, stress-testing tools

Benefits of Automation Testing

• Reduces time of testing
• Improve the bugs finding
• Deliver the quality product
• Allow to run tests many time with different data
• Getting more time for test planning
• Save resources or requires less
• Automation never tires, and expert person can work at a time many tools.

Advantages of Switching To Automated Testing from Manual Testing

• Efficient testing
• Consistency in testing
• Better quality software
• Automated testing is cheaper
• Automation testing is faster
• Automated testing is more reliable
• Automated testing reduces human and technical risks
• Automated testing is more powerful and versatile

Features of automated testing tools

• FAST Automation Engine
• Object Eye Internal Recorder
• Visual Recorder
• Multiple Browsers Support
• Dynamic Test Data Support
• Continuous Server Integration
• Mobile Testing Support
• Robust Reporting & Logs
• Reusable Methods
• Integration with Bug Tracking tools
• Integration with Test Management Tools
• Job Scheduler
• Image Comparison
• Distributed Test Execution
• Captcha Automation
• Risk Based Testing
 Static Testing Tool
• Static testing tools are used during static analysis of a system.
• Static testing tools are used throughout a software development life cycle, e.g. ,
tools usedfor verification purposes.
• There are many varieties of static testing tools used by different people as per the
type ofsystem being developed.
• Code complexity measurement tools can be used to measure the complexity of
a givencode.
• Similarly, data-profiling tools can be used to optimize a database.
• Code-profiling tools can be used to optimize code.
• Test-generators are used for generating a test plan form code.
• Syntax-checking tools are used to verify correctness of code.

Features of Test Tool:

Static Testing Tool

1. Static analysis tools are generally used by developers as part of the development and
component testing process. The key aspect is that the code (or other artefact) is not executed
or run but the tool itself is executed, and the source code we are interested in is the input data
to the tool.
2. These tools are mostly used by developers.
3. Static analysis tools are an extension of compiler technology – in fact some compilers do
offer static analysis features. It is worth checking what is available from existing compilers or
development environments before looking at purchasing a more sophisticated static analysis
tool.
4. Other than software code, static analysis can also be carried out on things like, static analysis
of requirements or static analysis of websites (for example, to assess for proper use of
accessibility tags or the following of HTML standards).
5. Static analysis tools for code can help the developers to understand the structure of the code,
and can also be used to enforce coding standards.

Features for selecting static test tools:

i. Assessment of the organization’s maturity (e.g. readiness for change);
ii. Identification of the areas within the organization where tool support will help to
improvetesting processes;
iii. Evaluation of tools against clear requirements and objective criteria;
iv. Proof-of-concept to see whether the product works as desired and meets the
requirements andobjectives defined for it;
v. Evaluation of the vendor (training, support and other commercial aspects) or
open-sourcenetwork of support;
vi. Identifying and planning internal implementation (including coaching and mentoring
for thosenew to the use of the tool).
• Static test tools includes:
1. Flow analyzer :ensures consistency in data flow from input to output
2. Path tests :finds unused codes and codes with contradictions
3. Coverage analyzer :all logical paths are tested
4. Interface analyzer :examines effects of passing variables and data between modules

Dynamic Testing Tool

• Dynamic testing tools are used at different levels of testing starting from unit
testing &which may go up to system testing & performance testing.
• These tools are generally used by tester.
• There are many different tools used for dynamic testing. Some of the areas
covered bytesting tools are:
• 1. Regression testing using automated tools.
• 2. Defect tracking and communication systems used by tracking & communication.
• Performance, Load, stress-testing tools.

Features for selecting dynamic test tools:

• To detect memory leaks;
• To identify pointer arithmetic errors such as null pointers;
• To identify time dependencies.

Dynamic test tools includes:

1. Test driver :includes data into module under test (MUT)
2. Test beds :simultaneously displays source code along with the program under execution
3. Emulators
Features of Test Tool:
2. Dynamic Testing Tool
i. Dynamic analysis tools are ‘dynamic’ because they require the code to be in a running
state. They are ‘analysis’ rather than ‘testing’ tools because they analyze what is happening
‘behind the scenes’ that is in the code while the software is running (whether being executed
with test cases or being used in operation).

ii. Let us take an example of a car to understand it in a better way. If you go to a showroom of
a car to buy it, you might sit in the car to see if is comfortable and see what sound the doors
make – this would be static analysis because the car is not being driven. If you take a test
drive, then you would check that how the car performs when it is in the running mode e.g. the
car turns right when you turn the steering wheel clockwise or when you press the break then
how the car will respond and can also check the oil pressure or the brake fluid, this would be
dynamic analysis, it can only be done while the engine is running.
 Advantages using Tools
1. Speed. The automation tools tests the software under tests with the very faster speed. There
is a vast difference between the speed of user entering the data and the automated tools
generating and entering the data required for the testing of the software. Speed of this software
also completes the work faster.
2. Efficiency. While testers are busy running test cases, testers can't be doing anything else.
If the tester have a test tool that reduces the time it takes for him to run his tests, he has
more time fortest planning and thinking up new tests.
3. Accuracy and Precision. A test tool will perform the same test and check the results
perfectly, each and every time.
4. Resource Reduction. Sometimes it can be physically impossible to perform a certain test
case. The number of people or the amount of equipment required to create the test condition
could be prohibitive. A test tool can be used to simulate the real world and greatly reduce the
physical resources necessary to perform the testing.
5. Simulation and Emulation. Test tools are often used to replace hardware or software that
would normally interface to your product. This "fake" device or application can then be used
to drive or respond to your software in ways that you choose and ways that might otherwise
be difficult to achieve.
6. Relentlessness. Test tools and automation never tire or give up. They can keep going and
going and on and on without any problem; whereas the tester gets tired to test again and again.

OR

Advantage and Disadvantage of Using Tools.

There are many benefits that can be gained by using tools to support testing.
2) Reduction of repetitive work: Repetitive work is very boring if it is done manually. People
tend to make mistakes when doing the same task over and over. Examples of this type of
repetitive work include running regression tests, entering the same test data again and again
(can be done by a test execution tool), checking against coding standards (which can be done
by a static analysis tool) or creating a specific test database (which can be done by a test data
preparation tool).
3) Greater consistency and repeatability: People have tendency to do the same task in a
slightly different way even when they think they are repeating something exactly. A tool will
exactly reproduce what it did before, so each time it is run the result is consistent.
4) Objective assessment: If a person calculates a value from the software or incident reports, by
mistake they may omit something, or their own onesided preconceived judgments or
convictions may lead them to interpret that data incorrectly. Using a tool means that
subjective preconceived notion is removed and the assessment is more repeatable and
consistently calculated. Examples include assessing the cyclomatic complexity or nesting
levels of a component (which can be done by a static analysis tool), coverage (coverage
measurement tool), system behavior (monitoring tools) and incident statistics (test
management tool).
5) Ease of access to information about tests or testing: Information presented visually is
much easier for the human mind to understand and interpret. For example, a chart or graph is
a better way to show information than a long list of numbers – this is why charts and graphs
in spreadsheets are so useful. Special purpose tools give these features directly for the
information they process. Examples include statistics and graphs about test progress (test
execution or test management tool), incident rates (incident management or test management
 tool) and performance (performance testing tool).

Disadvantage

1) Unrealistic expectations from the tool: Unrealistic expectations may be one of the greatest
risks to success with tools. The tools are just software and we all know that there are many
problems associated with any kind of software. It is very important to have clear and realistic
objectives for what the tool can do.
2) People often make mistakes by underestimating the time, cost and effort for the initial
introduction of a tool: Introducing something new into an organization is hardly
straightforward. Once you purchase a tool, you want to have a number of people being able
to use the tool in a way that will be beneficial. There will be some technical issues to
overcome, but there will also be resistance from other people – both need to be handled in
such a way that the tool will be of great success.
3) People frequently miscalculate the time and effort needed to achieve significant and
continuing benefits from the tool: Mostly in the initial phase when the tool is new to the
people, they miscalculate the time and Software Testing effort needed to achieve significant
and continuing benefits from the tool. Just think back to the last time you tried something
new for the very first time (learning to drive, riding a bike, skiing). Your first attempts were
unlikely to be very good but with more experience and practice you became much better.
Using a testing tool for the first time will not be your best use of the tool either. It takes time
to develop ways of using the tool in order to achieve what is expected.
4) Mostly people underestimate the effort required to maintain the test assets generated by
the tool: Generally, people underestimate the effort required to maintain the test assets
generated by the tool. Because of the insufficient planning for maintenance of the assets that
the tool produces there are chances that the tool might end up as ‘shelf-ware’, along with the
previously listed risks.
5) People depend on the tool a lot (over-reliance on the tool): Since there are many benefits
that can be gained by using tools to support testing like reduction of repetitive work, greater
consistency and repeatability, etc. people started to depend on the tool a lot. But the tools are
just a software they can do only what they have been designed to do (at least a good quality
tool can), but they cannot do everything. A tool can definitely help, but it cannot replace the
intelligence needed to know how best to use it, and how to evaluate current and future uses of
the tool. For example, a test execution tool does not replace the need for good test design and
should not be used for every test – some tests are still better executed manually. A test that
takes a very long time to automate and will not be run very often is better done manually.

Guidelines for selecting a tool:

1. The tool must match its intended use. Wrong selection of a tool can lead to
problems likelower efficiency and effectiveness of testing may be lost.
2. Different phases of a life cycle have different quality-factor requirements. Tools
required ateach stage may differ significantly.
3. Matching a tool with the skills of testers is also essential. If the testers do not have
propertraining and skill then they may not be able to work effectively.
4. Select affordable tools. Cost and benefits of various tools must be compared
beforemaking final decision.
5. Backdoor entry of tools must be prevented. Unauthorized entry results into failure
 of tooland creates a negative environment for new tool introduction.
OR
Selecting a Testing Tool.
1) While introducing the tool in the organization it must match a need within the organization,
and solve that need in a way that is both effective and efficient.
2) The tool should help in building the strengths of the organization and should also address its
weaknesses.
3) The organization needs to be ready for the changes that will come along with the new tool.
i. If the current testing practices are not good enough and the organization is not mature,
then it is always recommended to improve testing practices first rather than to try to
find tools to support poor practices. Certainly, we can sometimes improve our own
processes in parallel with introducing a tool to support those practices and we can
always pick up some good ideas for improvement from the ways that the tools work.
ii. Do not depend on the tool for everything, but it should provide support to your
organization as expected.
The following factors are important during tool selection:
1) Assessment of the organization’s maturity (e.g. readiness for change)
2) Identification of the areas within the organization where tool support will help to improve
testing processes;
3) Evaluation of tools against clear requirements and objective criteria;
4) Proof-of-concept to see whether the product works as desired and meets the requirements and
objectives defined for it;
5) Evaluation of the vendor (training, support and other commercial aspects) or open-source
network of support;
6) Identifying and planning internal implementation (including coaching and mentoring for
those new to the use of the tool).

Criteria for Selecting Test Tools:

• The Criteria's for selecting Test Tools are,
1. Meeting requirements;
2. Technology expectations;
3. Training/skills;
4. Management aspects.

1. Meeting requirements
There are plenty of tools available in the market but rarely do they meet all the requirements
of a given product or a given organization. Evaluating different tools for different
requirements involve significant effort, money, and time. Given of the plethora of choice
available, huge delay is involved in selecting and implementing test tools.
2. Technology expectations
Test tools in general may not allow test developers to extends/modify the functionality of the
framework. So extending the functionality requires going back to the tool vendor and involves
additional cost and effort. A good number of test tools require their libraries to be linked with
product binaries.
 3. Training/skills
While test tools require plenty of training, very few vendors provide the training to the
required level. Organization level training is needed to deploy the test tools, as the user of the
test suite are not only the test team but also the development team and other areas like
configuration management.
4. Management aspects
A test tool increases the system requirement and requires the hardware and software to be
upgraded. This increases the cost of the already- expensive test tool.

When to use automated test tools

• Stress, reliability, scalability and performance testing:
These types of testing require the test case to be run from a large number of different
machines for an extended period of time, such as 24 hours, 48 hours, and so on. It is
just not possible to have hundreds of users trying out the product they may be not
willing to perform the repetitive tasks, nor will it be possible to find that many people
with the required skill sets. Test cases belonging to these testing types become the
first candidates for automation.
• Regression tests: Regression tests are repetitive in nature .These test cases are
executed multiple times during the product development phase. Given the repetitive
nature of test cases, automation will save significant time and effort in the long run.
The time thus gainedcan be effectively utilized for other tests.
• Functional tests: These kinds of tests may require a complex set up and thus require
specialized skill, which may not be available on an ongoing basis. Automating these
once, using the expert skill sets, can enable using less-skilled people to run these test
on an ongoing basis

When to Use Automated TestTools,Testing Using Automated Tool.

1. When Does Test Automation Make Sense?
i. When there are many repetitive tests.
ii. When there are frequent regression testing iterations.
iii. When you need to simulate large number of users who are using the
application resources.
iv. When AUT is having comparatively stable UI.
v. When you have large set of BVT cases.
vi. When you can’t rely solely on manual test execution for critical functionality
criteria you need to consider before selecting any testing tool:
i. Do you have necessary skilled resource to allocate for automation tasks?
ii. What is your budget?
iii. Does the tool satisfy your testing needs? Is it suitable for the project
environment and technology you are using? Does it support all tools and objects
used in the code? Sometime you may get stuck for small tests due to inabilities
of the tool to identify the objects used in the application.
 What are Metrics and Measurements: Types of Metrics,
Project Metrics, Progress and Productivity Metrics?

In software testing there are three main areas which needs to be considered while
thinking about metrics and measurement.

(Question: Explain the concept of metrics and measurement. – 4 Marks)

i. Defining the Metrics
➢ Small and quality set of metrics should be chosen, large set of metrics
should be avoided as it is very confusing to understand large set of metrics.
➢ Metrics should also be uniform and everybody in team should agree with
it.
ii. Tracking test metrics
➢ After defining the metrics the next step is to track the metrics.
➢ Since tracking is a constant activity so it’s always nice to automate the
tracking part.
➢ Automation reduces time required to track the metrics, analyze them and
measure them.
iii. Reporting
➢ Reporting of the metrics is the most important step, you should report test
metrics to stakeholders so that they have clear picture of project progress.

A Metric is a quantitative measure of the degree to which a system, system component, or

process possesses a given attribute.
Metrics can be defined as “STANDARDS OF MEASUREMENT”.
Software Metrics are used to measure the quality of the project. Simply, Metric is a unit
used for describing an attribute. Metric is a scale for measurement.

“How many issues are found in thousand lines of code?”, here No. of issues is one
measurement & No. of lines of code is another measurement. Metric is defined from these
two measurements.

1. What is Software Test Measurement?

i. Measurement is the quantitative indication of extent, amount, dimension,

capacity, or size of some attribute of a product or process.
ii. Test measurement example: Total number of defects.
iii. Please refer below diagram for clear understanding of the difference
between Measurement & Metrics.
 2. Why Test Metrics?
Generation of Software Test Metrics is the most important responsibility of the
Software Test Lead/Manager.

Test Metrics are used to,

i. Take the decision for next phase of activities such as, estimate the cost &
schedule of future projects.
ii. Understand the kind of improvement required to success the project
iii. Take decision on process or technology to be modified etc.
Importance of Software Testing Metrics:

i. As explained above, Test Metrics are the most important to measure the
quality of the software.

ii. Now, how can we measure the quality of the software by using Metrics?
iii. Suppose, if a project does not have any metrics, then how the quality of
the work done by a Test analyst will be measured?

For Example: A Test Analyst has to,

i. Design the test cases for 5 requirements
ii. Execute the designed test cases
iii. Log the defects & need to fail the related test cases
iv. After the defect is resolved, need to re-test the defect & re-execute the
corresponding failed test case.

In above scenario, if metrics are not followed, then the work completed by the
test analyst will be subjective i.e. the test report will not have the proper
information to know the status of his work/project.
 If Metrics are involved in the project, then the exact status of his/her work with
proper numbers/data can be published.

I.e. in the Test report, we can publish:

1. How many test cases have been designed per requirement?
2. How many test cases are yet to design?
3. How many test cases are executed?
4. How many test cases are passed/failed/blocked?
5. How many test cases are not yet executed?
6. How many defects are identified & what is the severity of those defects?
7. How many test cases are failed due to one particular defect? etc.

Based on the project needs we can have more metrics than above mentioned list, to
know the status of the project in detail.

Based on the above metrics, test lead/manager will get the understanding of thebelow mentioned key points.
a) %ge of work completed
b) %ge of work yet to be completed
c) Time to complete the remaining work
d) Whether the project is going as per the schedule or lagging? etc.

Based on the metrics, if the project is not going to complete as per the schedule,
then the manager will raise the alarm to the client and other stake holders by
providing the reasons for lagging to avoid the last minute surprises.
3. Metrics Life Cycle

4. Types of Manual Test Metrics:

Testing Metrics are mainly divided into 2 categories.
1. Base Metrics
➢ Base Metrics are the Metrics which are derived from the data gathered by
the Test Analyst during the test case development and execution.
➢ This data will be tracked throughout the Test Life cycle. I.e. collecting the
data like, Total no. of test cases developed for a project (or) no. of test cases
need to be executed (or) no. of test cases passed/failed/blocked etc.

2. Calculated Metrics
➢ Calculated Metrics are derived from the data gathered in Base Metrics.
These Metrics are generally tracked by the test lead/manager for Test
Reporting purpose.
 Unit 5: Testing Tools and Measurements
➢ Examples of Software Testing Metrics:
Let’s take an example to calculate various test metrics used in software test reports:

Below is the table format for the data retrieved from the test analyst who is actually
involved in testing:
 Unit 5: Testing Tools and Measurements
5. Importance of metrics and measurement in SDLC
i. During all the software development life cycle it is very important to apply metrics
and measurement because metrics and measurement set expectations. If there are
well established metrics and measurements in project then the test analyst can
easily track and report quality results to the management.

ii. If the metrics & measurements are not established properly then the assessment of
software quality is purely subjective which arises disputes at the end of
development life cycle. You can consider some the following

areas where you can apply metrics and measurement. This list is not
exhaustive, you can have metrics for lot more things.
➢ Schedule of project
➢ Coverage
➢ Planned & actual cost
➢ Workload & resource usage
➢ Product risk & project risk
➢ Defects
iii. While doing test planning we set the expectations for the stakeholders or we set the
baselines for them. If you have established the baselines the test reporting is
consistent to the management and you can avoid subjective assessment of testing