Unit V:- Testing Tools and

Measurements (14 Marks)

• Manual Testing and Need for Automated Testing Tools
• Advantages and Disadvantages of using Tools
• Selecting a Testing Tool
• When to Use Automated Test Tools, Testing Using Automated Tools
• Metrics and Measurements: Types of Metrics, Product Metrics and
Process Metrics, Object Oriented metrics in testing
 Introduction
• Testing of s/w can be done by automation and manual testing
method.
• Manual testing is a method used by to run tests cases manually.
• Automation testing process involves testing with the help of
automation scripts and executing the scripts to run the application
with the help of some automation tool.
 Manual Testing
• It requires a tester to perform manual test operation on the s/w without the
help of test automation.
• Manual s/w testing is performed by a human sitting in front of a computer
carefully going through application screens, trying various usage and input
combinations, comparing the results to the expected behavior and recording
their observations.
• The tester takes over the role of end user and test the s/w to identify any
un-expected behavior or bug.
• There are different types for manual testing like unit testing, integration
testing, system testing and user acceptance testing.
• Testers use test plan, test cases or test scenarios to test s/w.
 Process of Manual Testing:
• The tester takes over the role of end user and test the s/w to identify
any un-expected behavior or bug.
• Manual testing is a process where in a tester often follows a written
test plan that leads them through a set of important test cases.
• A test case in s/w testing is a set of conditions under which a tester
will determine whether an s/w application is working correctly or not.
• Test all the requirements of application are met. There must be at
least two test cases for each requirement: positive and negative test.
 Process of Manual Testing
❑ How to Do Manual Testing
1. Requirement Analysis
2. Test Plan Creation
3. Test case Creation
4. Test case Execution
5. Defect Logging
6. Defect Fix & Re-Verification
Manual Testing
• Advantages of Manual Testing:
• It can be used for both small and big project.
• Easily we can reduce and add our test case according to project
movement.
• Easy to learn for new people who are entered in manual testing.
• It is more reliable than automated(in many case automated does not
cover all cases.)
 Manual Testing
❑Limitations of Manual Testing
⮚ • Manual testing is slow and costly- As it is labor intensive, it takes a long
time to complete test. Increasing headcount increases cost.

⮚ • Manual test do not scale well- As the complexity of the s/w increases,
the complexity of testing problem grows exponentially. This leads to the
increase in total time required for testing as well as total cost of testing.

⮚ • Manual testing is not consistent repeatable- One tester may approach

an perform a certain test differently from another, resulting in different
results on the same test, because tests are not being performed identically.
Manual Testing
⮚ • Lack of training is a common problem- Though manual testing it is
not unique, training is required.

⮚ • Testing is difficult to manage- There are more unknowns and

greater uncertainty in testing than in code development. Modern s/w
development are well structured, but if you do not have sufficient
structure in testing, it will be difficult to manage.
 Automation Testing
• Developing a s/w to test the s/w by using Automation tool is called test automation.

• Using any automation tool for testing to execute the test case is called as
automation testing.

• It is automating the manual testing process currently in use.

• It is the use of strategies, tools and artifacts that augment or reduce the need of
manual or human involvement or interaction in unskilled, repetitive tasks.

• Test engineers strive to catch the defects before the product is released but the
defects do reappear, even with the best manual testing process.

• Automated s/w testing is the best way to increase the effectiveness, efficiency and
coverage of your s/w testing.
 Automation Testing
• It is also called as Test automation where the tester writes scripts and uses
s/w to test the s/w. This process involves automation of a manual process.

• It is used to re-run the test scenarios that were performed manually,

quickly and repeatedly.

• It increases the test coverage, improve accuracy, saves time and money in
comparison to manual testing.

• It is able to playback pre recorded and pre defined actions, compare the
results to the expected behavior and report the success or failure of these
manual tests to a test engineer.
 Automation Testing
• Process Automation Testing:
• Following are the phases of automated testing process—
• Test automation feasibility analysis-
• First step is to check if the application can be automated or not.
• Test automation feasibility analysis is a crucial step in determining whether test
automation is a viable and cost-effective solution for a particular project or application.
• It helps you make an informed decision about whether to invest in test automation or
continue with manual testing.

• Appropriate tool selection- Select the tools. It depends on the technology in which
the application is built, its features and usage.

• Evaluate the suitable framework- Upon selecting the tool, select a suitable
framework.
Automation Testing
• Built the proof of concept(POC)- POC is developed with an end to end
scenario to evaluate if the tool can support the automation of the
application.

• Develop automation framework- After building POC, framework

development is carried out, which is crucial step for the success of
any tests automation project.

• Develop Test scripts, Execute and Analyze- Once script development

is completed, the scripts are executed, results are analyzed and
defects are logged, if any.
Automation Testing
• Advantages of Automation:
• It reduces testing efforts.
• It avoids human mistakes.
• It reduces overall cost of s/w development.
• It saves time as s/w can execute test cases faster than human.
• It improves s/w accuracy.
Difference between Automation and Manual Testing
 Need of Automated Testing Tools
• A test tool is a vehicle for performing test processes. They can improve the
efficiency of a testing process.
• In s/w testing, test automation is the use of special s/w to control
execution of tests and the comparison of actual outcomes with predicted
outcomes.
• Test automation can automate some repetitive but necessary tasks in a
formalized testing process already in place or add additional testing that
would be difficult to perform manually.
• It is basically using programs to do different testing tasks.
• Testing tools are computerized and help in assisting every type of testing.
 Need of Automated Testing Tools
• It is needed when you are testing a particular feature that means you may
need to run your tests not once, but dozens of times.

• Check that the bugs found in previous test are indeed fixed and that no new
bugs were introduced. This process of rerunning your tests is known as
regression testing.

• If a small project had several thousand test cases to run there might be
rarely enough time to execute them just once. Running them numerous
times might be impossible. So automation will be more efficient than
manual testing.
 Features of Test Tools
• Test automation is the use of s/w and h/w combination as the case may be to
create test plans, test cases, definition of test iterations, setting precondition,
test reporting functions.
• Each and every testing tool has some capabilities as well as some drawbacks
also.
• Some features of tool are listed below:
• Each tool has a specific purpose, knowing the tools technique is very
important while using the tool.
• The same tool may be used differently depending upon the technique
selecting for using it.
• Tools can be used in an entire SDLC for all verification and validation activities.
• A tool improves the efficiency and effectiveness of a testing process.
• A tool enhances a defect visualization process effectively.
 Features of Test Tools
• Testing tools are of following types:
• 1. Static Testing Tools
• 2. Dynamic Testing Tools

⮚ 1.Static Testing Tools—

• It is used during static analysis of a system.
• It is used throughout SDLC e.g. tools used for verification purpose.
• Static test tools do not involve actual input and output. Rather they take a
symbolic approach to testing i.e. they do not test the actual execution of
the s/w.
 Features of Test Tools
❑These tools include-
❖• Flow Analyzer- They ensure consistency in data flow from input to
output.
❖• Path tests- They find unused code and code with contradictions.
❖• Coverage analyzers- It ensures that all logic paths are tested.
❖• Interface Analyzers- It examines the effect of passing variables and data
between modules.
 Features of Test Tools
⮚ 2.Dynamic testing tools—
• They are used at different levels of testing starting from unit testing and which may
go up to system testing and performance testing.
• Dynamic testing process is performed by the dynamic test tools. These tools test the
software with existing or current data.
• They are generally used by s/w testers to test the s/w with ‘live data’.
❑ They include the following -
❖• Test Driver- It inputs data into a module under test
❖• Test Beds- it simultaneously displays source code along with the program under
execution.
❖• Emulators- the response facilities are used to emulate parts of the system not yet
developed.
❖• Mutation analyzers – the errors are deliberately ‘fed’ into the code in order to test
fault tolerance of the system
Software testing tools include
• Selenium
• Cypress
• Nightwatch.js
• Espresso
• JUnit
• TestNG
• TestRail
• TestComplete
 Advantages and Disadvantages of using Tools:
❑ Advantages-
⮚ • Fast- Runs test significantly faster than human users. Tools work faster than
human being.
⮚ • Repeatable- testers can test how the website or s/w reacts after repeated
execution of the same operation.
⮚ • Reusable- Tests can be reused on different versions of s/w.
⮚ • Reliability- Test perform precisely the same operation each time they are
run thereby eliminating human error.
⮚ • Comprehensive- testers can build test suites of test that covers every
feature in s/w application.
⮚ • Programmable- testers can program sophisticated tests that bring hidden
information .
⮚ • Consistent- Tools are consistent in behavior and are not prone to fatigue.
 Advantages and Disadvantages of using Tools
❑ Disadvantages-
⮚ • Unrealistic expectation from the tool- unrealistic expectations can lead to
greatest risk to success with tools. It is very important to have clear and
realistic objective for what the tool can do.
⮚ • People often make mistake by understanding the time, cost and effort for
the initial introduction of a tool- once the tool is purchased some people
might use it so that will be beneficial while some will resist. So both need to
be handled.
⮚ • People frequently miscalculate the time and effort needed to achieve
significant and continuing benefits from the tool- The first attempt of using
the tool were unlikely to be very good but with more experience and practice
you became much better.
Advantages and Disadvantages of using Tools

⮚ • Mostly people underestimate the effort required to maintain the

test assets generated by the tool. – Because of insufficient planning
for maintenance of the assets that the tool produces there are
chances that the tool might end up as ‘shelf ware’ along with
previously linked risks.
⮚ • People depend on the tool a lot- Tools are just the s/w they can do
only what they have been designed to do, but they cannot do
everything.
 Selecting a Testing Tool
❑ Selecting the test tool is an important aspects of test automation for
several reasons as given below:
⮚ 1. Free tools are not well supported and get phased out soon.

⮚ 2. Developing in-house tools takes time. Even though these tools can be less
expensive and can meet needs better, they are often developed by the
personal interest. They tend to have poor documentation and thus it may
be unusable if the person who developed leaves the organization.

⮚ 3. Test tools sold by vendors are expensive. Most organizations small and
medium enterprises will have to carefully evaluate the economic impact
before making investment.
 Selecting a Testing Tool
⮚ 4. Test tools require strong training. Test automation cannot be successful
unless the people using the tools are properly trained. Such training usually
involves getting familiar with the scripting languages that come with the tool,
customizing tool for use. The effort involved in using the tools even after such
training is quite high.

⮚ 5. Test tools generally do not meet all the requirements for automation. Since
tools are meant to be generic, they may not fully satisfy the needs of a
particular customer. That is why customization and extensibility become key
issues.

⮚ 6. Not all test tools run on all platforms. Portability of the tool and the scripts
to multiple platforms is therefore a key factor in deciding the test automation
tool.
Criteria for selecting test tools
❑The categories are-
⮚ 1. Meeting requirements
⮚ 2. Technology expectations.
⮚ 3. Training/skills and
⮚ 4. Management aspects
 Criteria for selecting test tools
⮚ 1. Meeting requirements
• There are plenty of tools available in the market but rarely do they meet all the
requirements of a given product or a given organization.
• Evaluating different tools for different requirements involve significant effort,
money, and time.
• Given of the plethora of choice available, huge delay is involved in selecting
and implementing test tools.
⮚ 2. Technology expectations
• Test tools in general may not allow test developers to extends/modify the
functionality of the framework.
• So extending the functionality requires going back to the tool vendor and
involves additional cost and effort.
• A good number of test tools require their libraries to be linked with product
binaries
 Criteria for selecting test tools

⮚ 3. Training/skills
• While test tools require plenty of training, very few vendors provide the
training to the required level.
• Organization level training is needed to deploy the test tools, as the user of
the test suite are not only the test team but also the development team
and other areas like configuration management.

⮚ 4. Management aspects
• A test tool increases the system requirement and requires the hardware
and software to be upgraded.
• This increases the cost of the already- expensive test tool.
 When to use Automated test Tool

• Automated tools are used to increase the effectiveness, efficiency and

coverage of s/w testing.
• Some key points to be remembered are-
• 1. Complexity of test cases-
• • Simple and easy repetitive test cases may be given a tool while complex
test cases, where testers have to take a lot of decisions, selecting
appropriate tool may take time for testers.
• • Tools are faster than human beings in any part of testing where complex
calculations are involved
 When to use Automated test Tool
• 2. Test case dependency-
• • When success or failure of one test case has a direct effect on the next
test case, automation may be avoided.
• • In automation, self cleaning and self standing test cases are preferred as
there may not be much dependency of success of any test case on another.
• 3. Number of iterations of testing-
• • If the same test is executed many times i.e. iteration automation is a
better option than doing it manually.
• • If the number of iterations is not very high, automation may be a very
costly affair.
 When to use Automated test Tool
• 4. Stress, reliability, scalability and performance testing:
• These types of testing require the test case to be run from a large number
of different machines for an extended period of time, such as 24 hours, 48
hours, and so on.
• It is just not possible to have hundreds of users trying out the product they
may be not willing to perform the repetitive tasks, nor will it be possible to
find that many people with the required skill sets.
• Test cases belonging to these testing types become the first candidates for
automation
 When to use Automated test Tool
• 5.Regression tests:
• Regression tests are repetitive in nature .These test cases are executed multiple
times during the product development phase. Given the repetitive nature of
test cases, automation will save significant time and effort in the long run. The
time thus gained can be effectively utilized for other tests.
• 6.Functional tests:
• These kinds of tests may require a complex set up and thus require specialized
skill, which may not be available on an ongoing basis. Automating these once,
using the expert skill sets, can enable using less-skilled people to run these test
on an ongoing basis
 What are Metrics and Measurements
• A s/w metric is a measure of some property of a piece of s/w or its
specification.
• Metrics help to measure the progress, quality and health of a s/w testing
effort.
• All significant activities in a project need to be tracked to ensure that the
project is going as per plan and to decide on any corrective actions.
• The measurement of key parameters is an essential part of tracking.
• Measurement first require collecting a set of data. The collected data have to
be analyzed in totality to draw the appropriate conclusions.
 What are Metrics and Measurements
• Metrics are simply a way to measure or evaluate something, often
quantitatively. They help in assessing and understanding various aspects of a
process, system, or performance. Metrics are used to track progress,
determine success, identify areas for improvement, and make informed
decisions. They provide a way to express and communicate data and
information in a standardized, objective, and easy-to-understand manner.

• For example, in a business context, metrics might include things like sales
revenue, customer satisfaction scores, or website traffic. These metrics help
businesses analyze their performance and make data-driven decisions to
achieve their goals.
 What are Metrics and Measurements
⮚ What is a Metric?
▪Metrics can be defined as standards of measurement.
▪Metric is unit used for describing or measuring an attribute.
▪Test metrics are means by which the s/w quality can be measured. E.g. how
many defects are existed within the module, what is test coverage %?
❑Metric is a standard unit of measurement that quantifies results. Metric
used for evaluating the s/w processes, products and services is termed as
s/w metrics.
 What are Metrics and Measurements

• Metric is a standard unit of measurement that quantifies results.

Metric used for evaluating the s/w processes, products and
services is termed as s/w metrics.

• Software metrics is a measurement-based technique which is

applied to processes, products and services to supply
engineering and management information and working on the
information supplied to improve processes, products and
services, if required.”
What are Metrics and Measurements
• Need of Software measurement:
• 1. Establish the quality of the current product or process.
• 2. To predict future qualities of the product or process.
• 3. To improve the quality of a product or process.
• 4. To determine the state of the project in relation to budget and
schedule.
What are Metrics and Measurements
• Types of Metrics
⮚ Metrics can be classified into different types based on:
• 1. What they measure? and
• 2. What area they focus on?
❑ At a very high level, metrics can be classified as product metrics and
process metrics.
What are Metrics and Measurements
❑Product Metrics:
• Product metrics are used for the purpose of describing the
characteristics of the product like size, complexity, design features,
performance, efficiency, reliability, portability etc.
• It gives the information regarding status of a software product.
• The data required for such metrics are also generated in the process
of testing and may help us to know the quality of the product.
 What are Metrics and Measurements
❑Some of the Basic Metrics are
• Number of failures experienced in a time interval.
• Time interval between failures.
• Time of failure.
• Estimated time for testing
• Test Efficiency
• Actual testing time.
• Time remaining to complete the testing
• Maximum and minimum failures experienced in any time interval.
• % of time spent=(actual time spent / estimated testing time) *100
• Average time interval between failures.
 What are Metrics and Measurements
❑Following are Important Types of Product Metrics:
• 1. Metrics for analysis model: These address various aspects of the analysis
model such as system functionality, system size and so on.
• 2. Metrics for design model: These allow to assess the quality of design and
include architectural design metrics, component level design metrics and so
on.
• 3. Metrics for source code: These assess source code complexity,
maintainability and other characteristics.
• 4. Metrics for Testing: These help to design efficient and effective test cases
and also evaluate effectiveness of testing.
• 5. Metrics for maintenance: These assess the stability of the software
product.
 What are Metrics and Measurements
• Process Metrics:
• Process metrics are used for the purpose of describing the effectiveness
as well as quality of the processes that produce the software product.
• These metrics are in general developed for the purpose of monitoring the
progress of testing, status of the design and development of test cases
and outcome of test cases after execution.
 What are Metrics and Measurements
❑Some of the Basic Process Metrics are:
• Number of test cases designed
• Number of test cases executed
• Number of test cases passed
• Number of test cases failed
• Test case execution time
• Total execution time
• Time spent for development of a test case
 What are Metrics and Measurements
❑Following are the Supplementary Metrics
• % of test cases executed
• % of test cases passed
• % of test cases failed
• Total actual execution time / total estimated execution time
• Average execution time of a test case.

⮚ Even if these metrics are simple, they are useful in knowing the
progress of testing and may supply significant data to the testers as well
as to the project manager.
 Object-Oriented Metrics

• Object-oriented metrics are quantitative measures used to assess various

aspects of software systems developed using object-oriented programming
(OOP) principles.
• These metrics provide insights into the complexity, maintainability, and
quality of the code.
• They are valuable for software developers, project managers, and quality
assurance teams to make informed decisions about the design and
maintenance of object-oriented software.
 Object-Oriented Metrics
• Object-oriented metrics are measurements used to evaluate the design
and complexity of software systems that are developed using object-
oriented programming principles. These metrics assess various aspects of
the code, such as classes, objects, and their relationships. Here's an easy
explanation of some common object-oriented metrics:
• Class Count: This metric measures the number of classes in a software
system. More classes can indicate a complex system.
• Method Count: It quantifies the number of methods or functions within
classes. A large number of methods in a class can suggest a complex and
potentially hard-to-maintain design.
 Object-Oriented Metrics
• Inheritance Depth: This metric evaluates how many levels of inheritance a
class has. A deep inheritance hierarchy can make the system more intricate
and harder to understand.
• Coupling: It assesses the interdependencies between classes. High coupling
means classes are tightly connected, which can make the code less flexible
and maintainable.
• Cohesion: This metric measures how closely related the functions and
attributes of a class are. A class with high cohesion is more focused on a
single task and is often considered better in terms of design.
• Lines of Code (LOC): It simply counts the number of lines of code in a class
or method. Longer code can be harder to maintain and understand.
 Object-Oriented Metrics
• Some common Object-Oriented Metrics:
• Class Metrics:
• Number of Methods (NOM): The count of methods within a class.
• Weighted Methods per Class (WMC): The sum of complexities of all methods
within a class. This is often measured using Cyclomatic Complexity or other
similar metrics.
• Response for a Class (RFC): The total number of methods that can be
potentially executed when an object of the class is created. This includes both
local and inherited methods.
• Number of Children (NOC): The number of subclasses that inherit from a
given class.
• Coupling Between Objects (CBO): The number of other classes a class is
directly connected to (e.g., through method calls or attribute references).
Object-Oriented Metrics

• Method Metrics:
• Cyclomatic Complexity: Measures the complexity of a method by
counting the number of linearly independent paths through its
control flow.
• Depth of Inheritance Tree (DIT): The length of the longest path from a
class to the root class in the inheritance hierarchy.
 Object-Oriented Metrics
• Inheritance Metrics:
• Afferent Couplings (Ca): The number of classes that depend on a particular
class.
• Efferent Couplings (Ce): The number of classes that a particular class
depends on.
• Coupling and Cohesion Metrics:
• Lack of Cohesion in Methods (LCOM): Measures the lack of cohesion
between methods in a class. High LCOM values indicate that a class may be
doing too much and should be split.
• Coupling Between Objects (CBO): The number of other classes a class is
directly connected to.
• Size Metrics:
• Lines of Code (LOC): The number of lines of code in a class or method.
• Number of Attributes (NOA): The count of attributes (fields) in a class
 Object-Oriented Metrics
• Conclusion:
• These metrics help in identifying areas of code that may need
improvement, such as refactoring to reduce complexity,
enhancing cohesion, and minimizing coupling.
• However, it's essential to use these metrics in conjunction with
qualitative analysis and not rely solely on them for software
evaluation.
• Additionally, different organizations may have their own
variations or customized metrics based on their specific needs
and coding standards.
Chapter 5 End…!!!

!!! Best of Luck !!!