Unit 5RK
Unit 5RK
People are an organization’s most important assets. The tasks of a manager are essentially people
oriented. Unless there is some understanding of people, management will be unsuccessful.
Software engineering is primarily a cognitive activity. Cognitive limitations effectively limit the
software process.
This argument is true about twenty years ago when most of the testing was manual and
the products were somewhere simplistic. Normally development engineers tend to focus on
specific modules. But, now testers require a holistic understanding of the entire product rather
than just a single module. Suppose if the product is more complex, testing requires deep
understanding of multiple domains.
In the early 1990s, most people sought to be expert in C, which moved to C++ and then
to java. Similarly, interms of platforms, shifted from mainframe to client server networking to
network centric computing. About ten years ago, people in the testing functions found very little
parallel to these kind of “resume enriching”. In fact, now a day the use of testing tools requires
programming in languages remarkably similar to those that developers use. And also the
automation tests can prove to be more challenging than even developing the code for the product.
Testing is a destructive process (in the sense of finding defects in a product), there are
more opportunities for out-of-box thinking.
In view of all of the above, testing offers sufficient technical challenges for an interested
professional and we can say “there is testing in all development and development in all
testing”.
Testing is not a devil and development is not an angel: opportunities abound equally in
testing and development.
Filling up positions for testing should not be treated as a second string function. People
are sometimes made to feel that they are in testing because they could not fit in anywhere else.
But, if a person is not suitable for development, for the same or similar reason: he or she may not
be suitable for testing. A person should be assigned to testing only when he or she has the right
aptitude and attitude for testing.
The main function of testing is to find defects in the products; it is easy for an adversary
attitude between the testing team and the development team. So, testing and developing teams
should reinforce each other.
Testing is not an end activity; it happens throughout and continues even after the release.
Test engineers not only say “something doesn’t work”, they also point out what works in
the product. The job of test engineers is not only to find defects but also prevent defects by
proactively reviewing this specifications, design, architecture and code.
Successful people are those who take initiative, volunteer, experiment and share their
learning with others. Testing people should be able to respond quickly to changing priorities,
delayed handover from development.
There are formal core courses on programming but few universities offer core courses
on software testing.
There are “lab courses” for various development tools but none or very few for
common testing tools.
More projects done as a part of the curriculum never ask for neither test plans nor
looking at testing effectiveness. The complete weightage is on coding.
Most courses and projects reward individual performance and present very little
opportunity for team work, which is essential for the success of development and test
engineers.
The senior management of an organization plays a vital role in the development of test
professionals. It is simply not enough to use words like “quality is our top concern” or “people
are our top priority”. This commitment has to be translated into visible actions. It is the
responsibility of the management for recognition and giving reward for the top performers
including testers.
Role of community
Test Engineers need not be just followers of technology – they can take an active part in
shaping and using new technology. Sometimes test engineers are under pressure to learn the
technology and test the product at the same time. Learning the technology and testing the product
at the same time may lead to ineffective testing. This scenario must change. Test Engineers must
understand the new technology areas proactively before they are implemented in a product.
Following the test processes for executing tests, maintaining tests and so on
Filing high quality defects
Categorizing the defects
Adhering to schedules specified
Developing high quality documentation
Generation of metrics related to testing
A number of organizational structures are possible, each having its own positives and
negatives. Figure shows a structure often used by small (fewer than 10) project teams. In this
structure, the test group reports into the Development Manager, the person managing the work
of the programmers.
The Development Manager's goal is to have his team develop software. Testers reporting
bugs just hinder that process. Testers doing their job well on one side make the programmers
look bad on the other. If the manager gives more resources and funding to the testers, they'll
probably find more bugs, but the more bugs they find, the more they'll crimp the manager's
goals of making software.
Despite these negatives, this structure can work well if the development manager is very
experienced and realizes that his goal isn't just to create software, but to create quality software.
Such a manager would value the testers as equals to the programmers. This is also a very good
organization for communications flow. There are minimal layers of management and the testers
and programmers can very efficiently work together.
Figure shows another common organizational structure where both the test group and the
development group report to the manager of the project. In this arrangement, the test group
often has its own lead or manager whose interest and attention is focused on the test team and
their work. This independence is a great advantage when critical decisions are made regarding
the software's quality. The test team's voice is equal to the voices of the programmers and other
groups contributing to the product.
• In this structure, the test group that reports to executive management has the most
independence, the most authority and the most responsibility.
• The independence that this group holds allows them to set standards and guidelines,
measure the results, and adopt processes that span multiple projects.
• A corporate quality standard that works well on database software might not work well
when applied to a computer game. To be effective, this independent quality organization
must find ways to work with all the projects they deal with and temper their enthusiasm
for quality with the practicality of releasing software.
• In software development and testing, one size doesn't necessarily fit all, and what works
for one team may not work for another. There are, however, some common metrics that
can be used to measure, and guidelines that can be followed, that have been proven to
work across different projects and teams for improving their quality levels.
*****
• At the top level are General organizational goals - Ex: Goal of VCET
• Individual projects have specific goals. It usually reflects organizational goals.
• Personal Goals – Each individual in an organization has a set of goals for self improvement.
So that they can effectively contribute to the project.
• Goal statements can express expectations in quantitative terms or be more general in nature.
Example:
• The degree of automation of regression testing is increased from 50% to 75% over the
next 3 years. (Quantitative)
• Testing activities are performed by a dedicated testing group.
• Testing group members have at least a bachelor- level degree and have taken a formal
course in software testing.
Note: Quantitative goals are measurable. It’s useful to evaluate progress towards achieving the
goal.
• In testing domain, goal statement should provide a high level vision of what testing is to
accomplish in the organization with respect to quality of process and product.
• In addition to general testing goal statements, lower level goal statement should be
developed for all levels of testing.
• Policy provides the vision and framework for decision making. It should be properly
documented and it should be available for all interested parties.
• Organization website – Disseminate the goals in website
• A policy statement should be formulated by a team consisting of upper management,
executive personnel and technical staff.
• Testing policy statements reflect, integrate and support achievements of testing goals. –
Which will be useful for increasing the quality and customer satisfaction.
• Test policies also provide high level guidance as to how testing is to be done in the
organization, how its effectiveness will be evaluated, who will be responsible and what
choices of resources are possible.
Example: How to test, what to test and who will test.
• Policies are not written in stone- If organization grows in maturity its policies will
change.
EXAMPLE: Testing policy:
• Our organization (Company Name) realizes that testing is an important component of the
software development process and has a high impact on software quality and the degree
of customer satisfaction.
• Delivering software of the highest quality is our company goal. The presence of defects
has a negative impact on software quality.
• Defect affects the correctness, reliability and usability of a software product, thus
rendering unsatisfactory to the client.
• Testing activities include traditional execution of the developing software, as well as
reviews of the software deliverables produced at all stages of the life cycle.
• The aggregation of all testing activities performed in a systematic manner supported by
organization policies, procedures and standards constitutes the testing process.
• A set of testing standards must be available to all interested parties on an intra
organizational website.
• Standards contain – test related documents, prescribed templates, methods, procedures and
tools.
*****
• Test Plan helps us determine the effort needed to validate the quality of the application
under test.
• Help people outside the test team such as developers, business managers,
customers understand the details of testing.
• Test Plan guides our thinking. It is like a rule book, which needs to be followed.
• Important aspects like test estimation, test scope, Test Strategy are documented in Test
Plan, so it can be reviewed by Management Team and re-used for other projects.
Making a Test Plan is the most important task of Test Management Process. Follow the seven
steps below to create a test plan as per IEEE 829
• How can you test a product without any information about it? The answer
is Impossible. You must learn a product thoroughly before testing it.
• The product under test is Guru99 banking website. You should research clients and the
end users to know their needs and expectations from the application
• Who will use the website?
• What is it used for?
• How will it work?
• What are software/ hardware the product uses?
• Test Strategy is a critical step in making a Test Plan. A Test Strategy document is a high-
level document, which is usually developed by Test Manager. This document defines:
• The project’s testing objectives and the means to achieve them
• Determines testing effort and costs
• Back to your project, you need to develop Test Strategy for testing that banking website.
You should follow steps below:
• Project Budget
• Product Specification
• Now should clearly define the "in scope" and "out of scope" of the testing.
• As the software requirement specification, the project Guru99 Bank only focus on testing
all the functions and external interface of website Guru99 Bank (in scope testing)
• Nonfunctional testing such as stress, performance or logical database currently will not
be tested. (out of scope)
The customer wants you to test his API. But the project budget does not permit to do so.
In such a case what will you do?
Well, in such case you need to convince the customer that API testing is extra work and will
consume significant resources. Give him data supporting your facts. Tell him if API testing
is included in-scope the budget will increase by XYZ amount.
The customer agrees and accordingly the new scopes, out of scope items are
In-scope items: Functional Testing, API Testing
Out of scope items: Database Testing, hardware & any other external interfaces
• There are tons of Testing Types for testing software product. Your team cannot
have enough efforts to handle all kind of testing. As Test Manager, you must
set priority of the Testing Types:
• Which Testing Types should be focused for web application testing?
• Which Testing Types should be ignored for saving cost?
• Risk is future’s uncertain event with a probability of occurrence and a potential for loss.
When the risk actually happens, it becomes the ‘issue’.
In Test Logistics, the Test Manager should answer the following questions:
You may not know exact names of the tester who will test, but the type of tester can be defined.
• To select the right member for specified task, you have to consider if his skill is qualified
for the task or not, also estimate the project budget. Selecting wrong member for the task
may cause the project to fail or delay.
• Person having the following skills is most ideal for performing software testing:
• Ability to understand customers point of view
• Strong desire for quality
• Attention to detail
• Good cooperation
When will the test occur?
• Test activities must be matched with associated development activities.
• You will start to test when you have all required items shown in following figure:
3. Test Objectives
• Test Objective is the overall goal and achievement of the test execution. The objective of
the testing is finding as many software defects as possible; ensure that the software under
test is bug free before release.
• To define the test objectives, you should do the following steps:
1. List all the software features (functionality, performance, GUI…) which may need to
test.
2. Define the target or the goal of the test based on above features.
Based on above features, you can define the Test Objective of the project Guru99 as following:
• Check that whether website Guru99 functionality (Account, Deposit…) is working as
expected without any error or bugs in real business environment
• Check that the external interface of the website such as UI is working as expected and &
meet the customer need.
• Verify the usability of the website. Are those functionalities convenient for user or not?
Exit Criteria
• It specifies the criteria that denote a successful completion of a test phase. The exit criteria
are the targeted results of the test and are necessary before proceeding to the next phase
of development. Example: 95% of all critical test cases must pass.
• Some methods of defining exit criteria are by specifying a targeted run rate and pass rate.
• Run rate is ratio between number test cases executed/total test cases of test specification.
For example, the test specification has total 120 TCs, but the tester only executed 100
TCs, So the run rate is 100/120 = 0.83 (83%)
• Pass rate is ratio between numbers test cases passed / test cases executed. For example, in
above 100 TCs executed, there’re 80 TCs that passed, so the pass rate is 80/100 = 0.8
(80%).
• This data can be retrieved in Test Metric documents.
• Run rate is mandatory to be 100% unless a clear reason is given.
• Pass rate is dependent on project scope, but achieving high pass rate is a goal.
• Example: Your Team has already done the test executions. They report the test result to
you, and they want you to confirm the Exit Criteria.
• In above case, the Run rate is mandatory is 100%, but the test team only completed 90%
of test cases. It means the Run rate is not satisfied, so do NOT confirm the Exit Criteria
5. Resource Planning
Resource plan is a detailed summary of all types of resources required to complete
project task. Resource could be human, equipment and materials needed to complete a
project
6. Plan Test Environment
Test Environment
A testing environment is a setup of software and hardware on which the testing team is
going to execute test cases. The test environment consists of real business and user environment,
as well as physical environments, such as server, front end running environment.
Schedule
8. Test Deliverables
Test Deliverables is a list of all the documents, tools and other components that has to be
developed and maintained in support of the testing effort. There are different test deliverables at
every phase of the software development lifecycle.
Test Scripts
Simulators.
Test Data
Test Traceability Matrix
Error logs and execution logs.
Test Results/reports
Defect Report
Installation/ Test procedures guidelines
Release notes
*****
A test cycle entails planning and running certain tests in cycles, each cycle using a
different build of the product. A test cycle report, at the end of each cycle gives
1. A summary of the activities carried out during those cycle.
2. Defects that were uncovered during that cycle, based on their severity and impact.
3. Progress from the previous cycle to the current cycle interms of defects fixed.
4. Any variations observed in effort or schedule.
The final step in a test cycle is to recommend the suitability of a product for release. A
report that summarizes the results of a test cycle is the test summary report.
Types
1. Phase wise test summary – which is produced at the end of every phase.
2. Final test summary reports – it is also called release test report
A summary report should present
1. A summary of the activities carried out during the test cycle or phase.
2. Variance of the activities carried out form the activities planned.
The tests that were planned to be run but could not be run ( with reasons)
Modifications to tests
Additional tests that were run
Difference in effort and scheduling
3. Summary of results should include
Test that failed, with any root cause
Severity of impact of the defects
4. Comprehensive assessment and recommendation for release should include
“Fit for release “ assessment and
Recommendation of release
*****
4.6. The Role of the Three Critical Groups in Testing Planning and Test Policy
Development
Three groups were identified as critical players in the testing process
Managers, Developers/Testers, and Users/Clients
In TMM terminology they are called the three critical views (CV)
Each group views the testing process from a different perspective that is related to their
particular goals, needs, and requirements.
The developers/testers work with client/user groups on quality-related activities and tasks
that concern user-oriented needs.
Prepared by Dr. R. Kavitha Page 17
SOFTWARE TESTING – UNIT 4
3. Ensure that the necessary training, education, and tools to carry out defined
testing/debugging goals is made available.
As representatives of the technical staff developers must ensure that the policies reflect best
testing practices, are implementable, receive management support, and support among technical
personnel.
Working with management to develop testing and debugging policies and goals.
Participating in the teams that oversee policy compliance and change management.
Familiarizing themselves with the approved set of testing/debugging goals and policies,
keeping up-to-date with revisions, and making suggestions for changes when appropriate.
When developing test plans, setting testing goals for each project at each level of test that
reflect organizational testing goals and policies.
Carrying out testing activities that are in compliance with organizational policies.
The goals and policies of users and clients reflect the organizations efforts to ensure
customer/client/user satisfaction.
*****
4.7. Test Policy
A Test Policy is a high level document and is at the top of the hierarchy of the Test
Documentation structure.
The purpose of the Test Policy document is to represent the testing philosophy of the
company as a whole and to provide a direction which the testing department should adhere to
and follow. It should apply to both new projects and maintenance work.
Setting an appropriate test policy by senior managers provides a robust framework within
which testing practitioners can then operate. This will help to ensure the maximization of the
strategic value inherent in every project.
1. Definition of Testing
Organizations need to be clear why they are testing. This will influence the remainder of
the policy document and also the detailed testing techniques that are selected by test managers at
the programme and project level.
From the understanding of why testing is required it is possible to specify what the
purpose of testing is within the organisation. Without this fundamental linkage the test effort is
destined to fail.
Example: “ensuring the software fulfills its requirements”
It is vital to establish a solid view towards the test process. We should address questions
like, which phases and subtasks will the test process include. Which roles will be involved and
the document structure associated with each tasks, as well as what test levels need to be
considered.
Example: “all test plans are written in accordance with company policy”
3. Test Evaluation
How are we going to evaluate the results of testing, what measures will we use to ensure
test effectiveness in the project?
Which quality criteria are going to be tested and which quality level is the system
required achieving prior to its release with regards to these criteria?
How often and when are we going to assess the usefulness of the current processes in
place and what elements need improving and techniques that shall be used to improve the
processes.
Example: “project review meetings to be held after project completion”
*****
Software testing specialists help develop the test plan and evaluate the quality of software
components. Their role is to detect major software flaws (e.g. bugs, errors, failures, breakdowns,
risks) in order to fix them and ensure the performance of the developed systems. Testing
specialists have a big responsibility - they inform the testing manager of all the improvements to
be made. The work of an entire team of software designers and developers depends on their
analysis.
The staff members who possess the following necessary skills and motivation are called
test specialists.
Development and application of test-related standards;
• Participating in requirements, design, and code reviews;
• Test planning
• Test design
• Test execution;
• Test measurement;
• Test monitoring (tasks, schedules, and costs);
• Defect tracking, and maintaining the defect repository;
• Acquisition of test tools and equipment;
• Identifying and applying new testing techniques, tools, and methodologies;
• Mentoring and training of new test personnel;
• Test reporting.
4.8.1. Skills Needed by a Test Specialist
Test specialist must have:
• Organizational and planning skills;
• The ability to keep track of, and pay attention to, details;
• The determination to discover and solve problems;
• The ability to work with others and be able to resolve conflicts;
• The ability to mentor and train others;
• The ability to work with users and clients;
• Strong written and oral communication skills;
• The ability to work in a variety of environments;
• The ability to think creatively
The first three skills are necessary because testing is detail and problem oriented. In addition,
testing involves policymaking, knowledge of different types of application areas, planning, and
the ability to organize and monitor information, tasks, and people.
Testing also requires inter actions with many other engineering professionals such as
project managers, developers, analysts, process personal, and software quality assurance staff.
Test professionals often interact with clients to prepare certain types of tests, for example
acceptance tests. Testers also have to prepare test-related documents and make presentations.
Training and mentoring of new hires to the testing group is also a part of the tester’s job. In
addition, test specialists must be creative, imaginative, and experiment oriented.
They need to be able to visualize the many ways that a software item should be tested,
and make hypotheses about the different types of defects that could occur and the different ways
the software could fail.
On the technical level testers need to have:
• An education that includes an understanding of general software engineering principles,
practices, and methodologies.
• Strong coding skills and an understanding of code structure and behavior;
• A good understanding of testing principles and practices;
• A good understanding of basic testing strategies, methods, and techniques;
• The ability and experience to plan, design, and execute test cases and test procedures on
multiple levels (unit, integration, etc.);
• A knowledge of process issues
• Knowledge of how networks, databases, and operating systems are organized and how they
work
• A knowledge of configuration management;
• A knowledge of test-related documents and the role each documents plays in the testing
process;
• The ability to define, collects, and analyzes test-related measurements;
• The ability, training, and motivation to work with testing tools and equipment;
• A knowledge of quality issues.
Testers must have knowledge of both white and black box techniques and methods and
the ability to use them to design test cases. Organizations need to realize that this knowledge is a
necessary prerequisite for tool use and test automation. Testers also need to understand quality
measures such as reliability, maintainability, usability, and how to test for them.
Finally, testers should have some knowledge of the problem domain for which the
software has been written. This knowledge, for example, can help them to understand the domain
vocabulary, domain operations, and domain requirements and constraints.
*****