6th Unit ST Notes
6th Unit ST Notes
Chapter Objectives
• Appreciate the need for automated testing tools
• Understand the requirements of testing tools
• Get to know the various categories of testing tools and their functionality
• Learn how to select a testing tool
• Learn the process of introducing the tools in an organization
• Learn how to use WinRunner, LoadRunner and Test Director to get an idea of the
power of testing tools
Software testing tools are of immense use to develop quality software. In this chapter, we
will study the importance of these tools, various categories of testing tools and the
process of introducing tools in your organization. How to use WinRunner, LoadRunner
and TestDirector are also explained-to get an idea of the power of the testing tools and to
illustrate how easy it is to carry out testing by using the tools.
6.1 Need for Tools
Testing the software manually is time-consuming and requires lot of infrastructure and
manpower. All these drawbacks can be overcome if the testing process is automated.
Using these tools has many advantages:
Once the software is ready for testing, the functionality of the software can be
tested repeatedly to improve the quality and the reliability.
Testing can be done unattended, for example during night time and during
holidays.
When the software has to be tested in different environments (different hardware
platforms, different operating systems, using different browsers etc.), the labor
involved can be reduced.
Performance testing can be done without the need for many computers and many
test engineers. The test tools simulate the multiple users on a single machine.
Testing process can be planned and managed effectively by using these tools.
Test reports can be generated automatically for later analysis and corrective
action.
The testing process can be managed efficiently—the planning can be done
systematically, the tests can be scheduled efficiently, and the bug tracking can be
done effectively.
Hence, using automated test tools results in:
Improvement in the quality and reliability of the software.
Drastic reduction in time, effort and money spent on testing.
A systematic approach to the testing process.
Efficient management of the testing process even if the teams are at different
geographical locations.
6.2 Classification of Tools
A wide variety of software testing tools are available to carry out the different types of
testing. These testing tools can be broadly divided into the following categories:
1. Functional/Regression Testing Tools: These tools are used to test the
application software and web applications such as web sites. As majority of the
applications involve Graphical User Interface (GUI), the tools test the GUI
objects and functionality automatically. These tools carry out black box testing.
Client/Server applications, Enterprise Resource Planning (ERP), software
packages such as SAP, Customer Relations Management (CRM) software
packages such as Siebel, web sites etc. can be tested for functionality by using
these tools.
Whenever a change is made to the software, the software needs to be retested and
hence these tools are also called regression testing tools.
Example:Compuware's QACenter, Segue Software’s SilkTest, IBM Rational's
Robot, Mecury Interactive’s WinRunner belong to this category.
2. Performance Testing Tools: These tools are used to carry out performance
testing or stress testing. These tools are very useful to test how the application
works when multiple users access the application simultaneously.
The application can be for example, a database or a web site. These tools simulate
multiple users on a single machine and hence you do not need many machines and
many test engineers to do the performance testing.
AutoTester's AutoController, Compuware's QALoad, Mercury Interactive's
LoadRunner, Segue Software's SilkPerformer, IBM Rational's Performance
Tester, Apache JMeter belong to this category.
3. Source Code Testing Tools: These tools check the source code of the application
software. The testing is white box testing and hence the implementation details
are taken into consideration. These tools are divided into two categories:
Static analysis tools
Dynamic analysis tools
Static analysis tools are used to check the source code without executing it. These
tools are used to find out a variety of problems in the source code such as:
Whether coding standards/guidelines are followed or not.
Whether the code is portable or any machine operating system dependent
features have been used.
variables declared but unused
dead code or unreachable code i.e., the code that will never be executed
Obtaining the coding metrics such as the ratio of commented lines to
uncommented lines, function point average etc.
Popular static analysis tools are IBM Rational Purify, McCabe Visual Quality
ToolSet, Parasoft CodeWizard, QA C and CheckMate. The utility "lint" available
in Unix/Linux systems is an excellent tool for checking the portability of C code
as well as for analyzing the various problems listed above.
Dynamic analysis tools are used to find out the potential problems in the code, by
executing the code. These tools are used
To detect memory errors such as memory Allocation errors, memory
leaks, buffer overflow problems
To check line coverage, branch/path/condition coverage
Time taken for execution of different portions of the code which is
required to carry out the code optimization etc.
IBM Rational's Purify is a popular tool for checking memory leaks. In Unix/Linux
systems, "profilers" are available (Iprof and prof commands) -- these profilers
display the number of times when each line is executed. Some application
software source code testing tools are: AutomatedQA's AQtime, Parasoft's
Insure++ and Telelogic's Logiscope.
4. Java Testing Tools: As Java has become a popular programming language in
recent years, a number of tools are available exclusively for testing Java
applications. These tools are for testing applications written in Java programming
language and for testing Java classes.
Jemmy is an open source library to create automated tests for Java GUI
applications. JMeter of Apache is open source software to do performance testing.
Parasoft's jtest is used for Java class testing.
5. Embedded Software Testing tools: Testing embedded software is a very
challenging task as the timing requirements for these applications are very
stringent. In embedded systems, the code has to be optimized so that it occupies
the minimum memory. IBM Rational Test Real Time is the widely used test tool
in this category.
6. Network Protocol Testing tools: As computer networks are becoming
widespread, testing networking/communication software has attained lot of
importance in recent years. A number of tools are available for testing networking
and communication protocols. Many test instrumentation vendors such as Agilent
Technologies, Rhode & Schwartz etc. supply protocol analyzers which generate
the necessary protocols based on international standards such as ITU-T standards.
netIQ's ANVL (Automated Network Validation Library) is to test routers and
other networking products. This software generates packets in correct and
incorrect formats to test the networking software. netIQ's Chariot is a network
performance testing tool.
7. Configuration Management/Bug tracking tools: In large software development
projects, configuration management is a very important process. Also, when the
test engineers report bugs, the managers have to track these bugs and ensure that
all the bugs are removed. To facilitate this activity, good workflow management
Software is important. Many such tools, which are web-based, are available.
bugzilla's Bugzilla is an open source defect tracking system. Samba's Jitterbug is a
freeware defect tracking system. Microsoft Excel is also used extensively for bug
tracking.
8. Test process Management Tools: These tools help in managing process-oriented
software testing. Using these tools, the QA manager can create a formal test plan,
allocate resources, schedule unattended testing, track the status of various
bugs/activities etc. AutoTester's AutoAdviser, Compuware's QADirector, QIS's
OCIT, Segue Software's SilkPlan Pro, IBM Rational Test Manager, Mercury
Interactive's TestDirector are some such tools.
9. Support Tools: A number of tools are available that are not directly used by the
test engineers, but these tools help the developers/testers/QA engineers in
different phases of software development. Some such tools are:
Requirements management tools are used to manage the requirements.
The tools help in keeping track of the priority of the requirement, whether
each requirement is converted into design/code and whether the
requirement is tested or not. Telelogic FOCAL POINT is a tool used
extensively for requirements management.
Design tools are used to carry out the analysis and the design of the
software. A large number of tools are available for OOA and OOD, such
as IBM Rational tools using which a detailed design can be carried out by
using object-oriented analysis methodology. However, note that these
tools are used by the developers/architects. Some of these tools are also
capable of generating the test cases from the design. Telelogic TAU G2 is
used to model the requirements of application software and Telelogic
Rhapsody is used to model the requirements of embedded systems.
Traceability tools help in tracing whether all the specifications have been
tested by linking each specification with the corresponding test case.
Unit testing tools are used to carry out the unit testing. Generally, only the
developers carry out the unit testing, not the test engineers. The tools nunit
(www.nunit.org) and junit (www.junit.org) are popular open source unit
testing tools for C++ and Java programs respectively.
6.2.1 Functional/Regression Testing Tools
In functional testing of software, the functionality of the software is tested without
bothering about the source code and implementation details.
Consider example of testing a login process. In many applications, initially a login screen
will appear and you need to type in a username and a password. Only users who are
authorized to use the application will be allowed to enter the application. Assume that a
user is given the username "john' and password as 'mary'. To test this simple GUI, you
need to generate the test cases. The specification says that the username and the password
are case-sensitive. Hence, if the user types "John", it is an invalid username. You can
generate the test cases as shown in Table 8.2.
Input: user name Input: Password Expected output
john mary Login successful
John Mary Login failed
Jhon [blank field] Login failed
JHON MARY Login failed
Jhon Jhon Login failed
Table 8.2 Test cases for Login screen
Just to test a login screen, you need to generate a good number of test cases. The
development engineer, during the initial stages of development, would give his software
with lot of bugs (the login will be successful if the password field is left blank). Every
time, he modifies the software, you need to run all the above test cases and check whether
the software still has any bugs. Obviously, if you record the test by using a tool, the test
script can be run repeatedly.
When a user enters his login name and password, the software will check in a database
whether a valid username and password are entered. So, the login name can be not only
john, but also donald, bush, prasad and so on. It is not possible to generate test cases by
entering so many usernames and passwords manually. Testing tools provide a beautiful
mechanism through data driven testing. In data driven testing, the username and
password are considered as variables. The values these variables can take can be obtained
from a database and the testing is carried out on them.
All functional/regression testing tools operate in a similar fashion. They capture the user
interactions with the GUI (keyboard entries and mouse clicks) and automatically create a
test script. The script can be run repeatedly without the need for interacting with the GUI
again and again. The scripting languages differ for each tool.
6.2.2 Performance/Load Testing Tools
Client/Server (C/S) applications, many users access the server application (such as a
database) simultaneously. In such a case, the software has to be tested for its
performance.
Consider a database application. When only one user accesses the database, the
transaction responses time to query the database and obtain the report is say, 10 Seconds.
When 10 users access the database simultaneously, what will be the response time--10
seconds or 40 seconds? When 1000 users access the database simultaneously what will
be the response time--10 seconds or 10 minutes? No user will accept a few minutes of
response time.
Generally, in the SRS document, the response time will be specified. How do you test the
software to find out the response time when say 100 users are simultaneously accessing
the database? As shown in Figure 8.2, you need to connect 100 client systems and then
put 100 test engineers on the job and use a communication network such as a Local Area
Network (LAN) or a Wide Area Network (WAN). Your company will go bankrupt by the
time you complete the testing. An economical approach is to use the performance testing
tools which will do the job with minimal infrastructure and minimal number of persons.
Principle 3: Product
Software engineers shall ensure that their products and related modifications meet the
highest professional standards possible. In particular, software engineers shall, as
appropriate:
Strive for high quality, acceptable cost, and a reasonable schedule, ensuring
significant tradeoffs are clear to and accepted by the employer and the client, and
are available for consideration by the user and the public.
Ensure proper and achievable goals and objectives for any project on which they
work or propose.
Identify, define, and address ethical, economic, cultural, legal, and environmental
issues related to work projects.
Ensure that they are qualified for any project on which they work or propose to
work, by an appropriate combination of education and training, and experience,
Ensure an appropriate method is used for any project on which they work or
propose to work.
Work to follow professional standards, when available, that are most appropriate
for the task at hand, departing from these only when ethically or technically
justified.
Strive to fully understand the specifications for software on which they work.
Ensure that the specifications for software on which they work have been well
documented, satisfy the users' requirements, and have the appropriate approval.
Ensure realistic quantitative estimates of cost, scheduling, personnel, quality, and
outcomes on any project on which they work or propose to work and provide an
uncertainty assessment of these estimates.
Ensure adequate testing, debugging, and review of software and related document
on which they work.
Ensure adequate documentation, including significant problems discovered and
solutions adopted, for any project on which they work.
Work to develop software and related documents that respect the privacy of those
who will be affected by that software.
Be careful to use only accurate data derived by ethical and lawful means, and use
it only in ways properly authorized.
Maintain the integrity of data, being sensitive to outdated or flawed occurrences
. Treat all forms of software maintenance with the same professionalism as new
development.
Principle 4: Judgment
Software engineers shall maintain integrity and independence in their professional
judgment. In particular, software engineers shall, as appropriate:
Temper all technical judgments by the need to support and maintain human
values.
Only endorse documents either prepared under their supervision or within their
areas of competence and with which they are in agreement.
Maintain professional objectivity with respect to any software or related
document they are asked to evaluate.
Not engage in deceptive financial practices such as bribery. double billing, or
other improper financial practices.
Disclose to all concerned parties those conflicts of interest that cannot reasonably
be avoided or escaped.
Refuse to participate, as members or advisors, in a private, governmental, or
professional body concerned with software-related issues in which they, their
employers, or their clients have undisclosed potential conflicts of interest.
Principle 5: Management
Software engineering managers and leaders shall subscribe to and promote an ethical
approach to the management of software development and maintenance.
In particular, those managing or leading software engineers shall, as appropriate:
Ensure good management for any project on which they work, including effective
procedures for promotion of quality and reduction of risk.
Ensure that software engineers are informed of standards before being held to
them.
Ensure that the software engineers know the employer's policies and procedures
for protecting passwords, files and information that is confidential to the employer
or confidential to others.
Assign work only after taking into account appropriate contributions of education
and experience tempered with a desire to further that education and experience.
Ensure realistic quantitative estimates of cost, scheduling, personnel, quality, and
outcomes on any project on which they work or propose to work, and provide an
uncertainty assessment of these estimates.
Attract potential software engineers only by full and accurate description of the
conditions of employment.
Offer fair and just remuneration.
Not unjustly prevent someone from taking a position for which that person is
suitably qualified.
Ensure that there is a fair agreement concerning ownership of any software,
processes, research, writing, or other intellectual property to which a software
engineer has contributed.
Provide for due process in hearing charges of violation of an employer's policy or
of this Code.
Not ask a software engineer to do anything inconsistent with this Code.
Not punish anyone for expressing ethical concerns about a project.
Principle 6: Profession
Software engineers shall advance the integrity and reputation of the profession consistent
with the public interest. In particular, software engineers shall, as appropriate:
Help develop an organizational environment favorable to acting ethically.
Promote public knowledge of software engineering.
Extend software engineering knowledge by appropriate participation in
professional organizations, meetings, and publications.
Support, as a member of a profession, other software engineers striving to follow
this Code.
Not promote their own interest at the expense of the profession, client, or
employer.
Obey all laws governing their work, unless, in exceptional circumstances, such
compliance is inconsistent with the public interest.
Be accurate in stating the characteristics of software on which they work,
avoiding not only false claims but also claims that might reasonably be supposed
to be speculative, vacuous, deceptive, misleading, or doubtful.
Take responsibility for detecting, correcting, and reporting errors in software and
associated documents on which they work.
Ensure that clients, employers, and supervisors know of the software engineer's
commitment to this Code of Ethics, and the subsequent ramifications of such
commitment.
Avoid association with businesses and organizations which are in conflict with
this code.
Recognize that violations of this Code are inconsistent with being a professional
software engineer.
Express concerns to the people involved when significant violations of this Code
are detected unless this is impossible, counterproductive, or dangerous.
Report significant violations of this Code to appropriate authorities when it is
clear that consultation with people involved in these significant violations is
impossible, counterproductive, or dangerous.
Principle 7: Colleagues
Software engineers shall be fair to and supportive of their colleagues. In particular,
software engineers shall, as appropriate:
Encourage colleagues to adhere to this Code.
Assist colleagues in professional development.
Credit fully the work of others and refrain from taking undue credit.
Review the work of others in an objective, candid, and properly documented way.
Give a fair hearing to the opinions, concerns, or complaints of a colleague.
Assist colleagues in being fully aware of current standard work practices
including policies and procedures for protecting passwords, files, and other
confidential information, and security measures in general.
Not unfairly intervene in the career of any colleague; however, concern for the
employer, the client, or public interest may compel software engineers, in good
faith, to question the competence of a colleague.
In situations outside of their own areas of competence, call upon the opinions of
other professionals who have competence in that area.
Principle 8: SELF
Software engineers shall participate in lifelong learning regarding the practice of their
professional and shall promote an ethical approach to the practice of the profession. In
particular, software engineers shall continually endeavor to:
Further their knowledge of development in the analysis, specification, design,
development, maintenance, and testing of software and related documents,
together with the management of the development process.
Improve their ability to create safe, reliable, and useful quality software at
reasonable cost and within a reasonable time.
Improve their ability to produce accurate, informative, and well-written
documentation
Improve their understanding of the software and related documents on which they
work and of the environment in which they will be used.
Improve their knowledge of relevant standards and the law governing the
software and related documents on which they work.
Improve their knowledge of this Code, its interpretation, and its application to
their work.
Not give unfair treatment to anyone because of any irrelevant prejudices.
Not influence others to undertake any action that involves a breach of this Code.
Recognize that personal violations of this Code are inconsistent with being a
professional software engineer.
If each and every software engineer follows this code of ethics and professional practice,
it can be ensured that the project execution will be smooth and every project will be great
success.
9.5 Ethical Issues: Right versus Wrong
While pursuing a career in software engineering, we all encounter situations in which
there may not be right/wrong answer for a particular question. In such cases, you should
always use you own judgment to decide what is right and what is wrong but that
judgment should be based on realistic reasoning and with adequate background
information. For instance, there are professionals who feel that working in defense
organizations particularly in projects for destroying mankind is wrong: but many of those
who work on such projects are proud of their contributions for the defense of their
countries. These are two different viewpoints-it is not correct to say that one is right and
the other is wrong. You have to decide for yourself.
The broad guidelines as given in the code of ethics are to ensure that no damage is
done to human life and society at large for the sake of personal advancement. Every
software professional's main objective must be to improve the quality of life.
Everywhere, money, power and fame drive human beings. To achieve their personal
goals, the ethics are set aside resulting in corruption, crime and unhealthy social life.
Professionals, who have a great role to play to make the world a better place to live, need
to follow the code of ethics. As software becomes the lifeline of many critical systems,
the software professionals have a great responsibility so that the society respects the
software profession. It is not enough to have technical competence; one has to have social
consciousness and contribute to improve the quality of life through ethical behavior.