White Paper

Ten Points for Improving the Testing

Process
Lars Mats and Matthew Graney

Telelogic AB, May 2002

Ten Points for Improving the Testing Process

Testing is an integral part of the production lifecycle, including software and

hardware design, and plays an important role in the overall quality assurance
process that should optimally be part of every project, in every organization
and business.
The following points are beneficial to consider when approaching the testing
process.

1. Reinforce the need for testing

Though testing is required, companies are often reluctant to allocate time and
resources to the process. As a concept, testing may be regarded as evidence
that there is an imperfection in the software manufacturing process, but it is the
only means through which project managers can verify the actual quality of the
work before it reaches its users. In addition, testing can often be subjected to
the far end effect; that is, it is perceived to be at the far end of the project
and therefore not given any attention until too late in the lifecycle.

2. Minimize testing requirements

Although one measure of a successful test process is to demonstrate that errors
can indeed be detected, if too many errors are found, emphasis needs to be
placed on the development process that allowed errors to propagate throughout
the life-cycle in the first place.
In order to reduce design flaws, more and more development teams are taking
advantage of graphical analysis and design notations such as the Specification
and Description Language (SDL, ITU-T Recommendation Z.100) or the
Unified Modeling Language (UML). These notations and the tools that support
them allow improve quality in several ways, including the early verification of
requirements through design test and simulation.

3. Allocate adequate resources at an early stage

A common reason for not detecting errors at an early stage is that the test tool
is inadequate or that the test team lacks the required skills or knowledge of the
application under test. The result is that the quality assurance of the project is
jeopardized. To prevent this from happening, the test team should have the
resources to detect any subtle errors that may be present in any design. Do not
rely on designers to detect errors as they normally can only detect a very small
percentage of errors they themselves create. Make sure the testing team is
properly trained and prepared and is independent of the design team (thus they
will find a higher percentage of the errors). Remember that testing is part of the

Telelogic AB, 2002

Page 2

Ten Points for Improving the Testing Process

critical path of a project therefore, it is as important to have skilled testing
personnel as it is to have skilled designers, engineers, and managers.
It is a well-known fact that the cost of fixing errors is dramatically reduced if
the error can be detected early in the development lifecycle. A test team needs
to be built up and maintained, as the requirements for a system are being
understood (errors in the requirements are the cheapest to fix if caught early
but the most expensive if caught too late). Testing should as early as possible,
i.e., before development has been completed.

4. Plan for testing

A test plan should be available, not only covering what to test, but also how to
test. Neglecting the latter may result in severe problems during the testing
phase of a project. User interfaces are just as important as the services that
need to be tested. Normally, this is not so much a problem for unit testing as it
is for system or deployment testing. In this instance, the following should be
asked early in the design phase: How do you test software when it is deployed
on the target platform, and how do you trace any software bugs that show up
only on the target?
Two things are vitally important. 1) If you have no reason for a test then that
test should not exist. This reason can only be documented by having clear
traceability back to the original requirement of a system. 2) If you have no
traceability information how do you know that your tests cover the
requirements adequately? Not getting a high enough level of requirements
coverage impacts the overall quality of the system that is produced.

5. Have clear objectives for testing

There are many objectives in testing: performance, regression, conformance,
usability, acceptance, performance, and more. Each of these tests requires
specific tools. Some tests may be purely algorithmic and automatic, while
others require that the application actually be bundled on the target hardware.
In some cases, manual testing is the most suitable method. Remember to
analyze what tools are required for each of the testing objectives. There is not
one single tool that solves all of the problems, but some, possibly in
combination with others, will give a very robust testing platform. Always
concentrate effort on a combination of testing the most important requirements
with the most effective way of finding errors.

6. Use standardized test suites

Some telecommunication protocol standards contain standardized test suites.
Always check the protocol standards to see if there are test suites for the
application you are designing or testing. If so, use the standard test suite as a

Telelogic AB, 2002

Page 3

Ten Points for Improving the Testing Process

starting point for testing, as this saves time and gives the test engineer the
additional benefit of using the standard tests with virtually no extra effort. If
you are designing a new protocol, consider designing test suites as well. The
European Telecommunication Standards Institute (ETSI), the Bluetooth
Special Interest Group and the ATM Forum are two organizations providing
such test suites. Other groups, such as the 1394 Trade Association (supporting
the IEEE 1394 standard, also known as Firewire and i.Link) have
mandated the creation of conformance test suites for future protocol
development. Some of these suites have been implemented by commercial
organizations and allow any company to purchase them in an executable
format. However, be aware that these test suites are often updated as the
standards evolve.

7. Design tests for reusability

It is important to design software for reuse, and it is also important to be able
to reuse tests. The standardized test suites provided by standards organizations
are only one category of reusable tests. Test engineers may think that these
standardized tests are only an outline of how the application is to behave
(which is true for some early standardized test cases), but most of todays
standardized conformance test suites are also executable (after some adaptation
to the implementation). These standardized tests provide a good outline for
how to compose platform- independent and reusable tests, using a technology
that has been successfully employed by European organizations for several
years.
It is also important to consider the maintainability of tests.

8. Use commercial tools for testing

Testing of complex applications cannot be conducted in a reliable and
repeatable manner without the proper tools. The principles of software
engineering apply to software testing as well no one would ever consider
creating a complex, real- world application without tools (CASE tools,
compilers, etc.). Engineers need tools for testing as well as for development.

9. Use non-proprietary test languages

Standardized languages are often better documented than their nonstandardized counterparts. For example, a project manager may choose to
develop a proprietary testing tool and language as part of a testing plan. This
decision may lead to more development time spent on that subproject than on
the actual main project. Needless to say, the outcome of the project would be
less than successful, and the developed test tool would fall into oblivion. In
order to alleviate this problem, some universities and companies offer courses
in the use of standardized and formal languages.

Telelogic AB, 2002

Page 4

Ten Points for Improving the Testing Process

Non-proprietary test languages also have a far more rigorous level of scrutiny
applied to them, considering such things as the language syntax to the detailed
semantics of what should happen when a test executes. This experience should
be reused when testing rather than attempting to reinvent the wheel by
developing an in-house test language.

10. Use a specialized test language

There are specialized languages for expressing tests. It is as important to select
the right language for testing as it is to select a suitable language for
development. TTCN is a language with a long and rich history in the area of
conformance testing of communications protocols, but has recently been
updated to apply to a range of other testing problems, and certainly not just
limited to the communications domain. The latest version is know as TTCN-3
(Test and Test Control Notation) and has been standardized by ETSI. Any test
language should have adequate support for timing, data definitions, value
definitions, concurrency and synchronous and asynchronous communication.
Using a specialized test language such as TTCN-3 brings opportunities to
outsource testing, to use commercial off- the-shelf test tools and to reuse or
purchase tests from 3rd party sources.

Conclusions
Testing is a critical part of any development organization and needs to treated
as suc h. Testing has its own life-cycle that parallels the design and
development life-cycle test planning, requirements analysis, design of test
cases, writing of test cases, test execution and traceability and can only be
managed effectively when the right level of resources are dedicated to it.

Lars Mats is a Senior Architect at Telelogic AB in Malm, Sweden. He has

been involved in the development of Telelogics tool family and in consulting
for test suite deployment and implementation for various sites throughout
Europe, the US and Australia. He received his MS in computer science from
Uppsala University, Uppsala, Sweden, and can be reached at
[email protected].
Matthew Graney is Product Marketing Manager for Telelogic Tau and the
real-time embedded marked for Telelogic North America Inc. He joined
Telelogic in October 1999, prior to which he worked with Motorola in
Australia in the development of TTCN System Test solutions for GSM/GPRS
cellular handsets. He graduated from the University of Adelaide, Australia in
1993 with an honors degree in Computer Systems Engineering, and can be
reached at [email protected].

Telelogic AB, 2002

Page 5