Test Driven Development Method in Software Development Process

Denis Duka, Lovre Hribar

Ericsson Nikola Tesla
Research & Development Center
Split, Croatia
[email protected]; [email protected]

operational phase of the system [3]. Given the error proneness

Abstract: Test-driven development (TDD) is an advanced
technique of using automated unit tests to drive the design of of humans, it is prudent to revisit software artifacts one or
software and force decoupling of dependencies. TDD does not more times to ensure that our understanding of the issues, our
replace traditional testing, instead it defines a proven way to designs, and our implementations are mutually conformant
ensure effective unit testing. The overall initiative results in and correct. Process feed-back and feed-forward loops for
continuous improvements and provides a mechanism for problem detection and fault elimination [3, 4] are beneficial.
propagating the lessons learned between projects to further These loops may be over different releases of the product,
improve the organization. This paper gives an overview of TDD over individual phases of a single release, and/or over
concept. Some initial results analysis as well as plans for further individual tasks. Reliable implementation of very tight fault-
practice enhancement are also presented.
elimination loops, especially those that are not just reactive
(i.e. that result from a problem that needs to be corrected), but
also proactive (forward error correction – preventive
1. I&TRODUCTIO&
activities and dynamic process improvement) are generally
associated with high Capability Maturity Model (CMM)
The pressure to improve software development process is
levels [5]. Additionally, the earlier one finds an error, the less
not new, but in today’s competitive environment there is even
expensive it is to fix [3, 6, 7].
greater emphasis on delivering a better service at lower cost.
Despite that, we still do not have reliable tools for ensuring Research and development of techniques for making
that complicated software systems intended for high- software development easier and faster have been going on
confidence tasks are free from faults and operational failures. for as long as software has existed. Still, projects commonly
Faults may result from root causes that range from knowledge spend at least 50 percent of their development effort on
errors, to communication errors, to incomplete analysis rework that could have been avoided or at least been fixed
errors, to transcription errors. Exacerbating the problem is the less expensively. That is, 20-80 percent depending on the
ever-growing expectations of the end-users and the growth in maturity of the organization and the types of systems the
the complexity of the tasks. There are essentially three ways organization develops. In a larger study on productivity
of dealing with faults [1]: improvement data, most of the effort savings generated by
improving software process maturity, software architectures
1. Fault-avoidance is achieved through appropriate
and software risk management came from reductions in
specification, design, implementation and maintenance
avoidable rework [8]. A major reason for this is that faults are
activities intended to avoid faults in the first place. This
cheaper to find and remove earlier in the development cycle.
includes use of advanced software construction methods,
formal methods and re-use of reliable self-describing software One approach to improve software development process
building blocks (objects), and active knowledge domain and to reduce rework is Test Driven Development (TDD). It
support. is a programming technique that relies on the repetition of a
very short development cycle: First the developer writes a
2. Fault-elimination is the analytical compensation for
failing automated test case that defines a desired improvement
errors committed during specification, design and
or new function, then produces code to pass that test and
implementation. It manifests as verification, validation and
finally refactors the new code to acceptable standards. In
testing.
other words, it is an evolutionary approach to development
3. Fault-tolerance is the run-time compensation for any which combines test-first development where you write a test
residual problems, out-of-specification changes in the before you write just enough production code to fulfill that
operational environment, user errors, etc [2]. test and refactoring. It is also one way to think through your
Absolute fault-avoidance may not be economical or design before your write your functional code.
feasible. The next best thing is to eliminate faults as soon as The concept of test-driven development has been
they occur, certainly before they propagate into the sporadically used for a long time, e.g. one usage case from as
early as the late 1960’s has been reported. However, it 2.1 Add a test
became popular with the emergence of the development
practice eXtreme Programming (XP). Among the practices In test-driven development, each new feature begins with
included in XP, TDD is considered as one of few that has writing a test. This test must inevitably fail because it is
standalone benefits. The main difference between TDD and a written before the feature has been implemented (if it does
typical test process is that in TDD the developers write the not fail, then the proposed new feature is obviated.) To write
tests before the code. A result of this is that the test cases a test, the developer must clearly understand the feature's
drive the design of the product since it is the test cases that specification and requirements. The developer can
decide what is required of each unit [9]. accomplish this through use cases and user stories that cover
This paper first describes the test driven development cycle the requirements and exception conditions. This could also
highlihgting the needed steps. As further described, there are imply a variant, or modification of an existing test. This is a
several approaches while implementing TDD model as a part differentiating feature of test-driven development versus
of regular software development process. The following writing unit tests after the code is written: it makes the
chapters provide comparison with traditional testing developer focus on the requirements before writing the code,
emphasizing also some advantages of this technique. TDD a subtle but important difference.
implementation in the field is presented in the sixth chapter.
Case study results as well as comparision with some previous 2.2 Run all tests and see if the new one fails
results are also given here.
This validates that the test harness is working correctly and
that the new test does not mistakenly pass without requiring
2. TEST DRIVE& DEVELOPME&T CYCLE any new code. This step also tests the test itself, in the
negative: it rules out the possibility that the new test will
The test cases can be seen as example-based specifications always pass, and therefore be worthless.
of the code. In short, a developer that uses traditional TDD The new test should also fail for the expected reason. This
works in the following way [10]: increases confidence (although it does not entirely guarantee)
1. Write the test case, that it is testing the right thing, and will pass only in intended
2. Execute the test case and verify that it fails as expected, cases.
3. Implement code that makes the test case passes,
4. Refactor the code if necessary. 2.3 Write some code

Test-driven development requires developers to create The next step is to write some code that will cause the test
automated unit tests that define code requirements to pass. The new code written at this stage will not be perfect
(immediately) before writing the code itself. The tests contain and may, for example, pass the test in an inelegant way. That
assertions that are either true or false. Passing the tests is acceptable because later steps will improve and hone it.
confirms correct behavior as developers evolve and refactor
the code. Developers often use testing frameworks to create It is important that the code written is only designed to pass
and automatically run sets of test cases. the test; no further (and therefore untested) functionality
should be predicted and allowed for at any stage.
The following figure presents the TDD cycle:
2.4 Run the automated tests and see them succeed

Test
Repeat
If all test cases now pass, the programmer can be confident
(Re) Write
a test
succeeds
that the code meets all the tested requirements. This is a good
point from which to begin the final step of the cycle.
Check if the
test fails

Test(s)
2.5 Refactor code
Write fail
production
code
Now the code can be cleaned up as necessary. By re-
Run all
tests running the test cases, the developer can be confident that
refactoring is not damaging any existing functionality. The
Clean up
code concept of removing duplication is an important aspect of any
software design. In this case, however, it also applies to
removing any duplication between the test code and the
Figure 1 – A Graphical Representation of the TDD Cycle
production code.
2.6 Repeate whether the software meets their requirements. With ATDD,
the development team now has a specific target to satisfy, the
Starting with another new test, the cycle is then repeated to acceptance tests, which keeps them continuously focused on
push forward the functionality. The size of the steps should what the customer really wants from that user story.
always be small, with as few as 1 to 10 edits between each
test run. If new code does not rapidly satisfy a new test, or
other tests fail unexpectedly, the programmer should undo or 4. TDD A&D TRADITIO&AL TESTI&G
revert in preference to excessive debugging. Continuous
Integration helps by providing revertible checkpoints. When TDD is primarily a design technique with a side effect of
using external libraries it is important not to make increments ensuring that your source code is thoroughly unit tested.
that are so small as to be effectively merely testing the library However, there is more to testing than this. You'll still need
itself [11] unless there is some reason to believe that the to consider other agile testing techniques such as agile
library is buggy or is not sufficiently feature-complete to acceptance testing and investigative testing. Much of this
serve all the needs of the main program being written. testing can also be done early in your project. In fact, in XP
the acceptance tests for a user story are specified by the
project stakeholder(s) either before or in parallel to the code
3. TDD DEVELOPME&T STYLE being written, giving stakeholders the confidence that the
system does in fact meet their requirements.
There are various aspects to using Test Driven With traditional testing a successful test finds one or more
Development. By focusing on writing only the code necessary defects. It is the same with TDD; when a test fails you have
to pass tests, designs can be cleaner and clearer than is often made progress because you now know that you need to
achieved by other methods [10]. resolve the problem. More importantly, you have a clear
To achieve some advanced design concept (such as a measure of success when the test no longer fails. TDD
design pattern), tests are written that will generate that increases your confidence that your system actually meets the
design. The code may remain simpler than the target pattern, requirements defined for it, that your system actually works
but still pass all required tests. This can be unsettling at first and therefore you can proceed with confidence [13].
but it allows the developer to focus only on what is important. As with traditional testing, the greater the risk profile of the
Write the tests first. The tests should be written before the system the more thorough your tests need to be. With both
functionality that is being tested. This has been claimed to traditional testing and TDD you are not striving for
have two benefits. It helps ensure that the application is perfection, instead you are testing to the importance of the
written for testability, as the developers must consider how to system. In other words, you should test with a purpose and
test the application from the outset, rather than worrying know why you are testing something and to what level it
about it later. It also ensures that tests for every feature will needs to be tested. An interesting side effect of TDD is that
be written. When writing feature-first code, there is a you achieve 100% coverage test – every single line of code is
tendency by developers and the development organizations to tested – something that traditional testing doesn’t guarantee.
push the developer onto the next feature, neglecting testing In general, TDD is a specification technique, a valuable side
entirely. effect is that it results in significantly better code testing than
First fail the test cases. The idea is to ensure that the test do traditional techniques.
really works and can catch an error. Once this is shown, the
underlying functionality can be implemented. This has been
coined the test-driven development mantra, known as 5. ADVA&TAGES OF TDD
red/green/refactor where red means fail and green is pass.
Developers list many advantages to working in this fast,
Test-driven development constantly repeats the steps of incremental manner. Highest on the list of advantages to
adding test cases that fail, passing them, and refactoring. practicing TDD are:
Receiving the expected test results at each stage reinforces the
programmer's mental model of the code, boosts confidence • Quick results: Developers can see the effect of
and increases productivity. design decisions within minutes.
Advanced practices of test-driven development can lead to • Flexibility: Changes are easy because of the short
Acceptance Test-Driven Development (ATDD) where the distance between commits.
criteria specified by the customer are automated into • Automatic catalog of regression tests: If
acceptance tests, which then drive the traditional Unit Test- something developed six months ago suddenly
Driven Development (UTDD) process [12]. This process breaks under today's code, it is known immediately.
ensures the customer has an automated mechanism to decide
 • Good, clean code that works early in the development cycle, preventing them from
There are also following advantages: becoming endemic and expensive problems. Eliminating
defects early in the process usually avoids lengthy and tedious
• The suite of unit tests provides constant feedback that debugging later in the project.
each component is still working.
TDD can lead to more modularized, flexible, and
• The unit tests act as documentation that cannot go out- extensible code. This effect often comes about because the
of-date, unlike separate documentation, which can and methodology requires that the developers think of the
frequently does. software in terms of small units that can be written and tested
• When the test passes and the production code is independently and integrated together later. This leads to
refactored to remove duplication, it is clear that the code is smaller, more focused classes, looser coupling, and cleaner
finished, and the developer can move on to a new test. interfaces.
• Test-driven development forces critical analysis and Because no more code is written than necessary to pass a
design because the developer cannot create the production failing test case, automated tests tend to cover every code
code without truly understanding what the desired result path. For example, in order for a TDD developer to add an
should be and how to test it. else branch off an existing if statement, the developer would
first have to write a failing test case that motivates the branch.
• The software tends to be better designed, that is, loosely As a result, the automated tests resulting from TDD tend to be
coupled and easily maintainable, because the developer is very thorough: they will detect any unexpected changes in the
free to make design decisions and refactor at any time with code's behavior. This detects problems that can arise where
confidence that the software is still working. This confidence fixing something later in the development cycle unexpectedly
is gained by running the tests. The need for a design pattern alters other functionality.
may emerge, and the code can be changed at that time.
• The test suite acts as a regression safety net on bugs: If a 6. TDD RESULTS A&ALYSIS
bug is found, the developer should create a test to reveal the
bug and then modify the production code so that the bug goes TDD programming method was applied on SW
away and all other tests still pass. On each successive test run, development project at Ericsson. The main reason for that
all previous bug fixes are verified. was the conclusion that insufficient component testing before
• Reduced debugging time. delivery to integration tests was caused by deadline pressure
that commonly occur shortly before the deliveries. During
Test-driven development offers more than just simple such time pressure, people tend to deliver the code with lower
validation of correctness, but can also drive the design of a quality assurance. To address this problem, a central part of
program. By focusing on the test cases first, one must imagine the process change was to introduce component-level test-
how the functionality will be used by clients (in the first case, driven development. The reason why TDD could make
the test cases). So, the programmer is only concerned with the developers test more efficient is that when writing the test
interface and not the implementation. This benefit is cases before the code it is more likely that the written tests are
complementary to Design by Contract as it approaches code executed before delivery. From this analysis, a proposal for a
through test cases rather than through mathematical assertions concept consisting of component-level test automation and
or preconceptions [1]. TDD was made.
The power test-driven development offers is the ability to The concept was implemented into a new product release
take small steps when required. It allows a programmer to and results were compared with its predecessors. During both
focus on the task at hand as the first goal is to make the test projects personnel turnover was very low and the
pass. Exceptional cases and error handling are not considered development processes were stable.
initially. Tests to create these extraneous circumstances are
implemented separately. Another advantage is, as already The chosen metric was Fault Slip Through i.e. measuring
mentioned, that test-driven development, when used properly, the number of faults per specific phase especially highlighting
ensures that all written code is covered by a test. This can the fault slippage from different development phases. FST
give the programming team, and subsequent users, a greater provides information regarding the number of faults not
level of confidence in the code. detected in a certain activity. These faults have instead been
detected in a later activity.
While it is true that more code is required with TDD than
without TDD because of the unit test code, total code In this particular case slippage from previous project phases
implementation time is typically shorter [14]. Large numbers to Function and System tests was measured. Test results are
of tests help to limit the number of defects in the code. The shown in the following table:
early and frequent nature of the tests helps to catch defects
 Table 1: Fault Statistics – Project Comparison The main disadvantage with TDD is that in the worst case,
the test cases duplicate the amount of code to write and
Without With Improvement maintain. However, this is the same problem as for all kinds
TDD TDD of test automation. Nevertheless, to what extent the amount of
Product FST to 73% 67% 8.2% code increases depends on the granularity of the test cases
Function Test and what module level the test cases encapsulates, e.g. class
Product FST to 57% 46% 19.3% level or component level.
System Test Although gathering data for more accurate statistical
analysis will require longer period of time, initial analysis that
was performed during case study presented in this paper is
As can be seen from the table, slippage from Unit to pointing to some potential areas of improvements in product
Function test and from Function to System test was reduced. life cycle.
Lower result in first case can be explained by fact that one
feature in Unit test was not fully following the new approach
(due to the specific test environment it was tested off-site) REFERE&CES
causing the higher fault slippage than expected.
Our future goal is further enhancing the TDD practice in a [1] L. Williams, E. M. Maximilien, M. Vouk, “Test-Driven Development
as a Defect-Reduction Practice”, Proceedings of International
away to include explicit estimation of the probability that the Symposium on Software Reliability Engineering, pp 66-75, October
software system performs according to its requirements. 1996
Some productivity impact of such approach will also be [2] J. D. Musa, “Software Reliability Engineering”, New York: McGraw-
examined. Hill, 1998.
[3] B. W. Boehm, “Software Engineering Economics”, Englewood Cliffs,
NJ: Prentice-Hall, Inc., 1981.
[4] S. E. Elmaghraby, E. I. Baxter, M. A. Vouk, “An Approach to the
7. CO&CLUSIO& Modeling and Analysis of Software Production Processes”, Intl. Trans.
Operations Res, vol.2, pp. 117-135, 1995.
Test-driven development is a software development [5] M. C. Paulk, B. Curtis, M. B. Chrisis, “Capability Maturity Model for
practice that has been used sporadically for decades. With Software Version 1.1”, Software Engineering Institute CMU/SEI-93-
TR, February 1993.
this practice, test cases (preferably automated) are
[6] W. S. Humprey, “A Discipline for Software Engineering”,
incrementally written before production code is implemented. Massachusetts: Addison Wesley Longman, Inc. 1995.
The most obvious advantage of TDD is the same as for test [7] I. Sommerville, “Software Engineering”, Sixth ed. Harlow, England:
Addison-Wesley, 2001.
automation in general, i.e. the possibility to do continuous
[8] L.-O. Damm, L. Lundberg, “Results from introducing component-level
quality assurance of the code. This gives both instant test automation and Test-driven Development”, Journal of Systems and
feedback to the developers about the state of their code and Software, Volume 79, Issue 7, July 2006.
most likely, a significantly lower percentage of faults left to [9] L.-O. Damm, L. Lundberg, “Early and Cost-Effective Software Fault
be found in later testing and at customer sites. Further, with Detection”, International Conference on Software Engineering,
Proceedings of the 29th international conference on Software
early quality assurance, a common problem with test Engineering, pp. 560–570.
automation is avoided. That is, when an organization [10] K. Beck, “Test-Driven Development by Example”, Addison Wesley,
introduces automated testing late in the development cycle, it 2003
becomes a catch for all faults just before delivery to the [11] JW. Newkirk and AA. Vorontsov , “Test-Driven Development in
customer. The corrections of found faults lead to a spiral of Microsoft. NET, Microsoft Press, 2004.
testing and re-testing which delays the delivery of the [12] L. Koskela, “Test Driven: TDD and Acceptance TDD for Java
Developers”, Manning Publications, 2007
product.
[13] ***, “Techniques for successful Evolutionary/Agile Database
A side effect of TDD is that the resulting tests are working Development”, http://www.agiledata.org
examples for invoking the code, thereby providing a working [14] M. Muller, F. Padberg, “About the Return on Investment of Test-
specification for the code. Driven Development”, Universitat Kalsruhe, pp. 6. http://www.
ipd.uka.de