Relational database management systems (RDBMSs) often persist mission-critical

data which is updated by

many applications and potentially thousands if not millions of end users. Furthermore, they
implement important functionality in the form of database methods (stored procedures, stored
functions, and/or triggers) and database objects (e.g. Java or ! instances). "he best way to
ensure the continuing #uality of these assets, at least from a technical point of view, you should
have a full regression test suite which you can run on a regular basis.
$n this article $ argue for a fully automated, continuous regression testing based approach to
database testing.
Just as agile software developers ta%e this approach to their application code, we should also do
the same for our databases.
1. Why Test an RDBMS?
"here are several reasons why you need to develop a comprehensive testing strategy for your
&'()*+
1. Data is an important corporate asset. 'oesn,t it ma%e sense to invest the effort
re#uired to validate the #uality of data via effective testing- )y July .//0 survey into the current
state of data management indicates that 12.34 of respondents believe that data is a corporate
asset. 5et of them only 6/.74 had a database test suite in place to validate the data and of those
without a test suite only 78.04 had even discussed the concept.
2. Mission-critical bsiness !nctionality is implemented in RDBMSs. $n the sr"ey,
07.34 of respondents indicated that their organi9ations did this, but of those only 604
had regression tests in place to validate the logic. *houldn,t we be doing better-
3. #rrent approaches aren$t s!!icient. "he current state of the art in many
organi9ations is for data professionals to control changes to the database schemas, for
developers to visually inspect the database during construction, and to perform some
form of formal testing during the test phase at the end of the lifecycle. :nfortunately,
none of these approaches prove effective. ;pplication developers will often go around
their organi9ation,s data management group because they find them too difficult to wor%
with, too slow in the way they wor%, or sometimes they don,t even %now they should be
wor%ing together. "he end result is that the teams don,t follow the desired data #uality
procedures and as a result #uality suffers. ;lthough "isal inspection of #uery results
is a good start it is little more than a debugging techni#ue in practice that will help you to
find problems but not prevent them. "esting late in the lifecycle is better than nothing,
but as (arry (oehm noted in the early </s it,s incredibly e%pensi"e to !i% any de!ects
you find at that point.
4. Testing pro"ides the concrete !eedbac& re'ired to identi!y de!ects. =ow do you
%now how good the #uality of your source data actually is without an effective test suite
which you can run whenever you need to-
5. Spport !or e"oltionary de"elopment. )any e"oltionary de"elopment techni#ues,
in particular database re!actoring, are predicated upon the idea that it must be
possible to determine if something in the database has been bro%en when a change has
been made. "he easiest way to do that is to simply run your regression test suite.
(ncom!ortable )estion*
$sn,t it time that we stopped tal%ing about data #uality and actually started doing something about
it-
=ere,s a few interesting #uestions to as% someone who isn,t convinced that you need to test the
'(+
8. $f you,re implementing code in the '( in the form of stored procedures, triggers, ...
shouldn,t you test that code to the same level that you test your app code-
.. "hin% of all the data #uality problems you,ve run into over the years. >ouldn,t it have
been nice if someone had originally tested and discovered those problems before you
did-
7. >ouldn,t it be nice to have a test suite to run so that you could determine how (and if) the
'( actually wor%s-
$ thin% that one of the reasons that we don,t hear much about database testing is because it is a
relatively new idea within the data community. )any traditional data professionals seem to thin%
that testing is something that other people do, particularly test/#uality assurance professionals,
do. "his reflects a penchant for o"er-speciali+ation and a serial approach to,ards
de"elopment by traditionalists, two ideas which have also been shown to be #uestionable
organi9ational approaches at best.
-. What Shold We Test?
Figure 8 indicates what you should consider testing when it comes to relational databases. "he
diagram is drawn from the point of view of a single database, the dashed lines indicate threat
boundaries, indicating that you need to consider threats both within the database (clear bo?
testing) and at the interface to the database (blac% bo? testing). "able 8 lists the issues which
you should consider testing for both internally within the database and at the interface to it. For
details, read the article >hat "o "est in an &'()*.
.igre 1. What to test.
Table 1. What to test in an RDBMS.
Blac&-Bo% Testing at the
/nter!ace
White0#lear-Bo% Testing /nternally Within the Database
@/& mappings
(including the meta
data)
$ncoming data values
@utgoing data values
(from #ueries, stored
functions, views ...)
*caffolding code (e.g. triggers or updateable views) which
support refactorings
"ypical unit tests for your stored procedures, functions, and
triggers
A?istence tests for database schema elements (tables,
procedures, ...)
Biew definitions
Re!erential integrity (R/) rules
'efault values for a column
'ata invariants for a single column
'ata invariants involving several columns

1. When Shold We Test?
;gile software developers ta%e a test-first approach to development where they write a test
before you write just enough production code to fulfill that test. "he steps of test first
development ("F') are overviewed in the :)C activity diagram of .igre -. "he first step is to
#uic%ly add a test, basically just enough code to fail. De?t you run your tests, often the complete
test suite although for sa%e of speed you may decide to run only a subset, to ensure that the new
test does in fact fail. 5ou then update your functional code to ma%e it pass the new tests. "he
fourth step is to run your tests again. $f they fail you need to update your functional code and
retest. @nce the tests pass the ne?t step is to start over.
.igre -. The 2rocess o! Test .irst De"elopment (T.D).
Test-dri"en de"elopment (TDD) is an e"oltionary approach to development which combines
test-first development and re!actoring. >hen an agile software developer goes to implement a
new feature, the first #uestion they as% themselves is E$s this the best design possible which
enables me to add this feature-E $f the answer is yes, then they do the wor% to add the feature. $f
the answer is no then they refactor the design to ma%e it the best possible then they continue with
a "F' approach. "his strategy is applicable to developing both your application code and your
database schema, two things that you would wor% on in parallel.
>hen you first start following a "'' approach to development you #uic%ly discover that to ma%e
it successful you need to automate as much of the process as possible- 'o you really want to
manually run the same build script(s) and the same testing script(s) over and over again- @f
course not. *o, agile developers have created @** tools such as 34T, Ma"en, and #rise
#ontrol (to name a few) which enable them to automate these tas%s. )ore importantly, it
enables them to automate their database testing script into the bild procedure itself.
;gile developers reali9e that testing is so important to their success that it is something they do
every day, not just at the end of the lifecycle. "hey test as often and early as possible, and better
yet they test first. ;s you can see with the agile system de"elopment li!ecycle (SD5#) of
.igre 1 testing is in fact something that occurs during the development and release cycles, not
just during release. Furthermore, many agile software developers reali9e that you can test more
than just your code, you can in fact validate every wor% product created on a software
development project if you choose to. "his philosophy is e?emplified by the .ll 5i!ecycle
6b7ect-6riented Testing (.566T) Methodology.
.igre 1. The 3gile 5i!ecycle.
8. 9o, to Test
;lthough you want to %eep your database testing efforts as simple as possible, at first you will
discover that you have a fair bit of both learning and set up to do. $n this section $ discuss the
need for various database sandbo%es in which people will test+ in short, if you want to do
database testing then you,re going to need test databases (sandbo?es) to wor% in. $ then
overview how to ,rite a database test and more importantly describe setp strategies !or
database tests. Finally, $ overview several database testing tools which you may want to
consider.
8.1 Database Sandbo%es
; common best practice on agile teams is to ensure that developers have their own
Esandbo%esE to wor% in. ; sandbo? is basically a technical environment whose scope is well
defined and respected. .igre 8 depicts the various types of sandbo?es which your team may
choose to wor% in. $n each sandbo? you,ll have a copy of the database. $n the development
sandbo? you,ll e?periment, implement new functionality, and refactor e?isting functionality,
validate your changes through testing, and then eventually you,ll promote your wor% once you,re
happy with it to the project integration sandbo?. $n this sandbo? you will rebuild your system and
then run all the tests to ensure you haven,t bro%en anything (if so, then bac% to the development
sandbo?). @ccasionally, at least once an iteration/cycle, you,ll deploy your wor% to the level
(demo and pre-production testing), and rerun your test suite (including database tests) each time
that you do so to ensure that your changes integrate with the changes made by other developers.
Avery so often (perhaps once every si? to twelve months) into production. "he primary advantage
of sandbo?es are that they help to reduce the ris% of technical errors adversely affecting a larger
group of people than is absolutely necessary at the time.
.igre 8. Sandbo%es.
8.- Writing Database Tests
"here,s no magic when it comes to writing a database test, you write them just li%e you would any
other type of test. 'atabase tests are typically a three-step process+
1. Setp the test. 5ou need to put your database into a %nown state before running tests
against it. "here are se"eral strategies for doing so.
2. Rn the test. :sing a database regression testing tool, run your database tests just
li%e you would run your application tests.
3. #hec& the reslts. 5ou,ll need to be able to do Etable dumpsE to obtain the current
values in the database so that you can compare them against the results which you
e?pected.
"he article What To Test in an RDBMS goes into greater detail.
8.1 Setting p Database Tests
"o successfully your database you must first %now the e?act state of the database, and the best
way to do that is to simply put the database in a %nown state before running your test suite.
"here are two common strategies for doing this+
8. .resh start. ; common practice is to rebuild the database, including both creation of
the schema as well as loading of initial test data, for every major test run (e.g. testing
that you do in your pro7ect integration or pre-prodction test sandbo%es).
.. Data reinitiali+ation. For testing in developer sandbo?es, something that you
should do every time you rebuild the system, you may want to forgo dropping and
rebuilding the database in favor of simply reinitiali9ing the source data. 5ou can do this
either by erasing all e?isting data and then inserting the initial data vales bac% into the
database, or you can simple run updates to reset the data values. "he first approach is
less ris%y and may even be faster for large amounts of data.
;n important part of writing database tests is the creation of test data. 5ou have several
strategies for doing so+
1. 9a"e sorce test data. 5ou can maintain an e?ternal definition of the test data,
perhaps in flat files, F)C files, or a secondary set of tables. "his data would be loaded
in from the e?ternal source as needed.
2. Test data creation scripts. 5ou develop and maintain scripts, perhaps using data
manipulation language (')C) *GC code or simply application source code (e.g. Java or
!), which does the necessary deletions, insertions, and/or updates re#uired to create
the test data.
3. Sel!-contained test cases. Aach individual test case puts the database into a
%nown state re#uired for the test.
"hese approaches to creating test data can be used alone or in combination. ; significant
advantage of writing creation scripts and self-contained test cases is that it is much more li%ely
that the developers of that code will place it under con!igration management (#M) control.
;lthough it is possible to put test data itself under ) control, worst case you generate an e?port
file that you chec% in, this isnHt a common practice and therefore may not occur as fre#uently as
re#uired. hoose an approach that reflects the culture of your organi9ation.
>here does test data come from- For unit testing, $ prefer to create sample data with %nown
values. "his way $ can predict the actual results for the tests that $ do write and $ %now $ have the
appropriate data values for those tests. For other forms of testing -- particularly load/stress,
system integration, and function testing, $ will use live data so as to better simulate real-world
conditions.
Be,are #opling*
@ne danger with database regression testing, and with regression testing in general, is coupling
between tests. $f you put the database into a %nown state, then run several tests against that
%nown state before resetting it, then those tests are potentially coupled to one another. oupling
between tests occurs when one test counts on another one to successfully run so as to put the
database into a %nown state for it. *elf-contained test cases do not suffer from this problem,
although may be potentially slower as a result due to the need for additional initiali9ation steps.
8.8 What Testing Tools 3re 3"ailable?
$ believe that there are several critical features which you need to successfully test &'()*s.
First, as .igre 1 implies you need two categories of database testing tools, one for interface
tests and one for internal database tests. *econd, these testing tools should support the
language that you,re developing in. For e?ample, for internal database testing if you,re a
)icrosoft *GC *erver developer, your "-*GC procedures should li%ely be tested using some form
of "-*GC framewor%. *imilarly, @racle '(;s should have a IC-*GC-based unit testing
framewor%. "hird, you need tools which help you to put your database into a %nown state, which
implies the need not only for test data generation but also for managing that data (li%e other
critical development assets, test data should be under con!igration management control).
"o ma%e a long story short, although we,re starting to see a glimmer of hope when it comes to
database testing tools, as you can see in Table -, but we still have a long way to go. Cuc%ily there
are some good tools being developed by the open source software (@**) community and there
are some commercial tools available as well. =aving said that, $)=@ there is still significant
opportunity for tool vendors to improve their database testing offerings.
Table -. Some database testing tools.
#ategory Description :%amples
:nit testing
tools
"ools which enable you to
regression test your database.
DB.it
DB(nit
4Db(nit
6(nit !or
6racle (being
replaced soon
by )te)
S)5(nit
TS)5(nit (for
testing "-*GC in
)* *GC *erver)
;isal Stdio
Team :dition
!or Database
2ro!essionals
includes testing
capabilities
<T(nit
"esting
tools for
load
testing
"ools simulate high usage loads
on your database, enabling you
to determine whether your
system,s architecture will stand
up to your true production needs.
:mpiri%
Mercry
/nteracti"e
Rad;ie,
Rational Site
Test Stdio
Web
2er!ormance
"est 'ata
Jenerator
'evelopers need test data
against which to validate their
systems. "est data generators
can be particularly useful when
you need large amounts of data,
perhaps for stress and load
testing.
Data .actory
Datatect
DTM Data
=enerator
Trbo Data
>. Who Shold Test?
'uring development cycles, the primary people responsible for doing database testing are
application de"elopers and agile DB3s. "hey will typically pair together, and because they
are hopefully ta%ing a TDD-approach to development the implication is that they,ll be doing
database unit testing on a continuous basis. 'uring the release cycle your testers, if you have
any, will be responsible for the final system testing efforts and therefore they will also be doing
database testing.
"he role of your data management (DM) grop, or $" management if your organi9ation has no
') group, should be to support your database testing efforts. "hey should promote the concept
that database testing is important, should help people get the re#uisite training that they re#uire,
and should help obtain database testing tools for your organi9ation. ;s you have seen,
database testing is something that is done continuously by the people on development teams, it
isn,t something that is done by another group (e?cept of course for system testing efforts). $n
short, the ') group needs to support database testing efforts and then get out of the way of the
people who are actually doing the wor%.
?. /ntrodcing Database Regression Testing into @or 6rgani+ation
'atabase testing is new to many people, and as a result you are li%ely to face several challenges+
8. /ns!!icient testing s&ills. "his problem can be overcome through training, through
pairing with someone with good testing s%ills (pairing a '(; without testing s%ills and a
tester without '(; s%ills still wor%s), or simply through trial and error. "he important
thing is that you recogni9e that you need to pic% up these s%ills.
.. /ns!!icient nit tests !or e%isting databases. Few organi9ations have yet to adopt the
practice of database testing, so it is li%ely that you will not have a sufficient test suite for
your e?isting database(s). ;lthough this is unfortunate, there is no better time than the
present to start writing your test suite.
7. /ns!!icient database testing tools. ;s $ said earlier, we still have a way to go with
respect to tools.
6. Reticent DM grops. )y e?perience is that some data management (')) groups may
see the introduction of database regression testing, and agile techni#ues such as test-
!irst de"elopment (T.D) and re!actoring, as a threat. @r, as my July .//0 Estate o!
data managementE survey shows, a large percentage of organi9ations are not only not
doing any database testing at all they haven,t even discussed it. For many in the data
management community the idea of doing database testing is rather new and it,s simply
going to ta%e a while for them to thin% it through. $,m not so sure that you should wait to
do such obvious process impro"ement.
$n general, $ highly suggest that you read my article 3dopting :"oltionary03gile Database
Techni'es and consider buying the boo% .earless #hange which describes a pattern language
for successfully implementing change within organi9ations.
A. Database Testing and Data /nspection
; common #uality techni#ue s to use data inspection tools to e?amine e?isting data within a
database. 5ou might use something as simple as a *GC-based #uery tool such as DB /nspect to
select a subset of the data within a database to visually inspect the results. For e?ample, you
may choose to view the uni#ue values in a column to determine what values are stored in it, or
compare the row count of a table with the count of the resulting rows from joining the table with
another one. $f the two counts are the same then you don,t have an &$ problem across the join.
;s Richard Dalla,ay points out, the problem with data inspection is that it is often done
manually and on an irregular basis. >hen you ma%e changes later, sometimes months or years
later, you need to redo your inspection efforts. "his is costly, time consuming, and error prone.
'ata inspection is more of a debugging techni#ue than it is a testing techni#ue. $t is clearly an
important techni#ue, but it,s not something that will greatly contribute to your efforts to ensure
data #uality within your organi9ation.
B. Best 2ractices
$,d li%e to conclude this article by sharing a few database testing Ebest practicesE with you+
1. (se an in-memory database !or regression testing. 5ou can dramatically
speed up your database tests by running them, or at least portions of them, against an
in-memory database such as =*GC'(. "he challenge with this approach is that
because database methods are implemented differently across database vendors that
any method tests will still need to run against the actual database server.
2. Start !resh each ma7or test rn. "o ensure a clean database, a common
strategy is that at the beginning of each test run you drop the database, then rebuild it
from scratch ta%ing into account all database refactorings and transformations to that
point, then reload the test data, and then run your tests. @f course, you wouldn,t do this
to your production database. K-)
3. Ta&e a continos approach to regression testing. $ can,t say this enough,
a "'' approach to development is an incredibly effective way to wor%.
4. Train people in testing. )any developers and '(;s have not been trained in
testing s%ills, and they almost certainly haven,t been trained in database testing s%ills.
$nvest in your people, and give them the training and education they need to do their
jobs.
5. 2air ,ith no"ices ,ith people that ha"e database testing e%perience.
@ne of the easiest ways to gain database testing s%ills is to pair program with someone
who already has them.