Chapter 1 Test Automation Framework

CHAPTER 1
TEST AUTOMATION FRAMEWORK
This section gives the introduction about test automation framework, various types of
the framework and the analysis of the best suitable framework for the application under test
(the application under test is referred as AUT). This also includes the detailed description of
the format of the input (the input to the framework is referred as the test tables) that is given
to our test automation framework.
1.1 RECORD/PLAYBACK MYTH
The test automation tool vendors market their product as the main feature of the tool
is the ability to capture the user actions and later to playback them. Here is the basic
paradigm for U!"based automated regression testing # the so called $ecord%&layback
method (also called as 'apture%$eplay approach)
(. )esign a test case in the test management tool.
*. Using the capture feature of the automation testing tool record the user actions. The
result is a macro"like script where each user action is presented.
+. ,nhance the recorded script with verification points, where some property or data is
verified against an e-isting baseline. Add delay and wait states points where the different
actions are synchroni.ed.
/. &layback the scripts and observe the results in the log of the test management tool.
The basic drawback in this method is the scripts resulting from this method contain
hard"coded values which must change if anything at all changes in our AUT. The costs
associated with maintaining such scripts are astronomical, and unacceptable. These scripts
are not reliable, even if the application has not changed, and often fail on replay (pop"up
windows, messages, and other things can happen that did not happen when the test was
recorded).
Chapter 1 Test Automation Framework
!f the tester makes an error entering data, etc., the test must be re"recorded. !f the
application changes the test must be re"recorded. All that is being tested are things that
already work. Areas that have errors are encountered in the recording process (which is
manual testing, after all). These bugs are reported, but a script cannot be recorded until the
software is corrected. 0o logically nothing is tested by this approach.
0o, avoid using 1$ecord%&layback1 as a method of automating testing. This method
is fraught with problems, and is the most costly (time consuming) of all methods over the
long term. The record%playback feature of the test tool is useful for determining how the tool
is trying to process or interact with the application under test, and can give us some ideas
about how to develop your test scripts, but beyond that, its usefulness ends 2uickly.
1.2 TYPES OF TEST AUTOMATION FRAMEWORKS
As we have eliminated $ecord%&layback method, let us e-plore about the e-isting
automation methodologies. There are several test automation frameworks available, among
these the selection is made based on the factors such as reusability of both the scripts and
the test assets. The different test automation frameworks available are as follows,
Test 0cript 3odularity
Test 4ibrary Architecture
)ata")riven Testing
5eyword")riven or Table")riven Testing
Hybrid Test Automation
1.2.1 Test Scrit M!"#$%rit&
The test script modularity framework is the most basic of the frameworks. !t6s a well"
known programming strategy to build an abstraction layer in front of a component to hide the
component from the rest of the application. This insulates the application from modifications
in the component and provides modularity in the application design. 7hen working with test
scripts (in any language or proprietary environment) this can be achieved by creating small,
independent scripts that represent modules, sections, and functions of the application"
Play with WinRunner by recording and playing it back.
Refer the WinRunner Tutorial & WinRunner User Guide
Chapter 1 Test Automation Framework
under"test. Then these small scripts are taken and combined them in a hierarchical fashion
to construct larger tests. The use of this framework will yield a higher degree of
modulari.ation and add to the overall maintainability of the test scripts.
1.2.2 Test Li'r%r& Arc(itect#re
The test library architecture framework is very similar to the test script modularity
framework and offers the same advantages, but it divides the application"under"test into
procedures and functions (or ob8ects and methods depending on the implementation
language) instead of scripts. This framework re2uires the creation of library files (09A:asic
libraries, A&!s, )44s, and such) that represent modules, sections, and functions of the
application"under"test. These library files are then called directly from the test case script.
3uch like script modulari.ation this framework also yields a high degree of modulari.ation
and adds to the overall maintainability of the tests.
1.2.) D%t%*Dri+e, Testi,-
A data"driven framework is where test input and output values are read from data
files (;):' sources, '<0 files, ,-cel files, )A; ob8ects, A); ob8ects, and such) and are
loaded into variables in captured or manually coded scripts. !n this framework, variables are
used for both input values and output verification values. =avigation through the program,
reading of the data files, and logging of test status and information are all coded in the test
script. This is similar to table"driven testing (which is discussed shortly) in that the test case
is contained in the data file and not in the script> the script is 8ust a 1driver,1 or delivery
mechanism, for the data. !n data"driven testing, only test data is contained in the data files.
1.2.).1 Merits !. "%t%*"ri+e, testi,-
The merits of the )ata")riven test automation framework are as follows,
0cripts may be developed while application development is still in progress
Utili.ing a modular design, and using files or records to both input and verify
data, reduces redundancy and duplication of effort in creating automated test
scripts
!f functionality changes, only the specific 1:usiness ?unction1 script needs to
be updated
Chapter 1 Test Automation Framework
)ata input%output and e-pected results are stored as easily maintainable te-t
records.
?unctions return 1T$U,1 or 1?A40,1 values to the calling script, rather than
aborting, allowing for more effective error handling, and increasing the
robustness of the test scripts. This, along with a well"designed 1recovery1
routine, enables 1unattended1 e-ecution of test scripts.
1.2.).2 De/erits !. "%t%*"ri+e, testi,-
The demerits of the )ata")riven test automation framework are as follows,
$e2uires proficiency in the 0cripting language used by the tool (technical
personnel)
3ultiple data"files are re2uired for each Test 'ase. There may be any
number of data"inputs and verifications re2uired, depending on how many
different screens are accessed. This usually re2uires data"files to be kept in
separate directories by Test 'ase
Tester must not only maintain the )etail Test &lan with specific data, but
must also re"enter this data in the various re2uired data"files
!f a simple 1te-t editor1 such as =otepad is used to create and maintain the
data"files, careful attention must be paid to the format re2uired by the
scripts%functions that process the files, or script"processing errors will occur
due to data"file format and%or content being incorrect
1.2.0 Ke&1!r"*Dri+e, Testi,-
This re2uires the development of data tables and keywords, independent of the test
automation tool used to e-ecute them and the test script code that 1drives1 the application"
under"test and the data. 5eyword"driven tests look very similar to manual test cases. !n a
keyword"driven test, the functionality of the application"under"test is documented in a table
as well as in step"by"step instructions for each test. !n this method, the entire process is
data"driven, including functionality.
1.2.0.1 E2%/$e
!n order to open a window, the following table is devised, and it can be used for any
other application, 8ust it re2uires 8ust changing the window name.
Chapter 1 Test Automation Framework
Test T%'$e .!r Oe,i,- % Wi,"!1
Wi,"!1 C!,tr!$ Acti!, Ar-#/e,ts
7indow =ame 3enu 'lick ?ile, ;pen
7indow =ame 3enu 'lick 'lose
7indow =ame &ushbutton 'lick ?older =ame
7indow =ame <erify $esults
;nce creating the test tables, a driver script or a set of scripts is written that reads in
each step e-ecutes the step based on the keyword contained the Action field, performs error
checking, and logs any relevant information.
1.2.0.2 Merits !. 3e&1!r" "ri+e, testi,-
The merits of the 5eyword )riven Testing are as follows,
The )etail Test &lan can be written in 0preadsheet format containing all input
and verification data.
!f 1utility1 scripts can be created by someone proficient in the automated tool@s
0cripting language prior to the )etail Test &lan being written, then the tester
can use the Automated Test Tool immediately via the 1spreadsheet"input1
method, without needing to learn the 0cripting language.
The tester need only learn the 15ey 7ords1 re2uired, and the specific format
to use within the Test &lan. This allows the tester to be productive with the
test tool very 2uickly, and allows more e-tensive training in the test tool to be
scheduled at a more convenient time.
1.2.0.) De/erits !. 3e&1!r" "ri+e, testi,-
The demerits of the 5eyword )riven Testing are as follows,
)evelopment of 1customi.ed1 (Application"0pecific) ?unctions and Utilities
re2uires proficiency in the tool@s 0cripting language. (=ote that this is also
true for any method)
!f application re2uires more than a few 1customi.ed1 Utilities, this will re2uire
the tester to learn a number of 15ey 7ords1 and special formats. This can be
time"consuming, and may have an initial impact on Test &lan )evelopment.
Chapter 1 Test Automation Framework
;nce the testers get used to this, however, the time re2uired to produce a
test case is greatly improved.
1.2.4 H&'ri" Test A#t!/%ti!, Fr%/e1!r3
The most commonly implemented framework is a combination of all of the above
techni2ues, pulling from their strengths and trying to mitigate their weaknesses. This hybrid
test automation framework is what most frameworks evolve into over time and multiple
pro8ects. The most successful automation frameworks generally accommodate both
5eyword")riven testing as well as )ata")riven scripts.
This allows data driven scripts to take advantage of the powerful libraries and utilities
that usually accompany a keyword driven architecture. The framework utilities can make the
data driven scripts more compact and less prone to failure than they otherwise would have
been.
The utilities can also facilitate the gradual and manageable conversion of e-isting
scripts to keyword driven e2uivalents when and where that appears desirable. ;n the other
hand, the framework can use scripts to perform some tasks that might be too difficult to re"
implement in a pure keyword driven approach, or where the keyword driven capabilities are
not yet in place. The following sections describe its architecture, merits and demerits.
1.2.4.1 H&'ri" Test A#t!/%ti!, Fr%/e1!r3 Arc(itect#re
The framework is defined by the 'ore )ata )riven ,ngine, the 'omponent
?unctions, and the 0upport 4ibraries. 7hile the 0upport 4ibraries provide generic routines
useful even outside the conte-t of a keyword driven framework, the core engine and
component functions are highly dependent on the e-istence of all three elements.
The test e-ecution starts with the 4AU='H T,0T(() script. This script invokes the
'ore )ata )riven ,ngine by providing one or more High"4evel Test Tables to
'ycle)river(*). 'ycle)river processes these test tables invoking the 0uite)river(+) for each
!ntermediate"4evel Test Table it encounters. 0uite)river processes these intermediate"level
tables invoking 0tep)river(/) for each 4ow"4evel Test Table it encounters. As 0tep)river
processes these low"level tables it attempts to keep the application in synch with the test.
7hen 0tep)river encounters a low"level command for a specific component, it determines
Our test automation framework is a
ybrid Test !utomation "ramework
The !pplication #ap is referred to !pp #ap. This !pp #ap in WR!"$
is the !pplication #ap file created from GU% #ap of WinRunner
Chapter 1 Test Automation Framework
what Type of component is involved and invokes the corresponding 'omponent ?unction(A)
module to handle the task.
All
of
these elements of the framework rely on the information provided in the App 3ap to
interface or bridge the automation framework with the application being tested. The App
3ap is the only means by which the framework could identify the ob8ects in the application
under test. ,ach of these elements is described in more detail in the following sections. The
following figure shows the diagrammatic representation of the Hybrid Test Automation
?ramework.


ybrid Test !utomation "ramework
Chapter 1 Test Automation Framework
APPLICATION MAP
The Application 3ap is one of the most critical components, which is used for
mapping the ob8ects from names humans can recogni.e to a data format useful for the
automation tool. ?or a given pro8ect it is needed to define a naming convention or specific
names for each component in each window as well as a name for the window itself. Then
use the Application 3ap to associate that name to the identification method needed by the
automation tool to locate and properly manipulate the correct ob8ect in the window.
Application 3ap not only gives the ability to provide useful names for the ob8ects, it
also enables the scripts and keyword driven tests to have a single point of maintenance on
the ob8ect identification strings. Thus, if a new version of an application changes the title of
the window or label of the components or the inde- of an image element within it, they
should not affect the test tables. The changes will re2uire only a 2uick modification in one
place""inside the Application 3ap.
COMPONENT FUNCTIONS
'omponent ?unctions are those functions that actively manipulate or interrogate
component ob8ects. !n test automation framework there are different 'omponent ?unction
modules for each type of component that are encountered (7indow, 'heck:o-, Te-t:o-,
!mage, 4ink, etc,).
'omponent ?unction modules are the application"independent e-tensions applied to
the functions already provided by the automation tool. However, unlike those provided by
the tool, the e-tra code to help with error detection, error correction, and synchroni.ation are
added. These modules can readily use the application"specific data stored in the Application
3ap and test tables as necessary. !n this way, these 'omponent ?unctions are developed
once and are used again and again by every application tested.
Another benefit from 'omponent ?unctions is that they provide a layer of insulation
between the application and the automation tool. 7ithout this e-tra layer, changes or
1enhancements1 in the automation tool itself can break e-isting scripts and the table driven
tests. ,ach 'omponent ?unction modules will define the keywords or 1action words1 that are
valid for the particular component type it handles.
The component ?unctions takes the windows name in which the component resides,
the actual component name on which the action is to be performed, the values needed for
performing the action and the type of action to be performed as its arguments. The
Chapter 1 Test Automation Framework
'omponent ?unction keywords and their arguments define the low"level vocabulary and
individual record formats will be used to develop the test tables.
TEST TABLES
The input to the framework apart from the application map are the test tables, which holds
the arguments needed for the 'omponent ?unctions and other information. There are three
levels in which the test tables are organi.ed, they are as follows,
4ow"4evel Test Tables (or) 0tep Tables
!ntermediate"4evel Test Tables (or) 0uite Tables
High"4evel Test Tables (or) 'ycle Tables.
LOW*LE5EL TEST TABLES
4ow"level Test Tables or 0tep Tables contain the detailed step"by"step instructions
of the tests. Using the ob8ect names found in the Application 3ap, and the vocabulary
defined by the 'omponent ?unctions> these tables specify what window, what component,
and what action to take on the component. The columns in the 0tep Tables are as follows,
Action 'ommand
7indow =ame
'omponent =ame
<alues =eed to &erform the 0pecified Action
The 0tep)river module is the one that initially parses and routes all low"level
instructions that ultimately drive our application.
INTERMEDIATE*LE5EL TEST TABLES
!ntermediate"level Test Tables or 0uite Tables do not normally contain such low"level
instructions. !nstead, these tables typically combine 0tep Tables into 0uites in order to
perform more useful tasks. The same 0tep Tables may be used in many 0uites. !n this way
the minimum numbers of 0tep Tables necessary are developed. Then they are organi.ed in
0uites according to the purpose and design of the tests, for ma-imum reusability. The
columns in the 0uite Tables are as follows,
0tep Table =ame
0pecific Arguments to be &assed to the 0tep Tables
The 0uite Tables are handled by the 0uite)river module which parses each record in
the 0uite Table and passes each 0tep Table to the 0tep)river module for processing.
Chapter 1 Test Automation Framework
HI6HER*LE5EL TEST TABLES
High"level Test Tables or 'ycle Tables combine intermediate"level 0uites into 'ycles.
The 0uites can be combined in different ways depending upon the testing 'ycle which is
efficient to e-ecute. ,ach 'ycle will likely specify a different type or number of tests. The
columns in the 'ycle Tables are as follows,
0uite Table =ame
0pecific Arguments to be &assed to the 0uite Table
These 'ycles are handled by the 'ycle)river module which passes each 0uite to
0uite)river for processing.
CORE DATA DRI5EN EN6INE
The 'ore )ata )riven ,ngine is the primary part of the framework and it has three
main modules, they are as follows
0tep)river
0uite)river
'ycle)river
'ycle)river processes 'ycles, which are high"level tables listing 0uites of tests to
e-ecute. 'ycle)river reads each record from the 'ycle Table, passing 0uite)river each
0uite Table it finds during this process. 0uite)river processes these 0uites, which are
intermediate"level tables listing 0tep Tables to e-ecute. 0uite)river reads each record from
the 0uite Table, passing 0tep)river each 0tep Table it finds during this process. The
following figure represents the 'ore )ata )riven ,ngine,

C!re D%t% Dri+e E,-i,e
0tep)river processes these 0tep Tables, which are records of low"level instructions
developed in the keyword vocabulary of the 'omponent ?unctions. 0tep)river parses these
records and performs some initial error detection, correction, and synchroni.ation making
certain that the window andBor the component planned to manipulate is available and active.
Chapter 1 Test Automation Framework
0tep)river then routes the complete instruction record to the appropriate 'omponent
?unction for final e-ecution.
SUPPORT LIBRARIES
The 0upport 4ibraries are the general"purpose routines and utilities that let the
overall automation framework do what it needs to do. They are the modules that provide
services like,
?ile Handling
0tring Handling
:uffer Handling
<ariable Handling
)atabase Access
4ogging Utilities
0ystemB,nvironment Handling
Application 3apping ?unctions
0ystem 3essaging or 0ystem A&! ,nhancements and 7rappers
They also provide traditional automation tool scripts access to the features of our
automation framework including the Application 3ap functions and the keyword driven
engine itself. :oth of these items can vastly improve the reliability and robustness of these
scripts until such time that they can be converted over to keyword driven test tables.