SILKTEST AND WINRUNNER

FEATURE DESCRIPTIONS

Horwath/Green/Lawler
Version 1.00 (7/6/00)
SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

Document Revision History

This section contains a description of each revision made to this document.
X.10 (5/21/00 tjh) initial draft.
X.11 (6/13/00 tjh) updated with J. Green’s comments and corrections.
1.00 (6/30/00 tjh) updated with T. Lawler’s comments and corrections.

ii Horwath/Green/Lawler
SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

Table of Contents

1. INTRODUCTION 1
1.1 Author Backgrounds ............................................................................................................ 1
2. FEATURE DESCRIPTIONS 2
2.1 Startup Initialization and Configuration .............................................................................. 2
2.2 Test Termination.................................................................................................................. 3
2.3 Addins and Extensions ........................................................................................................ 3
2.4 Visual Recorders.................................................................................................................. 4
2.5 Object Hierarchy.................................................................................................................. 5
2.6 Object Recognition .............................................................................................................. 5
2.7 Object Verification .............................................................................................................. 6
2.8 Custom Objects.................................................................................................................... 6
2.9 Internationalization (Language Localization) ..................................................................... 7
2.10 Database Interfaces.............................................................................................................. 7
2.11 Database Verification .......................................................................................................... 7
2.12 Data Driven Testing............................................................................................................. 8
2.13 Restoring an Application’s Initial State .............................................................................. 8
2.14 Scripting Language.............................................................................................................. 9
2.15 Exception Handling ............................................................................................................. 9
2.16 Test Results Analysis......................................................................................................... 10
2.17 Managing the Testing Process........................................................................................... 10
2.18 External Files..................................................................................................................... 11
2.19 Debugging.......................................................................................................................... 11

iii Horwath/Green/Lawler
SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

1. INTRODUCTION
This document provides feature descriptions of two GUI automated test tools and their
associated test management software:
• Mercury-Interactive’s WinRunner, version 6.02 (with TestDirector)
• Segue’s SilkTest, version 5.03 (with SilkOrganizer)

The goal of this document is to describe as objectively as possible the capabilities provided by
each tool for each feature, rather than trying to state that one tool’s implementation of a feature is
better than the other. The feature descriptions made in this document are intentionally terse and
condensed—therefore subtle nuances and deep details of a feature are not covered.
This document by itself should only be considered a good starting point for a complete test tool
evaluation by those readers who are considering one of these tools for their automated testing
efforts. Many excellent evaluation criteria ideas can be found in the AUTOMATED TESTING forum
on the www.betasoft.com website. Assigning a qualitative value to each feature is an evaluation
task which depends on the test team’s:
• testing environment,
• programming skills, and the
• application testing requirements.

There is no substitute for using each tool to develop a specific set of prototype testcases, in order
to determine the strength and weaknesses of each tool against a typical application in your
environment.

1.1 Author Backgrounds

The following people developed this report:
• Terry Horwath is the primary author and a test automation consultant. He has worked with
SilkTest, QaPartner, and SilkOrganizer since 1994. He has worked for the last year with
WinRunner and TestDirector on two large web testing projects and is a TestSuite 6.0
certified product specialist (CPS). Terry can be contacted at [email protected].
• John Green is a contributing author and owns Automation Expertise Inc., a company which
provides customized test automation solutions. He is a recognized expert using SilkTest,
QaPartner, and SilkOrganizer, working with these products since 1993. John can be
contacted at [email protected].
• Tom Lawler is a contributing author and is a test automation consultant. He has worked
with web, VB, PowerBuilder and Terminal Emulation (TE) applications using WinRunner
and TestDirector since 1999. He is a TestSuite 6.0 certified product specialist (CPS) in both
GUI and TE and can be contacted at [email protected].

©Copyright 2000 1 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2. FEATURE DESCRIPTIONS
For the sake of consistency alphabetical ordering was selected to describe SilkTest features first,
followed by WinRunner features in each of the following sections.

2.1 Startup Initialization and Configuration

• SilkTest derives its initial startup configuration settings from its partner.ini file. This
though is not important because SilkTest can be reconfigured at any point in the session by
either changing any setting in the Options menu or loading an Option Set.
An Option Set file (*.opt) permits customized configuration settings to be established for
each test project. The project specific Option Set is then be loaded [either interactively, or
under program control] prior to the execution of the project’s testcases.
The Options menu or an Option Set can also be used to load an include file (*.inc)
containing the project’s GUI Declarations [discussed in section 2.6 on page 5], along with
any number of other include files containing library functions, methods, and variables shared
by all testcases.
• WinRunner derives its initial startup configuration from a wrun.ini file of settings.
During startup the user is normally polled [this can be disabled] for the type of addins they
want to use during the session [refer to section 2.3 on page 3 for more information about
addins].
The default wrun.ini file is used when starting WinRunner directly, while project specific
initializations can be established by creating desktop shortcuts which reference a project
specific wrun.ini file. The use of customized wrun.ini files is important because once
WinRunner is started with a selected set of addins you must terminate WinRunner and restart
it to use a different set of addins.
The startup implementation supports the notion of a startup test which can be executed
during WinRunner initialization. This allows project-specific compiled modules [memory
resident libraries] and GUI Maps [discussed in section 2.6 on page 5] to be loaded. The
functions and variables contained in these modules can then be used by all tests that are run
during that WinRunner session.
Both tools allow most of the configuration setup established in these files to be over-ridden with
runtime code in library functions or the test scripts.

©Copyright 2000 2 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.2 Test Termination

• SilkTest tests terminate on exceptions which are not explicitly trapped in the testcase. For
example if a window fails to appear during the setup phase of testing [i.e. the phase driving
the application to a verification point], a test would terminate on the first object or window
timeout exception that is thrown after the errant window fails to appear.
• WinRunner tests run to termination [in unattended Batch mode] unless an explicit action is
taken to terminate the test early. Therefore tests which ignore this termination model will
continue running for long periods of time after a fatal error is encountered. For example if a
window fails to appear during the setup phase of testing, subsequent context sensitive
statements [i.e. clicking on a button, performing a menu pick, etc.] will fail—but this failure
occurs after a multi-second object/window “is not present” timeout expires for each missing
window and object. [When executing tests in non-Batch mode, that is in Debug, Verify, or
Update modes, WinRunner normally presents an interactive dialog box when implicit errors
such as missing objects and windows are encountered].

2.3 Addins and Extensions

Out of the box, under Windows, both tools can interrogate and work with objects and windows
created with the standard Microsoft Foundation Class (MFC) library. Objects and windows
created using a non-MFC technology [or non-standard class naming conventions] are treated as
custom objects. Dealing with truly custom objects is discussed further in section 2.8 on page 6.
But objects and windows created for web applications [i.e. applications which run in a browser],
Java applications, Visual Basic applications, and PowerBuilder applications are dealt with in a
special manner:
• SilkTest enables support for these technologies using optional extensions. Selected
extensions are enabled/disabled in the current Option Set [or the configuration established by
the default partner.ini option settings].
• WinRunner enables support for these technologies using optional addins. Selected addins
are enabled/disabled using either the Addin Manager at WinRunner startup, or by editing the
appropriate wrun.ini file prior to startup.
Note that (1) some combinations of addins [WinRunner] and extensions [SilkTest] are mutually
exclusive, (2) some of these addins/extensions may no longer be supported in the newest releases
of the tool, (3) some of these addins/extensions may only support the last one or two releases of
the technology [for example version 5 and 6 of Visual Basic] and (4) some of these addins and
extensions may have to be purchased at an addition cost.

©Copyright 2000 3 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.4 Visual Recorders

SilkTest provides visual recorders and wizards for the following activities:
• Creating a test frame with GUI declarations for a full application and adding/deleting
selective objects and windows in and existing GUI declarations frame file.
• Capturing user actions with the application into a test case, using either context sensitive
[object relative] or analog [X:Y screen coordinate relative] recording techniques.
• Inspecting identifiers, locations and physical tags of windows and objects.
• Checking window and object bitmaps [or parts thereof].
• Creating a verification statement [validating one or more object properties].

WinRunner provides visual recorders and wizards for the following activities:
• Creating an entire GUI Map for a full application and adding/deleting selective objects and
windows in an existing GUI Map. It is also possible to implicitly create GUI Map entries by
capturing user actions [using the recorder described next].
• Capturing user actions with the application into a test case, using either context sensitive
[object relative] or analog [X:Y screen coordinate relative] recording techniques.
• Inspecting logical names, locations and physical descriptions of windows and objects.
• Checking window and object bitmaps [or parts thereof].
• Creating a GUI checkpoint [validating one or more object properties].
• Creating a database checkpoint [validating information in a database].
• Creating a database query [extracting information from a database].
• Locating at runtime a missing object referenced in a testcase [and then adding that object to
the GUI Map].
• Teaching WinRunner to recognize a virtual object [a bitmap graphic with functionality].
• Creating Data Tables [used to drive a test from data stored in an Excel-like spreadsheet].
• Checking text on a non-text object [using a built-in character recognition capability].
• Creating a synchronization point in a testcase.
• Defining an exception handler.

Some of these recorders and wizards do not work completely for either tool against all
applications, under all conditions. For example neither tool’s recorder to create a full GUI Map
[WinRunner] or test frame [SilkTest] works against large applications, or any web application.
Evaluate the recorders and wizards of interest carefully against your applications if these utilities
are important to your automated testing efforts.

©Copyright 2000 4 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.5 Object Hierarchy

• SilkTest supports a true object-oriented hierarchy of parent-child-grandchild-etc.
relationships between windows and objects within windows. In this model an object such as
a menu is the child of its enclosing window and a parent to its menu item objects.
• WinRunner, with some rare exceptions [often nested tables on web pages], has a flat object
hierarchy where child objects exist in parent windows. Note that web page frames are treated
as windows, and not child objects of the enclosing window on web pages that are constructed
using frames.

2.6 Object Recognition

Both of these tools use a lookup table mechanism to isolate the variable name used to reference
an object in a test script from the description used by the operating system to access that object at
runtime:
• SilkTest normally places an application’s GUI declarations in a test frame file. There is
generally one GUI declaration for each window and each object in a window. A GUI
declaration consists of an object identifier—the variable used in a test script—and its class
and object tag definition used by the operating system to access that object at runtime.
SilkTest provides the following capabilities to define an object tag: (1) a string, which can
include wildcards; (2) an array reference which resolves to a string which can include
wildcards; (3) a function or method call that returns a string, which can include wildcards,
(4) an object class and class relative index number; and (5) multiple tags [multi-tags] each
optionally conditioned with (6) an OS/GUI/browser specifier [a qualification label].
• WinRunner normally places an application’s logical name/physical descriptor definitions in
a GUI Map file. There is generally one logical name/physical descriptor definition for each
window and each object in a window. The logical name is used to reference the object in a
test script, while the physical descriptor is used by the operating system to access that object
at runtime.
WinRunner provides the following capabilities to define a physical descriptor: (1) a variable
number of comma delimited strings which can include wildcards, where each string
identifies one property of the object. [While there is only a single method of defining a
physical descriptor, this definition can include a wide range and variable number of
obligatory, optional, and selector properties on an object by object basis].
The notion behind this lookup table mechanism is to permit changes to an object tag [SilkTest]
or a physical descriptor [WinRunner] definition without the need to change the associated
identifier [SilkTest] or logical name [WinRunner] used in the testcase.
In general the object tag [SilkTest] or physical descriptor [WinRunner] resolve to one or more
property definitions which uniquely identify the object in the context of its enclosing parent
window at runtime.
It is also possible with both tools to dynamically construct and use object tags [SilkTest] or
physical descriptors [WinRunner] at runtime to reference objects in test scripts.

©Copyright 2000 5 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.7 Object Verification

Both tools provide a variety of built-in library functions permitting a user to hand code simple
verification of a single object property [i.e. is/is not focused, is/is not enabled, has/does not have
an expected text value, etc.]. Complex multiple properties in a single object and multiple object
verifications are supported using visual recorders:
• SilkTest provides a Verify Window recorder which allows any combination of objects and
object properties in the currently displayed window to be selected and captured. Using this
tool results in the creation, within the testcase, of a VerifyProperties() method call
against the captured window.
• WinRunner provides several GUI Checkpoint recorders to validate (1) a single object
property, (2) multiple properties in a single object, and (3) multiple properties of multiple
objects in a window. The single property recorder places a verification statement directly in
the test code while the multiple property recorders create unique checklist [*.ckl] files in
the /chklists subdirectory [which describe the objects and properties to capture], as well
as an associated expected results [*.chk] file in the /exp subdirectory [which contains the
expected value of each object and property defined in the checklist file].
Both tools offer advanced features to define new verification properties [using mapping
techniques and/or built-in functions and/or external DLL functions] and/or to customize how
existing properties are captured for standard objects.

2.8 Custom Objects

Note: The description of this feature, more than any other in this report, is limited in its scope and
coverage. An entire white paper could be dedicated to exploring and describing how each of
these tools deal with custom objects. Therefore dedicate several days to evaluating how each
of these tools accommodate custom objects in your specific applications.

To deal with a custom object [i.e. an object that does not map to standard class] both tools
support the use of class mapping [i.e. mapping a custom class to a standard class with like
functionality], along with a variety of X:Y pixel coordinate clicking techniques [some screen
absolute, some object relative] to deal with bitmap objects, as well as the ability to use external
DLL functions to assist in object identification and verification. Beyond these shared capabilities
each tool has the following unique custom object capabilities:
• SilkTest has a feature to overlay a logical grid of X rows by Y columns on a graphic that has
evenly spaced “hot spots”[this grid definition is then used to define logical GUI declarations
for each hot spot]. These X:Y row/column coordinates are resolution independent [i.e. the
logical reference says “give me 4th column thing in the 2nd row”, where that grid expands or
contracts depending on screen resolution].
• WinRunner has a built-in text recognition engine which works with most standard fonts.
This capability can often be used to extract visible text from custom objects, position the
cursor over a custom object, etc. This engine can also be taught non-standard font types
which is does understand out of the box.
Both tools offer support for testing non-graphical controls through the advanced use of custom
DLLs [developed by the user], or the Extension Kit [SilkTest, which may have to be purchased at
an addition cost] and the Mercury API Function Library [WinRunner].

©Copyright 2000 6 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.9 Internationalization (Language Localization)

• SilkTest supports the single-byte IBM extended ASCII character set, and its Technical
Support has also indicated “that Segue has no commitment for unicode”. The user guide
chapter titled “Supporting Internationalized Applications” shows a straightforward technique
for supporting X number of [single-byte IBM extended ASCII character set] languages in a
single test frame of GUI declarations.
• WinRunner provides no documentation on how to use the product to test language localized
applications. Technical Support has indicated that (1) “WinRunner supports multi-byte
character sets for language localized testing…”, (2) “there is currently no commitment for
the unicode character set…”, and (3) “it is possible to convert a US English GUI Map to
another language using a [user developed] phrase dictionary and various gui_* built-in
functions”.
Check into the aspects of this feature very carefully if it is important to your testing effort.

2.10 Database Interfaces

Both tools provide a variety of built-in functions to perform Structure Query Language (SQL)
queries to control, alter, and extract information from any database which supports the Open
Database Connectivity (ODBC) interface.

2.11 Database Verification

Both tools provide a variety of built-in functions to make SQL queries to extract information
from an ODBC compliant database and save it to a variable [or if you wish, an external file].
Verification at this level is done with hand coding.
WinRunner also provides a visual recorder to create a Database Checkpoint used to validate the
selected contents of an ODBC compliant database within a testcase. This recorder creates side
files similar to GUI Checkpoints and has a built-in interface to (1) the Microsoft Query
application [which can be installed as part of Microsoft Office], and (2) to the Data Junction
application [which may have to be purchased at an addition cost], to assist in constructing and
saving complex database queries.

©Copyright 2000 7 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.12 Data Driven Testing

Both tools support the notion of data-driven tests, but implement this feature differently:
• SilkTest’s implementation is built around an array of user defined records. A record is a
data structure defining an arbitrary number of data items which are populated with values
when the array is initialized [statically or at runtime]. Non-programmers can think of an
array of records as a memory resident spreadsheet of X number of rows which contain Y
number columns where each row/column intersection contains a data item.
The test code, as well as the array itself, must be hand coded. It is also possible to populate
the array each time the test is run by extracting the array’s data values from an ODBC
compliant database, using a series of built-in SQL function calls. The test then iterates
through the array such that each iteration of the test uses the data items from the next record
in the array to drive the test or provide expected data values.
• WinRunner’s implementation is built around an Excel compatible spreadsheet file of X
number of rows which contain Y number of columns where each row/column intersection
contains a data item. This spreadsheet is referred to as a Data Table.
The test code, as well as the Data Table itself, can be created with hand coding or the use of
the DataDriver visual recorder. It is also possible to populate a Data Table file each time the
test is run by extracting the table’s data values from an ODBC compliant database using a
WinRunner wizard interfaced to the Microsoft Query application.. The test then iterates
through the Data Table such that each iteration of the test uses the data items from the next
row in the table to drive the test or provide expected data values.
Both tools also support the capability to pass data values into a testcase for a more modest
approach to data driving a test.

2.13 Restoring an Application’s Initial State

• SilkTest provides a built-in recovery system which restores the application to a stable state,
referred to as the basestate, when the test or application under test fails ungracefully. The
default basestate is defined to be: (1) the application under test is running; (2) the application
is not minimized; and (3) only the application’s main window is open and active. There are
many built-in functions and features which allow the test engineer to modify, extend, and
customize the recovery system to meet the needs of each application under test.
• WinRunner does not provide a built-in recovery system. You need to code routines to return
the application under test to its basestate—and dismiss all orphaned dialogs—when a test
fails ungracefully.

©Copyright 2000 8 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.14 Scripting Language

Both tools provide proprietary, interpreted, scripting languages. Each language provide the usual
flow control constructs, arithmetic and logical operators, and a variety of built-in library
functions to perform such activities as string manipulation, [limited] regular expression support,
standard input and output, etc. But that is where the similarity ends:
• SilkTest provides a strongly typed, object-oriented programming language called 4Test.
Variables and constants may be one of 19 built-in data types, along with a user defined
record data type. 4Test supports single- and multi-dimensional dynamic arrays and lists,
which can be initialized statically or dynamically. Exception handling is built into the
language [via the do… except statement].
• WinRunner provides a non-typed, C-like procedural programming language called TSL.
Variables and constants are either numbers or strings [conversion between these two types
occurs dynamically, depending on usage]. There is no user defined data type such as a record
or a data structure. TSL supports sparsely populated associative single- and [pseudo] multi-
dimension arrays, which can be initialized statically or dynamically—element access is
always done using string references—foobar[“1”] and foobar[1] access the same
element [as the second access is implicitly converted to an associative string index
reference]. Exception handling is not built into the language.
The only way to truly understand and appreciate the differences between these two programming
environments is to use and experiment with both of them.

2.15 Exception Handling

Both tools provide the ability to handle unexpected events and errors, referred to as exceptions,
but implement this feature differently:
• SilkTest’s exception handling is built into the 4Test language—using the do… except
construct you can handle the exception locally, instead of [implicitly] using SilkTest’s
default built-in exception handler [which terminates the currently running test and logs an
error]. If an exception is raised within the do block of the statement control is then
immediately passed to the except block of code. A variety of built-in functions
[LogError(), LogWarning, ExceptNum(), ExceptLog(), etc.] and 4Test statements
[raise, reraise, etc.] aid in the processing of trapped exceptions within the except block
of code.
• WinRunner’s exception handling is built around (1) defining an exception based on the type
of exception (Popup, TSL, or object) and relevant characteristics about the exeception (most
often its error number); (2) writing an exception hander, and (3) enabling and disabling that
exception handler at appropriate point(s) in the code. These tasks can be achieved by hand
coding or through the use of the Exception Handling visual recorder.

©Copyright 2000 9 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.16 Test Results Analysis

• SilkTest’s test results file resolves around the test run. For example if you run 3 testcases
[via a test suite or SilkOrganizer] all of the information for that test run will be stored in a
single results file. There is a viewer to analyze the results of the last test run or X previous to
the last run. Errors captured in the results file contain a full stack trace to the failing line of
code, which can be brought up in the editor by double-clicking on any line in that stack trace.
• WinRunner’s test results file revolves around each testcase. For example if you run 3
testcases [by hand or via a batch test or TestDirector] 3 test results files are created, each in a
subdirectory under its associated testcase. There is a viewer to analyze the results of a test’s
last run or if results have not been deleted, a previous run. Double clicking on events in the
log often expands that entry’s information, sometimes bringing up specialized viewers [for
example when that event is some type of checkpoint or some type of failure].

2.17 Managing the Testing Process

• SilkTest has a built-in facility, SilkOrganizer, for creating a testplan and then linking the
testplan to testcases. SilkOrganizer can also be used to track the automation process and
control the execution of selected groups of testcases. One or more user defined attributes
[such as “Test Category”, “Author”, “Module”, etc.] are assigned to each testcase and then
later used in testplan queries to select a group of tests for execution. There is also a modest
capability to add manual test placeholders in the testplan, and then manually add pass/fail
status to the results of a full test run. SilkTest also supports a test suite, which is a file
containing calls to one or more test scripts or other test suites.
• WinRunner integrates with a separate program called TestDirector [at a substantial
additional cost], for visually creating a test project and then linking WinRunner testcases into
that project. TestDirector is a database repository based application that provides a variety of
tools to analyze and manipulate the various database tables and test results stored in the
repository for the project. A bug reporting and tracking tool is included with TestDirector as
well [and this bug tracking tool supports a browser based client].
Using a visual recorder, testcases are added to one or more test sets [such as “Test
Category”, “Author”, “Module”, etc.] for execution as a group. There is a robust capability
for authoring manual test cases [i.e. describing of each test step and its expected results],
interactively executing each manual test, and then saving the pass/fail status for each test
step in the repository. TestDirector also allows test sets to be scheduled for execution at a
time and date in the future, as well as executing tests remotely on other machines [this last
capability requires the Enterprise version of TestDirector].
TestDirector is also capable of interfacing with and executing LoadRunner test scripts as
well as other 3rd party test scripts [but this later capability requires custom programming via
TestDirector APIs]. Additionally TestDirector provides API’s to allow WinRunner as well as
other 3rd party test tools [and programming environments] to interface with a TestDirector
database.

©Copyright 2000 10 Horwath/Green/Lawler

SilkTest and WinRunner Feature Descriptions Version 1.00 (7/6/00)

2.18 External Files

When the tool’s source code files are managed with a code control tool such as PVCS or Visual
SourceSafe, it is useful to understand what external side files are created:
• SilkTest implicitly creates *.*o bytecode-like executables after interpreting the source code
contained in testcases and include files [but it unlikely that most people will want to source
code control these files]. No side files are created in the course of using its recorders.
SilkTest does though create an explicit *.bmp files for storing the expected and actual
captured bitmap images when performing a bitmap verification.
• WinRunner implicitly creates many side files using a variety of extensions [*.eve, *.hdr,
*.asc, *.ckl, *.chk, and a few others] in a variety of implicitly created subdirectories
[/db, /exp, /chklist, /resX] under the testcase in the course of using its visual recorders
as well as storing pass/fail results at runtime.

2.19 Debugging
Both tools support a visual debugger with the typical capabilities of breakpoints, single step, run
to, step into, set out of, etc.

©Copyright 2000 11 Horwath/Green/Lawler