Manual Testopia
Manual Testopia
By Greg Hendricks
[email protected]
http://mozilla.org/projects/testopia
Table of Contents
Introduction......................................................................................................................................5
Testopia - Test Case Management................................................................................5
Testopia and Bugzilla..................................................................................................................5
The Mozilla Foundation and the Testopia Project......................................................................5
Installation........................................................................................................................................6
Requirements at a Glance............................................................................................................6
Recommended version Where to get it........................................................6
Database.....................................................................................................6
Additional Perl Modules............................................................................6
Libraries.....................................................................................................6
Client..........................................................................................................6
What You Need to Know............................................................................................................6
Downloading and Installing........................................................................................................7
Upgrading From Testopia 1.0.x..................................................................................................7
Configuration...................................................................................................................................7
Using Testopia.................................................................................................................................8
Why Test?...................................................................................................................................8
What Types of Testing can Testopia Help You With?...............................................................9
Black Box Testing................................................................................................................10
White box and Automated Testing.......................................................................................10
Getting Started...............................................................................................................................10
The Big Picture.....................................................................................................................10
Test Plans..................................................................................................................................11
Test Cases..................................................................................................................................11
Test Runs...................................................................................................................................11
Test Run Environments.............................................................................................................11
Builds........................................................................................................................................12
Test Case-Runs..........................................................................................................................12
Testing In Testopia........................................................................................................................12
Start With a Plan.......................................................................................................................12
Viewing A Test Plan...................................................................................................13
Attaching Files............................................................................................................13
Editing Plan Fields......................................................................................................13
Viewing Plan History..................................................................................................13
Adding Categories and Builds..................................................................................................13
Creating Test Cases........................................................................................................................14
Shortcut – Creating a Test Case..................................................................................14
Adding and Removing Components and Tags...........................................................15
Viewing Test Case-Run Results.................................................................................15
Attaching Files............................................................................................................15
Attaching Bugs............................................................................................................15
Editing Case Fields.....................................................................................................15
Test Case Dependencies..............................................................................................15
Creating Environments..............................................................................................................15
Environment Administration................................................................................................16
Categories........................................................................................................................16
Elements..........................................................................................................................16
Properties.........................................................................................................................17
Property Values...............................................................................................................17
Creating Your Environment......................................................................................................17
Creating A Test Run......................................................................................................................18
Shortcut – Creating a Test Run...................................................................................18
Viewing Your Test Run .............................................................................................18
Adding Cases..............................................................................................................19
Editing Test Run Fields...............................................................................................19
Running Your Tests.......................................................................................................................19
Filtering Your Test Cases in a Run.............................................................................19
Sorting Test Cases.......................................................................................................20
Passing and Failing Test Cases...................................................................................20
Adding Notes..............................................................................................................20
Attaching Bugs............................................................................................................20
Reassigning Tests........................................................................................................20
Changing Build or Environment on a Test Case.........................................................21
Classic View...............................................................................................................21
Deleting Case-Runs.....................................................................................................21
Updating Multiple Cases at Once...............................................................................21
Wrapping Up...............................................................................................................22
Getting Around Testopia................................................................................................................22
Searching...................................................................................................................................23
Sorting Search Results................................................................................................23
Paging..........................................................................................................................23
Batch Processing.........................................................................................................23
Saved Searches............................................................................................................23
Reporting...................................................................................................................................23
Plan Reports................................................................................................................24
Build Coverage Report.............................................................................24
Top Bugs..................................................................................................24
Bugs Found In This Plan..........................................................................24
Printable Percentage Report.....................................................................24
Test Case Reports........................................................................................................24
Estimated vs Actual Time........................................................................24
Historic Status Breakdown.......................................................................24
Cloning......................................................................................................................................24
Cloning Test Plans......................................................................................................25
Cloning Test Cases......................................................................................................25
Cloning a Run.............................................................................................................25
Deleting.....................................................................................................................................26
Test Cases.................................................................................................26
Test Plans.................................................................................................26
Test Runs..................................................................................................26
Importing and Exporting................................................................................................................26
Importing Test Cases..................................................................................................27
Testopia Security...........................................................................................................................27
The Testers Group.....................................................................................................................27
Test Plan Access Control Lists.................................................................................................27
User Regular Expression.............................................................................................27
Explicit Inclusion........................................................................................................28
Access Rights............................................................................................................................28
Read ...........................................................................................................................28
Write ...........................................................................................................................28
Delete .........................................................................................................................28
Admin .........................................................................................................................28
Tags................................................................................................................................................29
Adding Tags to an Object...........................................................................................29
Viewing Tags..............................................................................................................29
Using Testopia With Automated Test Scripts...............................................................................29
Field Descriptions..........................................................................................................................29
Test Plans..............................................................................................................................29
Test Cases.............................................................................................................................30
Test Runs..............................................................................................................................32
Test Case-Runs.....................................................................................................................33
Glossary.........................................................................................................................................35
Getting Help...................................................................................................................................38
Bibliography..................................................................................................................................39
Introduction
Testing, especially software testing, can be a tedious and overwhelming task. Often there are
many levels of testing requirements and many iterations during a project life cycle. Keeping
track of the results of tests is important both to the customer, and the developer. Customers want
to know that their requirements are being met and developers want to know if there are bugs that
they missed. Testing is an important part of the software development process since “Software
should be predictable and consistent, offering no surprises to users” (Meyers 2004, ch 1).
Requirements at a Glance
Database
MySQL 5.0 http://mysql.com
Libraries
Ext JS toolkit 2.0.1 http://extjs.com
Client
Firefox 2.0+ http://getfirefox.com
These instructions assume you are installing on a fresh Bugzilla on Linux/Unix systems. For
more detailed instructions, and instructions for other operating systems such as Windows, please
see the README file in the testopia directory.
You should now be able to See the Testopia links in the Bugzilla footer after logging into
Bugzilla. Make sure you update the new Testers group to include any users that are going to need
access to test cases.
Upgrading
To upgrade an existing release of Testopia, you can simply follow the steps outlined above. after
first reversing the current patch. You can also upgrade to the latest CVS tip using the
cvs_update.sh script located in the testopia/scripts directory. Fore more, see the README file.
Upgrading from Testrunner (v0.7 and prior) is not supported.
Configuration
Testopia adds a number of additional parameters to the Bugzilla Parameters page. They are
located under the Testopia tab. If you are installing Testopia with a new installation of Bugzilla,
you should be sure to set the urlbase parameter. Following is a list of required Testopia
parameters and their explanations:
allow-test-deletion
If this option is on, users can delete objects including plans and cases. Delete rights are
maintained by the plan access control lists.
testopia-allow-group-member-deletes
If this option is on, members of the Testers group will be allowed to delete test objects. This
overrides settings for individuals in the Testers group.
testopia-default-plan-testers-regexp
This is the default regular expression for granting access to new test plans.
default-test-case-status
Default status for newly created test cases.
testopia-max-allowed-plan-testers
Limit to how many users a plan access regular expression should match.
Using Testopia
Why Test?
The following example illustrates why software testing is so important:
Taken from http://techiecorner.blogspot.com .
In March 1992, a man living in Newtown near Boston, Massachusetts, received a bill
for his as yet unused credit card stating that he owed $0.00. He ignored it and threw
it away.
The following month, our hero decided that it was about time that he tried out the
troublesome credit card figuring that if there were purchases on his account it would
put an end to his ridiculous predicament. However, in the first store that he produced
his credit card in payment for his purchases, he found that his card had been
canceled.
He called the credit card company who apologized for the computer error once again
and said that they would take care of it. The next day he got a bill for $0.00 stating
that payment was now overdue. Assuming that, having spoken to the credit card
company only the previous day, the latest bill was yet another mistake, he ignored it,
trusting that the company would be as good as their word and sort the problem out.
The next month, he got a bill for $0.00 stating that he had 10 days to pay his account
or the company would have to take steps to recover the debt.
Finally giving in, he thought he would play the company at their own game and
mailed them a cheque for $0.00. The computer duly processed his account and
returned a statement to the effect that he now owed the credit card company nothing
at all.
A week later, the man's bank called him asking him what he was doing writing a
cheque for $0.00. After a lengthy explanation, the bank replied that the $0.00 cheque
had caused their cheque processing software to fail. The bank could now not process
ANY cheques from ANY of their customers that day because the cheque for $0.00
was causing the bank's computer to crash.
The following month, the man received a letter from the credit card company
claiming that his cheque had bounced and that he now owed them $0.00 and unless
he sent a cheque by return of post them would be taking steps to recover the debt.
Though never proven to be true, this rather humorous anecdote explains the impact that
untested software can have. In the case of both the credit card company and the bank, this
seemingly minor issue may have cost both institutions significant sums of money in lost
transactions and down time, to say nothing of the frustration of the customer.
Many times software errors cause only minor inconveniences. Other times they can have
serious consequences. Think of the life support systems on the space shuttle or the flight
control systems on an airliner. Software has become such an integral part of our lives that
many times we don't realize how much we depend on it until something goes wrong.
What Types of Testing can Testopia Help You With?
There are two general classifications of software testing: black box and white box.
Black box testing is a strategy in which testing is based solely on the requirements
and specifications. Unlike its complement, white box testing, black box testing
requires no knowledge of the internal paths, structure, or implementation of the
software under test (Copeland, 2004,Sec I).
Since Testopia is an extension to Bugzilla, it relies on many of the same objects that Bugzilla
uses to track bugs. In order to use Testopia you first need to set up products and versions using
the Bugzilla product editor. Testopia makes use of Bugzilla's target milestones feature as well so
you may want to turn this optional field on in the parameters.
Testopia is comprised of several objects that are interdependent. Together, they make managing
the testing procedure possible. Lets look at each of these in turn.
Test Plans
At the top of the Testopia hierarchy are test plans. Before you can do anything else in Testopia,
you need a test plan. Test plans are associated with a single product in Bugzilla, though you can
have multiple test plans for each product. Your test plan will serve as the storage point for all
related test cases and test runs and it will act as the dashboard for your testing. It will also serve
to determine who will have access to update test cases.
Test Cases
Test cases are the heart of all testing. Test cases spell out what steps to take while running a test
and what outcomes to expect. If a particular run through the steps fails to produce the expected
outcome, the test fails. Test cases are semi-independent in Testopia. Each test case can be
associated with multiple test plans so care should be taken to ensure that updating a test case
does not interfere with testing in another test plan than your own. A list of associated test plans is
displayed with each test case though, so this should not be too difficult.
Test cases can be divided into categories. You can define as many categories for your product as
you like from the test plan page. These should not be confused with components however. Each
product in Bugzilla is divided into components and you can apply multiple components to each
test case, however each test case can only belong to one category at a time.
Test Runs
Test runs are the focal point of the testing effort. Once you have defined a set of test cases, you
are ready to run through those tests in a test run. Each run is associated with a single test plan
and can consist of any number of test cases from that plan. Before we can run any test however,
we need to have a couple more items first.
Builds
Software development is usually an iterative affair. Developers write code which is then
compiled and included in a system. As bugs and enhancement requests come in, the developer
rewrites portions over again to fix or enhance a product. In Testopia, each iteration is called a
build. Builds are often associated with milestones of the project. This is reflected in the build's
relationship to Bugzilla's target milestones object. Regardless of whether you use target
milestones in Bugzilla, you need to define at least one build for your product before you can
begin a test run.
Test Case-Runs
A test case-run is the record of how a particular test case fared in a particular run for a given
build in a given environment. When you create a test run, records for each test case in that run
are created. By default these take on the build and environment of the test run, however, it is
possible to change these attributes on a particular case-run, essentially creating a new case-run
for each combination. This is desirable in situations where most test cases are fairly generic in
environmental scope but single test cases might require specific conditions. It also aids in
iterative testing as each test case then be updated in a single run rather than having to create new
runs to test the failed cases.
Testing In Testopia
Attaching Files
Either during creation or once you have created your test plan, you can upload attachments to it.
To create an attachment, simply click the browse button and locate the file you wish to attach,
type a description in the Description field and click the attach button.
Attachments are treated very much the same as they are in Bugzilla. The size limit will be
determined by the max-attachment-size parameter the same as it is for Bugzilla
To edit an attachment click the edit link. This allows you to set the description, filename, and
mime-type for the attachment You can view it by clicking the ID link if it is a viewable type or
download it otherwise.
Attaching Files
Just like test plans, you can attach files to your test cases. This works the same as for test plans.
Attaching Bugs
Unlike test plans, you have the option of attaching Bugzilla bugs to your test cases. Each test
case can have multiple bugs attached. To attach a bug, enter the bug number in the field and click
the Add button. A table of attached bugs is displayed.
Creating Environments
As mentioned above, environments are the where in testing. In software testing in particular, this
might include such things as which operating system and what hardware platform a test was
conducted on.. Environments can be as broad or narrow as you define them. The most basic
environment consists of an OS and platform chosen from Bugzilla's lists of these objects.
However it can be much more complicated such as a suite of applications and other products. It
could be a browser or other client package.
Creating an environment in Testopia requires two steps. The first involves defining a set of
variables to be used in your environment. The second is to create the environment from the set of
possible elements.
Environment Administration
When you first install Testopia, you must first define the set of environment variables that will
be used to construct your environments. When editing an environment, the environment
variables are arranged in a hierarchy of objects that is represented as a tree. There are four major
levels: Category, Element, Property, and Property Values.
Categories
Environment Categories are similar to test case categories in that they provide a sorting
mechanism for your environment elements. Each category is associated with a single product or
in the special bucket labeled –ALL-- . The –All-- denotes all products, meaning it holds
categories of elements that are not specific to any product.
When you first install Testopia you will see that the –ALL-- bucket contains two Categories, OS
and Platform. Expanding these you will see that there are elements representing each of the OS
and Platform values defined in Bugzilla. This is list is generated at the time you first install
Testopia and is maintained separately from the Bugzilla lists thereafter.
To create a category, right click on the product or –ALL-- and choose Add Category. You can
then click on the newly create category in the tree which will pop up a form that allows you to
edit the category name or change the product it is associated with.
Elements
Once you have a category defined for your product or the –ALL-- bucket, you can add elements
to that category. Elements are the crux of what makes up your environment. To create an
element, right click on the category you wish to add it to and choose create element.
This will create an element labeled “New Element” which you can edit by clicking on it in the
tree, or right clicking and choosing Edit.
Elements can be nested inside other elements. To create a sub element, right click on the element
and choose Add Element. You can edit this child element in the same manner as its parent. You
can create as many levels of elements as you need to represent the complexity of your
environment.
Properties
Properties describe your element. You can add properties to your elements by right clicking the
element and choosing Add Property. You can add as many properties to your elements as you
need. Properties cannot be nested however.
To edit your property, click on it in the tree or right click on it and choose Edit.
Property Values
Once you have defined a property for your element, you will need to provide a list of values
from which to select for you environment. Right click on your property and choose Add Value to
create a value for your property. You can edit property values by right clicking and choosing
Edit, or by clicking on one of the values under your property.
You can change the name or reorder the list of values from the form provided. You must hit Save
Changes in order for you changes to be committed.
Adding Cases
As mentioned earlier, you can add test cases to your run by clicking the Add Cases button below
the case-run table. This will take you back to the list of test cases you saw when creating your
test run excluding any test cases in your plan that are already in this run.
Editing Test Run Fields
As with test cases and test plans, you can update any of the values you applied when you created
the test run in the form at the bottom of the page. Note that changing the build and environment
will not affect the case-runs already in the run but it will apply to any test cases added after the
change. Changes to these fields are tracked and a history is kept much the same way as with test
plans and test cases. Clicking the History button will display this history.
Adding Notes
Updating the status will add a line to the notes field with the time of the change and who made it.
You can add additional notes by typing them in the Add Note field and clicking Append Note.
Attaching Bugs
If a test case has failed, or for any other reason, you may want to attach a bug to this test. You
can attach existing bugs or create new ones. To attach an existing bug, type the bug number in
the box and click Attach Bug. The bug will be displayed in the Bugs Detected field. If you wish
to log a new bug, click the New button. You will be taken to the enter bug page with information
about the test case already provided in the form. You can now provide any additional details and
submit the bug. This is done in a new window so that you do not lose your place in testing. To
return to your run, close the new window.
Reassigning Tests
The assignee field is used to help testers track their own test cases. If you wish to change the
assignee of a particular test case, you can do so by entering their Bugzilla user name in the
assignee field and clicking Assign. Updating test cases is not limited by assignee. Anyone with
rights to edit the case-run can pass or fail the test. If a test case is closed (PASSED, FAILED, or
BLOCKED), the name of the person who performed the action is captured in the Tested By
field.
Deleting Case-Runs
Sometimes you might mistakingly add a test case to a run that does not belong or you may select
an invalid build and environment combination. In these cases it is easiest to remove them by
deleting the case-run all together. You must have the proper permissions to do so and the
administrator needs to allow this in the parameters. To delete a single case run or a group of
them, select it from the list and click the Delete button in the toolbar. You will be prompted to
confirm the action before the record is removed.
Wrapping Up
Once all tests in a run are complete, you should set the status on the run to STOPPED. This will
prevent further updates to case-runs in the run.
Object Prefix
TEST CASE (Optional for cases)
• case
Object Prefix
• TC
• c
TAGS • tag
So for example, to find test run 45 you could type r 45 or tr 45 or run 45.
To find environments with 'Linux' in the name you could type e:linux or TE Linux.
Searching
Bugzilla has very powerful searching capabilities. Testopia has applied the Bugzilla search
engine to searching for test cases, plans, runs, environments, and case-runs. Each of these has its
own set of search parameters that can be used to narrow the scope of a search. To perform a
search, click the Search link in the footer and then click the tab representing the object you wish
to search from.. From here you select the parameters you wish to query on and click the Submit
button. This will take you to the corresponding list page for that object.
Paging
By default, Testopia will display 25 records at a time. You can use the links at the bottom of the
list to navigate from one page of search results to another or use the Jump To Page box to jump
directly to a page of results. You change the number of records you view at a time by selecting
an option from the page size drop down. Lastly, if there are not too many records, you can opt to
view all of the records at once using the View All link.
Batch Processing
From the search results page of cases, runs, plans, and case-runs, you can update a batch of
objects at once. Do this by selecting which objects to update right clicking and choosing edit. For
example, with test cases, you can apply changes to virtually any fields as well as add these cases
to test runs or link them to a set of plans. You can also export test cases as CSV or XML. More
on this later.
Saved Searches
Like Bugzilla, Testopia offers the user the opportunity to save a search for later retrieval. You do
this by clicking the save button in the toolbar, typing a name in the save search as field and
clicking the Save button. Saved searches appear below the other Testopia links in the footer and
from the searches tab on the product dashboard. You can remove a saved search by selecting it in
the searches tab and right clicking and choosing Delete saved search.
Reporting
As with searching, Testopia has modified the Bugzilla reporting engine to provide basic tabular
reports based on a set of search criteria. These reports can be run on test cases, runs, plans, and
case-runs. To run a report, click the Reports menu on the product dashboard and click the tab
corresponding to the object you wish to report on. Select the fields for the horizontal and vertical
axes of the report and then select your search criteria. The default report is in the form of tabular
data. You can then switch between bar and line graph modes or export the data as a CSV
(comma separated values) file which can be opened in a spreadsheet.
In addition to these generic reports, certain objects have other reports associated with them.
These are accessed from the object pages directly.
Plan Reports
Completion Reports
Right clicking a selection of runs from the run tab or builds from the build tab in the product
dashboard and going to the reports menu allows you to view a report of case-runs by run or
build. As with searches, these reports can be saved. Click the tool icon in the dashboard portal
above the report and select save.
Top Bugs
This report displays the bugs found in test cases within the test plan ordered by the bugs attached
to the most test cases. In other words, which bugs are failing the most test cases.
Cloning
Often it is desirable to copy test information for use in a new development cycle. Testopia allows
you to clone test cases, test run, and whole test plans, allowing you more efficiently move from
one product release to another. Cloning is performed by clicking the Clone button on the object
you wish to clone.
Cloning a Run
To clone a test run:
1. Navigate to the test run you wish to clone and click the Clone button.
2. Enter a summary for the copy.
3. Choose a plan within the product to clone to.
4. Choose a product version and build.
5. Select clone options.
6. Click the Clone button.
In addition to the options to copy the tags and making yourself the manager of the cloned run,
you have the option of copying over the test cases. You can limit the clone to only include test
cases of a certain status. For instance you may want to only include cases that failed in the
cloned run.
Deleting
It is possible to delete test cases, plans, runs and case-runs. To delete, you must have the allow-
test-deletion parameter turned on and have the proper rights. Bugzilla admins (members of the
admin group) can delete regardless of the parameter settings and have rights to delete anything in
the Testopia system.
To delete an object, click the Delete button on that objects page. You will be taken to a
confirmation screen detailing what other objects will be deleted, if any, as a result of deleting this
object. To confirm, click the Delete This Test button. Deleting in Testopia is irreversible. If you
wish to simply hide an object instead you can do so using the methods outlined below:
Test Cases
To hide a test case from search results, place it in the DISABLED status.
Test Plans
To hide test plans from search results, you can archive them by clicking the Archive button on
the plan page.
Test Runs
To hide test runs, place them in the STOPPED state.
Testopia Security
Like Bugzilla, Testopia provides several methods for restricting access to test data. Because test
data is often more forward looking than bug reports, Testopia requires that users log in to see any
test objects. There are several layers of security that a user must pass through to view those
objects however. These are determined in part by Bugzilla group membership, but mostly by the
test plan access control list.
Explicit Inclusion
If you do not wish to grant access to a whole group at once, you can add individual users by
entering their Bugzilla login in the field provided and clicking the Add User button. This allows
the most fine grained control as to who can do what within your test plan. However, if you add a
user that matches the regular expression they will have the greater of the two rights.
Access Rights
Users on the test plan access control lists can be granted rights to read, write, delete, and admin
test plans and their associated objects.
Read
Allows viewing rights to the plan and all test cases, test runs, and test case-runs associated
with it. Test cases linked to more than one plan will be visible to users in both plans.
Write
Implies Read. Allows rights to modify the plan and associated cases, runs, and case-runs.
Test cases linked to more than one plan will not be writable unless the user has write rights
in all plans.
Delete
Implies Read and Write. Allows rights to delete the plan and associated cases, runs, and
case-runs. Test cases linked to more than one plan will not be deletable unless the user has
delete rights in all plans.
Admin
Implies Read, Write, and Delete. Allows rights to modify the plan's access controls.
Tags
Testopia utilizes a tagging mechanism to help organize test objects. Unlike the traditional
method of placing things into virtual folders, tags allow a many-to-many relationship by
allowing multiple objects to have the same tag while allowing each object to have multiple tags.
Tags are a novel approach to categorizing objects. They have become very popular with
numerous popular web 2.0 tools and have met with much success. Tags are similar to keywords
in Bugzilla but do not require an administrator to create. The act of tagging an object creates the
tag which can then be used anywhere.
The idea behind tags is that each user can categorize each item to his or her own liking without
destroying other users' categorizations. Test cases, plans, and runs can all have tags associated
with them. Though tags are not directly associated with a product, Testopia uses a smart typing
approach to recommend tags based on relationships to other objects within the same product.
Tags have a knowledge of who added them to a particular object as well, allowing users to
manage the tags that they have created.
Viewing Tags
To see a list of your tags click the Tags link in the header above the tags section in any object.
From this screen you have the option to look up tags based on product or entered by another
user.
Field Descriptions
Test Plans
Name
Short descriptor for a test plan. Does not need to be unique.
Product
Bugzilla product that this plan is associated with.
Product Version
The Bugzilla product version. Used as the default for new runs.
Type
The testing type for this test plan. Possible values include system, integration, unit, functional
and acceptance. This list is modifiable by the Bugzilla admins from the Testopai Admin link.
Archive
If this bit is set to true, the plan will be archived and not display in searches by default.To
archive a plan click the Archive button. To unarchive click Unarchive.
Plan Document
This is document that spells out the type of testing and testing methods used for this plan.
Tags
See tags description above.
Test Cases
Summary
A short description of the test case.
Default Tester
The person who will be assigned to this test case when a new run is created in which this
case is included.
Alias
A unique string that can be used to identify this test case. This can be used in place of the test
case ID but must be globally unique to the database.
Requirement
The requirement number or URL to a document containing the requirement this test case is
designed to test.
Status
Test case status determines whether this test case can be included in new test runs.
● PROPOSED – This test case is a work in progress and has not been reviewed for
accuracy. It is therefore not ready to be included in test runs.
● CONFIRMED – Test case has passed review and is ready to be included in future test
runs. Only test cases with this status can be included in new test runs.
● DISABLED – This test case is no longer applicable to current testing.
Priority
This denotes the level of testing. Higher priority test cases should be run first and more often
then lower priorities.
Category
The product category that this test case belongs to. Each product has a default category.
Additional categories can be added which can be used to further classify your test cases.
Because this is a product attribute, this list will be the same for all plans in a given product.
Estimated Time
The estimated time (in HH:MM:SS format) that this test case should take to complete.
Add To Runs
Entering a run ID here will include this test case in a test run if it is not already included. You
can also enter a comma separated list of run numbers to add to multiple runs at once.
Automatic
Test cases can be either Automatic or Manual. Automatic test cases are run by a script while
manual test cases are performed by a tester.
Script
If this is an automatic test case, you can enter the name of the script that runs this test case.
TESTOPIA DOES NOT RUN YOUR SCRIPT FOR YOU. You must run your script and
capture the results to send to Testopia. You can use the XMLRPC interface to have your
script update the test case-run results automatically.
Arguments
If there are specific arguments sent to the script that apply to this test case, they can be stored
here. TESTOPIA DOES NOT RUN YOUR SCRIPTS FOR YOU. This field is only used to
store the information. The XMLRPC can be used by your script to capture this from the test
case.
Attach Bugs
You can attach bugs to your test cases. This field accepts a comma separated list of bug
numbers.
Depends on
If this test case requires other test cases be run before this one, their case numbers should
appear here. Enter a list of comma separated case numbers to add dependencies.
Blocks
If this test case prevents others from being run their ID numbers will appear here. This field
accepts a comma separated list of case numbers.
Set Up
This field details the steps that are necessary to prepare to run a test
Break Down
This field details the steps required to reset the test case to a known state in preparation for
the next run.
Action
This field lists the steps of testing.
Expected Results
This details the expected outcomes of a test.
Component
You can associate Bugzilla components to your test cases. Because testing might be across
multiple components and products, you can add multiple components from any product.
Selecting the product from the drop down list displays the components available. Click Add
to associate the selected component with the test case.
Tags
See above for a description.
Test Runs
Product Version
This is the Bugzilla product version that this test run is testing.
Plan Version
This is the version of the plan document that this test run is using.
Manager
The user in charge of this test run
Build
The default product build. This is applied to test cases added to the run.
Status
Determines if this test run is active.
● RUNNING – This test run is still active. Test cases in this run can be updated.
● STOPPED – Test cases in a stopped run cannot be modified.
Environment
This is the environment in use for this test run. Test cases added to this run will receive this
environment. Environments must be created using the Environment editor before a test run
can be created.
Summary
A short description of this test run
Notes
This field is a place to make notes and observations about this run.
Test Case-Runs
Status
The status of a case-run determined by whether it passed or failed or is in some other state.
- IDLE: This is the default status. Case has not yet been examined.
- PASSED: This test case met the requirement or ran as expected.
- FAILED: This test case did not run as expected or produced an unhandled exception.
- RUNNING: This test case is currently being examined.
- PAUSED: This status is used to denote a problem with the test case itself that
prevents the test from being completed.
- BLOCKED: This test case has a dependency that failed.
Index
A user defined sort index. This can be used to place your test cases in a specific order.
Add Notes
This is a place to append notes to the case-run.
Notes
This field combines the notes from all case-runs records for this case in this run as well as
status change history.
Attach Bugs
This field allows you to attach bugs to this test case. Enter a comma separated list of bug
numbers to attach more than one bug at a time.
Assignee
This is the person assigned to run this test case.
Tested By
This is the person that actually ran the test. This is updated when a test case is placed into the
PASSED, FAILED, or BLOCKED status.
Close Date
The time this test case was closed (PASSED, FAILED, or BLOCKED)
Build
The product build used for this test case-run.
Environment
The product environment used for this test case run.
Glossary
Term Objects Description
Action Test Case The list of steps that a test case must complete.
Alias Test Case A globally unique string that identifies a test case in
conjunction with the test case ID.
Archive Test Plan Test plans may be archived and hidden from regular
searches.
Arguments Test Case A list of parameters to send to an automatic test script .
Assignee The person responsible for applying a status to a test case-
run
Blocks A test case that blocks another test case.
BLOCKED A status of a test case-run indicating the prerequisite test
case failed.
Build In software testing, a string denoting the compiled results
of a period of development.
Category A property of a product that is used to classify test cases.
Clone An exact replica of data between two objects. In Testopia
you can clone plans, runs, and cases.
Component A Bugzilla component. An attribute of a product.
CONFIRMED A status of a case. Confirmed test cases have been
approved for use in test runs.
Default Tester The default person responsible for applying a status to the
test case-run for a given test case.
Dependency Test cases can be dependent on other test cases. There are
two types of relationships, depends on and blocks. A test
case that is blocked by another should not be examined
before the prerequisite test case as success is determined in
part by the outcome of the predecessor.
Depends on Sets up a dependency between test cases. Depends on lists
the test cases that a particular test case requires to be
completed before this case.
DISABLED A status of a test case denoting it is no longer used for
active testing. Similar to archival for a plan.
Expected The expected results upon completing the action of a test
Results case.
Environment A list of the surrounding conditions that a test run is
performed in.
FAILED A status of a test case-run. Denotes the test case failed in
the given run.
IDLE A status of a test case-run. Denotes the test case has not
been examined in the given run.
Manager The person in charge of a given run.
Milestone A Bugzilla object. A property of a product that implies
when a given bug or feature will be fixed or included.
Testopia builds are associated with milestones.
PASSED A status of a test case-run. Denotes the test case has met
the conditions of success detailed in the Expected Results
of the test case in the given run.
PAUSED A status of a test case-run. Denotes the test case has been
under examination in the given run and is on hold. Used
primarily for performance tests that may span long time
periods.
Plan Document The information of a test plan detailing what the test plan
will cover for what by when. Depending on the level of
scrutiny required it can be very verbose.
Plan Version The version of the plan document used for a particular run.
Priority The Bugzilla priority. Test cases can be assigned a priority
similar to bugs.
PROPOSED A status of a test case that denotes it has not yet been
approved for use in test runs.
Requirement A field of a test case provided to capture information about
a requirement. Typically an ID of a requirement in a
separate requirement tracking system.
RUNNING A status of a test case-run. Denotes the test case is in the
process of being examined in the given run.
Running A status of a test run. Running test runs can have case-runs
updated and implies that there is further testing to be done
in the run.
Script A path to an external automated test script for a given test
case. Testopia does not run this script, the field is only
provided as a way of informing the user where to find it.
Stopped A status of a test run. Stopped test runs can not have case-
runs updated. This status denotes the run is complete.
Tag A user defined string used to classify test plans, cases, and
runs.
Test Case A list of conditions and expected results for success for a
particular feature or object under scrutiny. Test cases are
associated with one or more test plans and with zero or
more test runs.
Test Case-run The union of a test case and a test run. Each time a test case
is included in a new test run, an entry is made for it in the
test case-runs table. This captures whether the test case
passed or failed in the given run. Each case-run should be
associated with only one build for a given status.
Test Plan The defining object in Testopia. Organizes the other
objects in Testopia.
Test Run The instance of performance in Testopia. Each run is
associated with a single plan and environment. It contains a
list of test cases to be examined and stores the results in the
case-runs table.
Tested By The person who examined and applied a status to a given
case-run
Type The plan type. Plan types might include System, Unit,
Integration etc. Each plan can be of only one type.
Getting Help
There are a number of resources for getting help in Testopia. You should first check out the FAQ
on the wiki for additional pointers not included in this manual. If your problem is not there,
asking your question on the mailing lists or in the chat room are your next best options. Please do
not email the developers directly with support questions. They are busy people like you who
have other commitments and cannot expect to be available at all times. Besides, asking your
questions in a public forum allow more people the chance to respond and might get you a
solution more quickly.
Mailing Lists
● [email protected]
● [email protected]
IRC Chatroom
irc://irc.mozilla.org/testopia
Wiki
http://wiki.mozilla.org/Testopia
Reporting Bugs
Testopia is still a work in progress. As such, it still has a lot of bugs to be worked out. If you
encounter a bug or find some way to make Testopia better, please log it at
http://bugzilla.mozilla.org/enter_bug.cgi?product=Testopia
Be sure to include steps to reproduce the problem and what browser you are using to access
Testopia.
Bibliography
Copeland, L. (2004). A Practitioner's Guide to Software Test Design
Norwood, MA: Artech House
IEEE Std 829-1998 IEEE Standard for Software Test Documentation (n.d).
Retrieved February 28, 2007 from
http://standards.ieee.org/reading/ieee/std_public/description/se/829-1998_desc.html
Lohmeyer, J. (2004, April 27). Open-Source Bug Tracking with Bugzilla. Linux Journal.
Retrieved February 27, 2007, from http://www.linuxjournal.com/article/7216