Release Notes:

What Do Test
Engineers Need to
Know? by Marc René

Author’s Introduction information that is necessary for accept- marked as ‘fixed’ in the defect tracking
Recently, I was asked to investigate and ance and evaluation of the software system and tries to prioritize the retest
create a Release Notes Standards docu- release by the Test Engineers. In addition, effort. Craig finds thirty defects that are
ment to improve the content, format and Release Notes provide a historical record marked as ‘fixed’. However, he realizes
consistency of my company’s Release of the release, tracing release content that some of these defects were marked as
Notes documentation. After searching the through the evolution of the software. By ‘fixed’ after 4:00 p.m. yesterday. Nancy’s
Web and various Software Engineering establishing an agreed upon scope, con- note had indicated the build occurred at
books, I discovered that there is little infor- tent, and format for Release Notes, as well 4:00 p.m. As a result, Craig too, must wait
mation on this topic. As a result, I had to as identifying the resources required to for Nancy to arrive to determine which
rely on information obtained during inter- create them, GTECH has greatly ‘fixed’ defects are included within this
views I conducted within my company. improved the transition from development release.
Our experienced Test Engineers and to test.
Software Engineers were asked what they It is now 9:00 a.m. Nancy arrives at work
considered ‘good’ and ‘bad’ practices with Release Day to find Craig and Cheryl waiting anxious-
regard to Release Notes. The data was Today is release day. Cheryl and Craig ly for her. After a few minutes, Nancy is
compiled and the Release Notes Standards arrive at work at 7 AM to find a note on the able to resolve Cheryl’s issue and provide
document that resulted became the basis door of the Test Lab. It reads, “Built last Craig with the correct list of ‘fixed’defects
for this article. night around 4:00 p.m. Get the new stuff that are included within this release.
on the M: drive. Defects are marked as
Overview ‘fixed’ in the defect tracking system. At 9:15 a.m., Cheryl and Craig again
My personal experiences indicate that Nancy.” begin their tasks. Cheryl follows Nancy’s
clear and concise communication between verbal directions for the installation.
Test Engineers and Software Engineers Not knowing how long the install will take Hoping everything installed successfully,
facilitate a successful handoff of a soft- or how many defects have been corrected Cheryl informs Craig that they can finally
ware release. In most cases, the Release within this release, the Test Engineers pro- begin testing. It is 10:00 a.m. and approx-
Notes become the most important docu- ceed, hoping to begin testing before lunch. imately three hours (six person hours) of
ment during the transition of the software test time is wasted because the Test
from the development phase to the test Cheryl begins installing the ‘new’ version Engineers were not given all of the infor-
phase. Missing or incomplete Release of the software and immediately runs into mation required to commence the testing
Notes often result in wasted time, missed trouble. She ends the installation and now activities.
schedules, and inadequate testing. must wait for Nancy to arrive to resolve
the issue. Meanwhile, Craig’s prioritization efforts
Release Notes provide the mechanism for are fruitless. Many of the ‘fixed’ descrip-
Software Engineers to convey all of the Craig begins printing out the defects tions indicate only “problem resolved”.

December 2002 http://www.testinginstitute.com Journal of Software Testing Professionals 1

The descriptions do not provide details on What are Release Notes? Why Create a Release Note
any possible test considerations. Without Release Notes are a release description Standards Document?
meaningful guidance from the Software document prepared to identify the content In order to gain consistency in the format
Engineers, Craig prioritizes the ‘fixed’ and changes made for each release of soft- and content of Release Notes, a Release
defects based solely on their severity level. ware. This information aids the Test Note Standards document is written for
High severity defects will be verified first. Engineer in determining which areas use throughout an organization. Release
and/or items that must be verified or Note Standards provide general guidelines
After retesting several ‘fixed’ defects, retested. Release Notes allow Software for the creation of Release Notes.
Cheryl uncovers a serious problem. It Engineers to convey the exact code that Adhering to the guidelines set forth in the
takes her over an hour to isolate the steps has been changed, the reason that the code standard’s document will enable the fol-
to reproduce the problem. Cheryl is has been changed, and any special testing lowing goals to be attained:
recording the defect in the defect tracking conditions that the Test Group should
system when Nancy enters the lab to see exercise. In addition, information relating • Uniformity in the presentation and the
how the testing is progressing. When to code reviews and design reviews for the content of Release Notes across projects
Nancy sees the problem that Cheryl is log- released software is included within the • Facilitation of the hand-off of software to
ging, she indicates that she “knew about Release Notes. Review information will the Test Group
the problem last week” and that she “fixed assist Test Engineers with the prioritiza- • Increase visibility into release content
it this morning”. tion of retest efforts. Fixes or new func-
tionality that have undergone design or A Release Note Standards document has
Frustrated, Cheryl and Craig decide to go code review may be considered lower risk benefits for Software Engineers and
to lunch; it has been a very tough morning. and prioritized for test after more risky Project Managers as well as Test
functions or fixes. Engineers. Guidance on Release Note
What do Test Engineers Need presentation and content frees Software
to Know? At GTECH, we require Release Notes for Engineers from the mundane and repeti-
Cheryl and Craig’s problems are not all software releases. We do not allow tive tasks of determining what information
unique. Many Test Engineers struggle exceptions. Developers supply Release is relevant to the Test Engineers and how
with releases because they do not receive Notes to the Test Engineers prior to the to present that information. Once Release
adequate Release Notes. In some cases, software being released for testing. The Note standards are set, Software
Test Engineers do not receive any release Test Engineers reject any software release Engineers complete the appropriate sec-
guidance at all! Often this occurs because if Release Notes are missing or incom- tions providing the information indicated
Software Engineers do not appreciate the plete. avoiding the need to ‘re-invent the wheel’
importance of Release Notes, or do not for each new release. This frees Software
know what should be included in them. The suggested Release Note content Engineers to do what they are best at, cre-
includes the information specified on the ate software. When Test Engineers receive
Release Notes can expedite the testing next page. The sequence listed is for the complete Release Notes, they are less like-
associated with a software release. Release sake of presentation in this article, and the ly to interrupt Software Engineers because
Notes provide installation instructions, actual order in the Release Notes might be they already have the release information
including a method to determine if the different. For example, the ordering may required. Consistent Release Notes also
installation worked and any back-out steps be different if utilizing defect-tracking allow historical release information to be
if problems are encountered. They also tools to generate portions of the Release stored and if necessary retrieved later.
provide information to assist Test Notes. Defect-tracking tools often allow Project Managers also benefit from con-
Engineers in the prioritization of test/retest for the generation of lists of defects sistent and complete Release Notes as the
efforts that include details on any reviews addressed and any special test considera- content assists with the determination of
performed on fixes and any special test tions specified when the problem was cor- schedule accuracy. Since Release Note
considerations. Release Notes also pro- rected. In this case, the sections containing content includes functionality added to a
vide information about known issues so defects addressed and test considerations release, the Project Manager can compare
that Test Engineers do not waste valuable would be combined. the schedule and the listed functionality to
testing time identifying and reproducing ensure the project is on schedule.
them. If Cheryl and Craig had received
this type of guidance, their release day
would have been much more productive
and efficient.

December 2002 http://www.testinginstitute.com Journal of Software Testing Professionals 7

 Suggested Release Note Content
1. Release Identifier Specific Version Number. In addition, a clear description of the type of software that is being
released must be provided (i.e., the specific functional area or component names). If the released
software is part of a larger release, then information regarding the Overall Release Name and the
Overall Release Number is also included.

2. Release History Document the revision number, the date of the release, and a brief explanation for all previous soft-
ware releases. For convenience, this section is listed in reverse chronological order (newest first.)

3. Release References Include all references to earlier revisions of the software or other products. Software Requirements
Specification(s) including version number, and Design Documentation including version number.

4. Release Overview Includes a narrative description of the functionality contained in the software release. If applicable,
include comparisons to older releases of the software. If the release occurs during an incremental
build/delivery cycle, the Release Notes refer to the specific build/delivery release by name and list
specific items scheduled in the current release.

4.1 Functionality Added Describe all ‘new’ functionality that has been added to the software in the release. For initial releas-
es, it is not necessary for all functionality to be listed. However, any planned functionality not being
released can be listed, making future reference to this functionality easier.

4.2 Functionality Removed Document any and all functionality that was removed within the software release along with
rationale for the removal.

4.3 Known Problems Not Documented by Defects include any known problem(s) the software might have that have not been
Documented by Defects previously documented in the defect tracking system.

5. Release Operational Describe the changes relating to intra-system interfaces or system operation that may be depend-
Considerations ent upon another product’s release or any non-software solutions. These dependencies are docu-
mented to identify any change that the Test Group should make to the Test Environment that are
not included as part of the standard release (i.e., Operating System patch or hardware upgrade).

6. Defects and/or Enhance- Include a list of all defects and/or enhancements that have been addressed and released for testing.
ments Addressed Some defect-tracking systems provide the ability to generate this list.

7. File Descriptions Provide a comprehensive list of the specific files or, at a minimum, the directories that are being
released. These lists assist the Test Engineers with determining the impact of the changes and addi-
tions and assist with assuring correct installation.

7.1 Files Modified in Release Document the names and purpose of source, executable, data, and command files (as applicable)
that have been modified within the software release. This subsection also includes an overview of
code changes that were made as well as a list of the files that were modified.

7.2 New Files in Release Document the names and purpose of source, executable, data, and command files (as applicable)
that are ‘new’ to the software release.

7.3 Files Removed in Release Document the names and purpose of source, executable, data, and/or command files (as applica-
ble) that are ‘removed’ since the previous release.

8 Journal of Software Testing Professionals http://www.testinginstitute.com December 2002

 Suggested Release Note Content (continued)
8. Tools and Utilities Document the names and the impact of all ‘new’ tools and/or utilities contained within the release.
If changes were made to existing tools and/or utilities, they must also be clearly described. If
changes within the release impact testing tools, these changes are listed in detail. This section also
indicates if any special tools or utilities are required to utilize this release or to test it.

9. Code/Design Reviews Document the results of all code or design reviews associated with the software release. The results
Performed are detailed in this section of the Release Notes documentation or a ‘path’ is provided where this
information can be found. If the inspection data is stored in a tool, specify the location in the tool
where the inspection data can be found.

10. Unit/Integration Tests Provide detailed documentation on how the code was tested by the Software Engineers. The results
Performed of Unit/Integration Testing are documented in detail in this section of the Release Notes or the
Software Engineer provides a ‘path’ where this information can be found. Additionally, special test
harnesses used to perform Unit Testing must be referenced and described along with their results.

11. Test Considerations Include any special test or setup considerations the Test Engineer should be aware of when testing
the release. The Software Engineer must list any special procedures for testing each correction and
document any unique set-up and/or tools required. The defect tracking system may provide the
ability to generate this information for each defect in the release.

12. Installation Details Provide specific details on the set-up and verification procedures for the installation of the soft-
ware release.

12.1 Location Provide a detailed description of the location (i.e., file path name) of the software release.

12.2 Procedure Describe the systematic procedure for installing the released software. This subsection also
includes any special release installation instructions if the release instructions differ significantly
from previous releases.

12.3 Schedule and Duration Schedule and Duration Include the software release schedule details (date, time and location) as
well as an estimate of the expected duration of the installation for the release.

12.4 Post Installation Provide the information necessary so that Test Engineers can be assured that the release has been
Checklist installed successfully and that testing may begin.

13. Contingencies Include a detailed recovery or ‘back-out’ process. This section provides instructions for Disaster
Control and the method to be used to return to an earlier release version if necessary. If a release
cannot be returned to a previous version because of a file format change or other major redesign,
it must be specified in this section of the Release Notes.

14. Sign Off Contains authorization by the Software Engineer, Lead Test Engineer, Project Manager and any
other interested parties. All parties must agree on and accept the contents of the Release Notes.
Sign Off does not necessarily imply an actual signature, but stresses the importance of agreement
by that all parties that the Release Notes are complete and correct.

December 2002 http://www.testinginstitute.com Journal of Software Testing Professionals 9

What is a Release Note Summary through design and code reviews, so these
Standards Document? Release Notes serve an important purpose can be prioritized lower on the list.
A Release Note Standards document facilitating the successful handoff of a
describes the basic conventions and guide- software release. Release Notes provide It is 7:30 a.m and Cheryl and Craig are
lines to be used for the generation of the mechanism for the Software Engineers ready to start testing the new release.
Release Notes. These conventions ensure to convey to the Test Engineers all of the
uniform content of Release Notes that information that is necessary to accept and After retesting several ‘fixed’ defects,
result in improved maintainability and evaluate the software release. Project Cheryl uncovers a serious problem. She
testability of software releases. The docu- Teams that fail to understand the impor- checks the Release Notes and learns that
ment is written for the technical staff that tance of complete, clear and concise the problem, although serious, is known
is responsible for software development. Release Notes often waste time, miss by the development team. She moves on
schedules and perform inadequate testing. to the next defect and does not spend any
GTECH’s® Release Note Standards doc- time trying to reproduce the problem.
ument contains information on content, Creating a Release Note Standards docu-
responsibilities and potential areas for tai- ment is a mechanism to establish the con- By the time Cheryl and Craig decide to go
loring the standards. In addition, an exam- tent of and responsibilities for the creation to lunch, they have verified all the high-
ple template is provided to illustrate a of Release Notes. These standards assure risk fixes and only found one bad fix. It
potential format for presentation (refer to that Test Engineers have the information has been a very productive morning.
the example provided). required to test a release. They assist the
Software Engineer by providing a check- Bibliography
The content defined in the standard is as list of information to include in a release. Guide to the Software Engineering Body
illustrated in the above table. This set was They allow the Project Manager greater of Knowledge
chosen because of the complexity of visibility into release content and sched- IEEE – Trial Version 1.00 – May 2001
GTECH’s® systems and our need to man- ule.
age geographically separated development Software Release Methodology
and test teams. Smaller teams or less com- It should be noted that although this article Michael E. Bays, Prentice Hall PTR,
plex systems or products may not need the stresses the importance of Release Notes 1999. This book illustrates best practices
complete set above. as a communication vehicle for software for every stage of a successful product
releases between the Software Engineers release. It includes carefully designed,
As defined in the GTECH® standard, the and the Test Group, many of the same con- practical solutions that enhance quality,
responsibility for creating Release Notes cepts can be applied for Release Notes reduce costs, and increase time to market.
falls on the Software Engineers. Project given to customers as well.
About the Author
Managers are responsible for ensuring that
Release Day – Revisited Marc René is a Software Quality
Software Engineers follow the standards.
Today is release day. Cheryl and Craig Assurance Engineer for GTECH
arrive at work at 7 AM to find a complete Corporation. He is responsible for the def-
The level and extent of tailoring is speci-
set of Release Notes tacked to the Test Lab inition, collection and evaluation of met-
fied in the standard. In general, it is expect-
door. rics to analyze product and process quali-
ed that Release Notes will contain all sec-
ty. Marc works with the SEPG and is
tions defined in the Release Notes
Cheryl begins installing the ‘new’ version responsible for the generation of tools as
Standards document. If a particular section
of the software following the detailed well as creating many corporate processes
is not applicable for a release, it is sug-
installation procedures. Once installed, and procedures. His experience with
gested that the section remain and its con-
Cheryl follows the procedure to verify that Software Engineering, SQA and Testing
tent be listed as ‘Not Applicable’. The
the software has been installed correctly. spans 12+ years.
GTECH® standard expects and encour-
ages projects to expand upon the standard The software installed without incident.
Marc has spoken at STAREAST 2000 and
as needed for their specific environment.
Craig starts prioritizing the retest effort other SQA conferences. Marc received a
However, the project must not violate any
based upon the fix descriptions, test con- Bachelors Degree in Computer Science
convention within the Release Note
siderations, design review and code and Psychology from Rhode Island
Standards document.
review information. Craig finds that all of College and is a member of the RIC
the HIGH severity defects have been Computer Science Program’s Industry
Advisory Board.

10 Journal of Software Testing Professionals http://www.testinginstitute.com December 2002

 Project Name: Release Name:
Date of Release: Release Version Number:
Overall Release Overall Release Version
Name: Number:

Contents of Release
Description: <Insert brief description >

Release History (repeat as often as required)

Revision <Insert Date of the <Insert Release <Insert information>
Number information> Software’s information> Descript
: Release: ion:

Release References
Reference(s) to earlier versions of the software or other <Insert detailed information>
products:
Software Requi rements Specification(s) with Version <Insert detailed information>
Number(s):
Design Documentation with Version Number(s): <Insert detailed information>

Release Overview
Functionality contained in the Release: <Insert description>
Functionality Added : <Insert description>
Functionality Removed: <Insert description>
Known problems not documented by <Insert description>
defects:

Detailed Release Description

<Insert description>

Defect and/or Enhancements Addressed (repeat as often as required)

Number: <Insert Number> Product <Insert Product Name>
Name:
Short Description: <Insert Short Description>
Fix Description: <Insert Fix Description>
Test Considerations <Provide the set of recommended tests>

Release Operational Considerations

<Insert description>

Code/Design Reviews
“Code Review
Description and
Location?

December 2002 http://www.testinginstitute.com Journal of Software Testing Professionals 11

 File Descriptions
Files Modified (repeat as often as required)
Name of file:
Overview of Code
changes:
‘New’ Files in Release (repeat as often as required)
Name and purpose of
file:
‘Removed’ Files in Release (repeat as often as required)
Name and Purpose of
file:

Tool & Utilities

Name and Impact of ‘new’ Tools and/or <Insert Name and Impact>
Utilities
Name and Impact of ‘existing’ <Insert Name and Impact>
Tools and/or Utilities changed:
Special Tools and/or Utilities required for <List and describe any special Tools and/or Utilities required>
Release:

Unit/Integration Testing
Description and Location of <Provide details as to how the code was tested and any special test harnesses used to
Unit/Integration Test and Special perform unit testing>
Test Harnesses:
Description of Unit/Integration <Provide detailed results of the Unit/Integration Tests>
Test Results:

Installation Details
Detailed Description of location <Insert ‘path’ for the Software Release>
of the Software Release:
Step-by-step installation <Provide the steps required for the installation of the Release. Include any special
procedure: instructions if needed>
Release Schedule Details & <Include Release date, time, location, and estimated duration of installation>
Estimated Duration:
Post Installation Checklist: <Provide a detailed list so that the Test Group can be assured the Release has been
installed properly>
Contingency Plans – recovery <Provide instructions for Disaster Control and the method to be used to return to an
process: earlier release version if necessary>

Sign Off
Software Integration Engineer
Engineer: (if applicable):
Test Lead: Technology Lead:

12 Journal of Software Testing Professionals http://www.testinginstitute.com December 2002