Release Management Process and Procedure

Fermilab Computing Division-RM-1.0 Docdb#3737

Release Management
Fermilab Process and Procedures

Fermi National
Laboratory
March 25, 2010

Page 1 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

GENERAL
This document establishes a Release Management (RM) process and
procedures for the Fermilab Computing Division. Adoption and
Description implementation of this process and supporting procedures ensures the
timely recovery of services and will minimize the adverse impact on
business operations.

The purpose of this process is to establish a Release Management (RM)

process for the Fermilab Computing Division. Adoption and implementation
Purpose
of this process provides a structured method to implement approved
changes to the IT infrastructure to minimize risk and disruption to services.

Applicable to Release Management process in support of the ISO20000 initiative.

Supersedes N/A

Document
Release Manager Owner Org Computing Division
Owner

Effective Dates 04-01-2010 to 04-01-2011 Revision Date 03-01-2010

VERSION HISTORY
Version Date Author(s) Change Summary

1.0 .01 3/17/10 David Whitten – Initial document

Plexent LLP

Page 2 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

TABLE OF CONTENTS
RELEASE MANAGEMENT GOAL, BENEFITS...................................................................4
RELEASE MANAGEMENT OVERALL PROCESS FLOW..................................................7
RELEASE MANAGEMENT PROCESS ROLES AND RESPONSIBILITIES........................8
RELEASE MANAGEMENT RACI MATRIX.......................................................................15
RELEASE MANAGEMENT PROCESS MEASUREMENTS (KPIS)..................................16
14.1 RELEASE MANAGEMENT PLANNING PROCEDURE............................................17
14.1 RELEASE MANAGEMENT PLANNING BUSINESS PROCEDURE RULES..................................18
14.1 RELEASE MANAGEMENT PLANNING PROCEDURE NARRATIVE...........................................18
14.1 RELEASE MANAGEMENT RECORDING & ACCEPTANCE RISKS...........................................19
14.1 RELEASE MANAGEMENT PLANNING RECORDS AND OUTPUTS...........................................20
14.2 RELEASE CONFIGURATION (BUILD) PROCEDURE..............................................21
14.2 RELEASE MANAGEMENT CONFIGURATION (BUILD) PROCEDURE BUSINESS RULES............22
14.2 RELEASE MANAGEMENT CONFIGURATION (BUILD) NARRATIVE.........................................22
14.2 RELEASE MANAGEMENT CONFIGURATION (BUILD) RISKS.................................................24
14.3 RELEASE TEST AND ACCEPTANCE PROCEDURE...............................................25
14.3 RELEASE TEST AND ACCEPTANCE BUSINESS PROCEDURE RULES...................................26
14.3 RELEASE TEST AND ACCEPTANCE PROCEDURE NARRATIVE.............................................26
14.3 RELEASE MANAGEMENT TEST AND ACCEPTANCE RISKS..................................................27
14.4 RELEASE MANAGEMENT RELEASE PREPARATION PROCEDURE....................28
14.4 RELEASE MANAGEMENT RELEASE PREPARATION BUSINESS PROCEDURE RULES.............29
14.4 RELEASE MANAGEMENT RELEASE PREPARATION NARRATIVE..........................................29
14.4 RELEASE MANAGEMENT RELEASE PREPARATION RISKS..................................................31
14.5 RELEASE DEPLOYMENT PROCEDURE.................................................................32
14.5 RELEASE MANAGEMENT DEPLOYMENT BUSINESS PROCEDURE RULES.............................33
14. 5 RELEASE MANAGEMENT DEPLOYMENT PROCEDURE NARRATIVE.....................................33
14.5 RELEASE MANAGEMENT DEPLOYMENT RISKS..................................................................35
SUPPORTING DOCUMENTS............................................................................................36

Page 3 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT GOAL, BENEFITS

Goal The goals of Release Management are:
 to plan and oversee the successful rollout of hardware and software
changes into production.
 to design and implement efficient procedures for the distribution and
installation of changes, as defined by Change Management, to the
Enterprise.
 to ensure that the changes made to hardware and software can be
tracked, using the services of Configuration Management, and that only
correct, authorized and tested hardware/software versions are
installed.
 to insure that sufficient communication concerning the proposed
change has occurred among all relevant parties to the change,
including the Customers.
 to agree to the content and schedule of the rollout plan, testing plan,
and all other documentation pertinent to the change, and to
communicate agreement of those plans and documentation to Change
Management.
 working as a Peer Process to Configuration Management and Change
Management, to implement new hardware or software Releases into
the operational environment.
 To oversee the release of new updated CI’s into the production
environment, upon receipt of a change request from Change
Management.
 To advise Change Management as to the acceptability, or lack thereof,
of supporting plans, test results, or other documentation required by
Release Management for the proposed change to occur.
 to ensure that master copies of all software are secured in the
Definitive Software Library (DSL) and that the Configuration
Management Database (CMDB) is updated as required.
 to manage the Definitive Software Library (DSL) and Definitive
Hardware Store (DHS)
The focus of Release Management is the protection of the production
environment and its services using formal procedures and checks.

Benefits The principal benefits of Release Management, when combined with effective
Configuration Management, Change Management and operational testing
functions are:

Page 4 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT GOAL, BENEFITS

 a greater success rate in the Release of hardware and software and
therefore an improved quality of service delivered to the business
 consistency in the Release processes of the hardware platforms or
software environments
 minimization of the disruption of the service to the business through
synchronization of Releases within packages involving hardware and
software components from different platforms and environments
 assurance that the hardware and software in live use is of good (or
known) quality, because the Releases are built properly, from hardware
and software components that have been subject to quality control and
effective testing, and have been constructed under Change
Management
 stable test and live environments, because Changes are normally
combined into Releases and so there should be fewer individual
implementations
 better use of User resources because of combined efforts when testing
new Releases - this also means that it will be easier to justify the cost
of system-wide regression testing
 minimization of regression-testing requirements, offering greater
coverage than is possible with small Changes that occur frequently or
too close together
 better expectation setting within the organization on publication of a
Release schedule in advance
 error reduction through the controlled Release of hardware and
software to the live environment, e.g. avoiding incorporating an
incorrect version into the Release
 a complete record (or audit trail) of Changes to the live environment is
maintained, both of software distributions and of hardware Changes
 proper control and safeguarding of hardware and software assets,
upon which an organization may be heavily dependent
 an ability to absorb high rates of Change to the live systems, effectively
and without adversely affecting IT service quality - this is achieved by
releasing a large number of Changes together as a single, controlled
and well-understood Release
 the ability to build and control the software used at remote sites from a
central location
 savings in support costs through the ability to maintain consistent

Page 5 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT GOAL, BENEFITS

software over a large number of locations
 reduced likelihood of there being illegal copies of software in use at any
location
 easier detection of incorrect or outdated versions or unauthorized
copies of software
 reduced risk of unnoticed introduction of viruses or other malicious
software
 reduced time to Release and fewer delays
 fewer Releases to be rolled out to Customers
 Smoother transitions of Releases from the development activities
(projects) to the Customer's business environment.
As the efficiency and effectiveness of Release Management grows, the
productivity of IT services staff is likely to increase.

Page 6 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT OVERALL PROCESS FLOW

14.1
Planning

14.2
Configuration

14.3
Testing and
Acceptance

14.4
Preparation

14.5
Deployment

Page 7 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT PROCESS ROLES AND RESPONSIBILITIES

Roles Responsibilities

 Ensures policies and procedures exist to support service delivery.

 Ensures that Release Meetings are developed and occurring on a
regular basis, as determined Release Manager.
 Ensures that the Release Coordinators coordinate with other
Release Coordinators for Release packages that span multiple
operational areas, if required.
 Sets the Release Management policy and communicates all
changes to the policy to appropriate parties (e.g., CAB, change
owners, change requesters, etc.)
 Ensures that the Release is represented at the Change Advisory
Board.
 Ensures that a Release Go/No Go meeting is conducted by the
Release Coordinators well in advance of presentation of the
change to the CAB, if the meeting is required.
 Manage the Definitive Software Library (DSL) and Definitive
Release
Hardware Store (DHS), as they are developed.
Manager
 Coordinates the type of information to be recorded for the release
package into the Configuration Management Database (CMDB)
with the Configuration Manager.
 Review RFCs to ensure compliance to the Release Process and
Policy statement.
 Reviews & approves Release Coordinator findings
 Reviews Request for Change tickets and assigns them to the
appropriate Release Coordinator.
 Administer change-specific Release Go/No Go meetings as
needed, or delegates the administration of the meetings as
required.
 Scope of Role: Release Manager is responsible for the Release
Process and for measuring the success of the process. The
Release Manager will amend the Release process and procedure
documents required for continuous improvement to the process.

Page 8 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT PROCESS ROLES AND RESPONSIBILITIES

Roles Responsibilities

 Review assigned Request For Change (RFC’s) tickets for updates

and modifications and act accordingly.
 Work with change owners/project Lead to insure completion of
required Release Management Process checkpoints.
 Coordinate with Configuration Manager to record information
about the Release package into the configuration management
database (CMDB)
 May represent the Release at the Change Advisory Board if
necessary
 Participates in a change-specific Release Go/No Go meeting as
needed, or administers the meeting as required.
 Provides updates for the Forward Schedule of Change to the
Release Manager
 Acts as representative to oversee Releases within their area
(segment, service).
Release  Coordinates with other Release Coordinators for Release
Coordinator packages as required for changes that span different Release
areas.
 Reviews Release test findings; insures testing was as appropriate
and as thorough as possible.
 Ensure the completion of planning, configuration (vis-à-vis the
Release sub-process), testing, preparing, and deployment of all
releases.
 Ensure that the change owner/project leader has arranged for
adequate training for support and administrative staff prior to
release deployment.
 Ensure advanced communication of scheduled releases to all
appropriate stakeholders.
 Ensure that the change owner/project Lead has arranged that
release resources are assembled for deployment and back-out.
Scope of Role: Release Coordinator is responsible for Releases within
their assigned areas of coverage.

Configuration Ensure Configuration Management Database (CMDB) reflects status of

Page 9 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT PROCESS ROLES AND RESPONSIBILITIES

Roles Responsibilities

Manager all release items and attributes related to Release Management.

 Individual requesting the change.

 Responsible for User Acceptance Testing (UAT) and documentation
Customer of results
 Responsible for attending the Go/No Go meeting
 Responsible for post install verification

 Prepare tests and procedures and execute testing strategies.

 Identify testers and assign testing responsibilities.
 Develop and manage testing schedules.
 Document test procedures and problems.
 Gather, document, and analyze test results.
Testing
Coordinator  Validate readiness of release package to be integrated into the
production environment.
 Oversee the creation of the test plan detailing the testing
requirements, constraints, and risks associated with the Release
based on the needs of the customer.
 Identify and document the test acceptance criteria.

Support Lead  Backup the production system prior to beginning a Release. Ensure
current configuration settings are documented for hardware and
software releases.
 Implement the Release into the production environment within the
established SLA. If the implementation is unsuccessful, follow the
steps outlined in the back-out plan to restore the environment to the
pre-implementation state.
 Update change records to indicate the implementation status of the
deployment.
 Select the processes, tools, and technologies that will be used to
deploy the release package.

Page 10 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT PROCESS ROLES AND RESPONSIBILITIES

Roles Responsibilities

 Produce/configure master copies of the installation media and

instructions for inclusion in the Definitive Software Library (DSL).

 Confirms the test plan detailing the testing requirements, constraints,

and risks associated with the requirements.
 If deemed necessary, deploy a pilot of the release, with the prior
approval of the Release Manager.
 Evaluate the pilot results. Input will be gathered from change
managers, change requestors, and pilot users.
 Present Release Package (documentation, plan, test findings, etc.) to
Release Management for review and approval prior to Release
Management’s transmission to the CAB for approval to implement the
Project Lead release.
 Schedule and ensure applicable Release specific training materials
are available.
 Ensure training materials are provided in preparation for the Release
to support staff and Customers.
 Create support documentation to include the system overview and
revised support procedures.
 Use the selected processes, tools, and technologies to build the
release package

 Design and build a test environment that accurately models the

production environment.
Test  Hardware Releases are to be staged to ensure the hardware is
Environment functional and properly configured.
Coordinator
 Ideally, the test environment will be a replica of production. Test
(Includes Team installation procedures.
Mgr/Lead)
 Test Release components.
 Test back-out plans.

Test Lead  Evaluate the results.

Page 11 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT PROCESS ROLES AND RESPONSIBILITIES

Roles Responsibilities

 Document known defects that will be carried into the production

environment.

Service Desk Dispatch service calls for which Change Management is required, to the
Administrator appropriate specialists group.

Problem Dispatch problems for which a permanent solution has been proposed, to
Manager the appropriate specialist group.

Change  Initiate the change.

Requestor
 Follow processes for submitting a Request for Change (RFC),
including reference to any applicable Incident or Problem records.
 Provide change details to the Change Manager for updating the RFC.
 Member of the Emergency Change Advisory Board (ECAB) for their
submitted Emergency Changes.

Change  Receive RFC’s and ensure that they are properly recorded in the
Coordinator change log.
 Ensure proper linkage of applicable incident and problem records to
change records.
 Validates all RFC’s.
 Communicates with the respective teams for the purpose of validating
and conducting RFC impact analyses and risk assessments on
changes.
 Assists the Change Manager as needed in the preparation of the CAB
meeting agendas
 Assists the Change Manager in the notification to Change Requestor
and other affected parties upon rejection or implementation.
 Assists in the monitoring the progress of changes.
 Assists in the monitoring the completion of all RFC’s to ensure that
the implementation procedures used follow the change schedule.
 Working with the Change Manager, review change statistics and KPIs

Page 12 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT PROCESS ROLES AND RESPONSIBILITIES

Roles Responsibilities

for trends.
 Review and evaluate the change process.

Change  Receive RFC’s and ensure that they are properly recorded in the
Manager change log.
 Ensure proper linkage of applicable incident and problem records to
change records.
 Assess, filter, and classify all RFCs.
 Assign people or teams to conduct RFC impact analyses and risk
assessments on major changes. In many cases this is the Change
Owner.
 Prepare CAB meeting agendas and provide all necessary review
information to the CAB members prior to board meetings.
 Select and invite Change Advisory Board (CAB) members and Chair
CAB meetings.
 Analyze, prioritize, classify, and schedule RFCs.
 Provide change notification to Change Requestor and other affected
parties upon rejection or implementation.
 Monitor the progress of changes.
 Monitor the completion of all RFCs to ensure that the implementation
procedures used follow the change schedule.
 Ensure the verification of change implementations with the Change
Owner.
 Schedule, and chair Post Implementation Reviews (PIR).
 Review change statistics and KPIs for trends.
 Review and evaluate the change process.

Change Owner  Receive approved changes from the Change Manager.

 Follow the change schedule that the Change Manager approves.
 Coordinate the change development (as delegated).
 Provide project status feedback to the Change Manager and CAB for
Page 13 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT PROCESS ROLES AND RESPONSIBILITIES

Roles Responsibilities

major, complex Changes that require development.

 Identify problems as they arise.
 Work with the Change Requestor, if a different person, to ensure that
the change meets the Change Requestor requirements.
 Report status and present findings to the CAB on major Changes.
 Prepare for, and assist in PIR.

Page 14 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT RACI MATRIX

Process Name Release Management

R - Responsible Person responsible for getting the work done
A - Accountable Only one person can be accountable for each activity
C - Consult The people who are consulted and whose opinions are sought
I - Inform The people who are kept up-to-date on progress

Primary Roles in Process

Primary Interactions
Secondary Roles

Service
Release Configuration Problem Test Release Service Desk Change
Process Activities Change Manager Level Project Lead Support Lead Customer
Coordinator Manager Manager Lead Manager Administrator Requesters
Manager
14.1 Release Planning Procedure
14.1.1 Identify scope of release R I C C A C C C
14.1.2 Determine if a Standard CR or not R C A C C
14.1.3 Determine Release Team Composition Roles R C C A R C
14.1.4 Identify training, logistics & communications requirements I I I A R C
14.1.5 Update the RFC/Change Ticket R I I C A I
14.1.6 Have requirements changed? R I I C A I C C

14.2 Release Configuration Procedure (Build)

14.2.1 Select Release Mechanism I A I R
14.2.2 Design Release Package C I A I R
14.2.3 Build Release Package I I A R C
14.2.4Test Release Package I R A I
14.2.5 Determine test success I R A R C C
14.2.6 Update Release Manager for updating of Release Record R C A R
14.2.7 Coordinate and Documentation of Strategy for Release in CMDB R C A C I
14.2.8 Determine If release requirements have changed R I I C A C C C

14.3 Release Test & Acceptance Procedure

14.3.1 Determine Testing Requirements, Constraints and Risks C C A R C C C
14.3.2 Design and Build an Accurate Test Environment I R A C C C
14.3.3 Test and Evaluate I R A I I C
14.3.4 Determine success of test I I R A I I C C

14.4 Release Preparation Procedure

14.4.1 Assemble Resources R A R I C I
14.4.2 Provide Advance Communication of Release R I C A R I I I I
14.4.3 Coordinate Training of Support and Administrative Staff I I A R C I C
14.4.4 Document Strategy for Release in Change Record(s) R I I A R I C
14.4.5 Determine if requirements have changed R I A R C C C
14.4.6 Conduct Release meeting - Go/No go decision R C C C C A A C C C C
14.4.7 Perform Go/No go decision R I C C C C A C I I C C
14.4.8 Change Management/CAB approval C R C C C C A C I I C I

14.5 Release Deployment Procedure

14.5.1 Deploy Release I I I I I A R I I R I
14.5.2 Conduct post install verification I I I I I R A R R C
14.5.3 Determine success of release I I I I I C A R I I C I
14.5.4 Initiate Communication Plan I I i I I I A R I I R I
14.5.5 Cancel deployment decision point C I I I I A R I C C C, I

Page 15 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

Page 16 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

RELEASE MANAGEMENT PROCESS MEASUREMENTS (KPIS)

Note: The listed measurements are intended as a guide. As Fermilab Computing Division
matures in their Release Management process, the reports will require adjustment to match
their maturity. This might involve customizing the measurements to reflect the process as it
relates to Fermilab’s organizational requirements. Computing Division is free to develop
reports or measurements specific to the organization that match the needs of Fermilab.

Short Term (0-3 Months – Learning the Process)

 Percentage of Releases backed out due to errors or failures

 Average deviation between planned and actual implementation date/time
 Number of releases performed as scheduled

Medium Term (4-9 Months – Process is maturing)

 Percentage of accuracy of The CMDB after implementation of Releases

 Percentage of backed out Releases

Long Term 9+ Months

 Percentage of unauthorized software

 Percentage of unused software licenses
 Percentage of overused software licenses

Page 17 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.1 RELEASE MANAGEMENT PLANNING PROCEDURE

Change
Management

14.1.1
Identify scope of
Release

14.4
14.1.2
Yes Release
Is RFC standard?
Preparation

No

14.1.3
Determine Release
team composition &
roles

14.1.4
Identify Training,
Logistics &
Communications
Requirements

14.1.5 Initial Release

Update the RFC/ Plan
Change ticket

14.1.6
Have Requirements Change
Yes
changed? Management

14.2
Release
Configuration

Page 18 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.1 Release Management Planning Business Procedure

Rules
 Approved RFC(s)
Inputs
 CAB Output

Entry
Approved Change Request
Criteria

Each Approved Change is considered a single Release following a one-to-one

General model of each individual Change equals a single Release. As the organization
Comments matures, Fermilab Computing Division may elect to incorporate a one-to-many
model where multiple changes are managed via a single Release.

14.1 Release Management Planning Procedure Narrative

Step Responsible Action

Role

14.1.1 Release  Identify scope and content of an approved change

Coordinator and release requirements for a successful
deployment

14.1.2 Release  Determine if the RFC falls under the standard

Coordinator category:
 If yes, then move Release to Preparation.
 If no, then coordinate with Release Project
Lead to determine Release team Composition
and Roles.

14.1.3 Release  Determine Release team composition and roles.

Coordinator

Page 19 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.1 Release Management Planning Procedure Narrative

Step Responsible Action

Role

14.1.4 Project Lead  Identify training, logistics, and communication

requirements, communicate this back to the Release
Manager.
 Update the RFC/Change Ticket

14.1.5 Release  Identify if the Release is bundling +1 RFC’s

Coordinator
 Update the Release Number in the Change
Management Plan
 Create and document Initial Release Plan

14.1.6 Release  Determine if requirements have changed for the

Coordinator release:
 If yes, then resubmit to Change Management.
 If no, then submit to Release Configuration
(Build).

Exit
Release Plan complete
Criteria

Outputs Release Plan for Request for Change (RFC) ready for configuration (Build)

14.1 Release Management Recording & Acceptance Risks

Risk Impact

The release is inadequately planned Release damages the production

environment

Page 20 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.1 Release Management Planning Records and Outputs

Document Review Period Technique

RFC Once, during this procedure  Release Coordinator

Review
 Project Lead Review

Page 21 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.2 RELEASE CONFIGURATION (BUILD) PROCEDURE

14.2.1 14.3
14.1
Select release Testing and
Release Planning
mechanism Acceptance

14.2.2 14.2.3 Release Plan

Design release package Build release package

No 14.2.4
Test release package

14.2.6 14.2.7
14.2.5
Update Release Coordinate Strategy for
Was test Yes
Manager for updating of Release documentation
successful?
Release Record in CMDB

14.2.8
Change
Yes Have requirements
Management
changed?

No

14.3
Testing and
Acceptance

Page 22 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.2 Release Management Configuration (Build) Procedure

Business Rules

Inputs Approved Request For Change scheduled for inclusion in the Release plan

Entry
Release plan for Request For Change that are to be configured
Criteria

General
Comment N/A
s

14.2 Release Management Configuration (Build) Narrative

Step Responsible Role Action

14.2.1 Support Lead  Obtain the Release Package from Release

Planning or from Release Testing &
Acceptance
 Determine a suitable mechanism for the
release: Automated or manual

14.2.2 Support Lead  Coordinates the selection of the processes,

tools, and technologies and the design of the
release package.

14.2.3 Project Lead  Ensures that the selected processes, tools,

and technologies to build the release package
occur.

14.2.4 Support Lead  Ensure readiness of the release package for

(Development) effectiveness of deployment and alignment
with requirements.

Page 23 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.2 Release Management Configuration (Build) Narrative

Step Responsible Role Action

14.2.5 Project Lead/ Support  Determine success or failure of test

Lead
 If yes, go to 14.2.6 and update release
(Development) manager for update to Release record
 If no, send back to 14.2.2 to Project
Leader for re-design of Release package
and recompile the package as necessary
Note: This step is intended as a test of the
different Release Package components prior to
submitting to the testing team

14.2.6 Release Manager  Accept update notices for updating the release
record

14.2.7 Release Manager  Coordinate documentation of Strategy for

Release in CMDB

14.2.8 Release Coordinator  Determine if requirements have changed

 If yes, then resubmit to Change
Management
 If no, then submit to Release Testing &
Acceptance

Exit Criteria Release Package tested and aligned with requirements

Release package has been tested for effectiveness of deployment and aligns
Outputs
with requirements

Page 24 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.2 Release Management Configuration (Build) Risks

Risk Impact

Release does not meet the need of the

Release is not aligned with requirements
customer

Release does not function as the customer

Release has not been adequately tested
expects

Page 25 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.3 RELEASE TEST AND ACCEPTANCE PROCEDURE

14.2
Release
Configuration

14.3.1
Determine testing
requirements,
constraints and risks

14.3.2
Design and build
accurate test
environment

14.3.3
Test and evaluate

14.3.4 14.2
Was test No Release
successful? Configuration

14.4
Yes Release
Preparation

Page 26 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.3 Release Test and Acceptance Business Procedure

Rules

Inputs Release Package for Change from Release Configuration

Entry
Release Package for Change to be tested
Criteria

General Pilot in live environment if necessary. Release and Change Managers and
Comment piloted users will participate in the evaluation of the results and determine scope
s of implementation

14.3 Release Test and Acceptance Procedure Narrative

Step Responsible Role Action

14.3.1 Project Lead  Determine the test requirements as gathered

from the project team and include any constraints
and risks.
 Identify and document the test acceptance
criteria.

14.3.2 Test Lead  Design and build an accurate test environment,

validate against the gathered requirements,
including constraints and risks delivered from the
project leader.
 Ensure the test environment models the live
production environment.

14.3.3 Test Lead  Test installation procedures.

 Test Release components.
 Test back-out plans.
 Evaluate the results.

Page 27 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.3 Release Test and Acceptance Procedure Narrative

Step Responsible Role Action

14.3.4 Test Lead  Determine success of test based on test

acceptance criteria as determined in Step 14.3.1
 If test is successful, go to step 14.4 Release
Preparation
 If test is not successful, resubmit to 14.2
Release Configuration for re-work.

Outputs Release fully tested and ready for production

Exit Criteria Tested and accepted Release

14.3 Release Management Test and Acceptance Risks

Risk Impact

Release allowed into production and fails to

Release not fully tested
operate as planned

Page 28 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.4 RELEASE MANAGEMENT RELEASE PREPARATION PROCEDURE

14.3
14.1 14.4.1
Testing and
Release Planning Assemble resources
Acceptance

14.4.2 14.4.3 14.4.4

Provide advance Coordinate training of Document Strategy for
communication of support and Release in Change
release administrative staff Record(s)

14.4.5
Change
No Have requirements/ Yes
Management
Scope changed?

No

14.4.6
14.4.8 14.5
Conduct Release 14.4.7
Yes CM Yes Release
Meeting – go/no go Go?
Approved? Deployment
decision

Freeze – No adjustments or modifications to the Release allowed at this time

Page 29 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.4 Release Management Release Preparation Business

Procedure Rules

Inputs Tested and accepted Release Package for the Change Request

Entry
Tested and accepted release package ready for preparation to be deployed
Criteria

General
Comment Release ready for production environment
s

14.4 Release Management Release Preparation Narrative

Step Responsible Role Action

14.4.1 Project Lead  Confirm access to the resources required for

deployment and back out plan. These resources
may include:
o People
o Hardware
o Bandwidth
o Vendor Support.

14.4.2  Send advanced communication of release to all

Project Lead stakeholders. Communication should include:
 Proposed date of Release
 Purpose of Release.

14.4.3 Project Lead  Coordinate training of support and administrative

staff with Release team. Training should include:
o Date of scheduled Release
o Purpose of release
o Expected call volume
o Possible questions
o Known Errors

Page 30 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.4 Release Management Release Preparation Narrative

o Workaround.

14.4.4 Project Lead  Update the request ticket with the following
information including impact to specific
Configuration Items. The documentation should:
o Record status of resource
o Record communication effort
o Record training effort
o Record impacted Configuration Item(s)

14.4.5 Release  Validate if requirements have changed.

Coordinator/
 If yes, resubmit to Change Management.
Release Manager
 If no, conduct release go/no go meeting
NOTE: Once the decision has been made to proceed
to the “Go/No go” meeting, then the release is frozen
and no further adjustments, testing, or any other activity
is allowed.

14.4.6 Release  Conduct go/no go meeting

Coordinator/
 Inform/Validate with Change Management for
Release Manager deployment approval
 Submit to Release Deployment
 Standard Changes are assumed as a “go”

14.4.7 Release  If decision is a “go” then submit to 14.4.8 for final

Coordinator/ CAB approval
Release Manager

14.4.8 Change Manager  If Release is approved, then move to Release

Deployment
 If not approved, then return to Change
Management for rework

Page 31 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

 Release Plan
Outputs
 Release package ready for deployment

Exit  Release approved by Go/No Go decision from the project team

Criteria  Release approved by Change Management to move into production

14.4 Release Management Release Preparation Risks

Risk Impact

Release Plan not fully developed Deployment team does not have adequate
deployment plans

Release Package not ready for deployment Release fails to function per the customer
expectations when deployed to production

Release Package not ready for deployment Release Package does not deploy to
production per the deployment schedule

Page 32 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.5 RELEASE DEPLOYMENT PROCEDURE

14.4 14.5.2
14.5.1
Release Conduct Post install
Deploy release
Preparation validation

14.5.7
14.5.3 14.5.4
Yes Update change
Successful? Initiate Communication
record(s)

No

14.5.5 Submit to
Cancel Yes CM for PIR
deployment?

No

Change
14.5.6
Management
Implement with
mitigiation

Page 33 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.5 Release Management Deployment Business Procedure

Rules

Inputs Tested and accepted Release Package for Change Record(s)

Entry
Tested and accepted Release Package ready for deployment
Criteria

General
Comment Deploy to production and interact with Change Management as necessary
s

14. 5 Release Management Deployment Procedure

Narrative

Step Responsible Role Action

14.5.1 Project Lead  Manage deployment of the accepted Release

Package.
 Include any validation steps performed during the
installation to complete the release.
 (note: the validation steps should not change the
intent nor the purpose and scope of the change)

14.5.2 Support  Conduct Post Install Verification

Technicians/
 Post Install Verification – Testing Team
Customer
 Post Install Verification – Customer/Change
Requester

Page 34 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14. 5 Release Management Deployment Procedure

Narrative

Step Responsible Role Action

14.5.3 Project Lead  Determine if deployment was successful:

 If yes, submit to 14.5.4 Initiate Communication
 If no, the send to 14.5.5

14.5.4 Project Lead Based on the outcome of the deployment, initiate the
communication plan to the pre-determined recipients

14.5.5 Project Lead  Cancel Deployment?

 If yes, initiate back-out plan as required and
move to 14.5.4
 If no, and the decision is to not cancel the
deployment, move to 14.5.6

14.5.6 Project Lead  Implement with mitigation

 If the decision is to not back-out the deployment
and to move forward, initiate the pre-defined
mitigation plan and move to 14.5.4 to initiate the
mitigation communication plan

Page 35 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14. 5 Release Management Deployment Procedure

Narrative

Step Responsible Role Action

14.5.7 Project  Update the change records to reflect the new

Lead/Support status of the CI, this should include items such
Lead/Release as:
Mgr/Coordinator
 Actual date & time of change
 Details of the outcome of the change
 Results of the change (successful, not
successful, etc.)
 Move to Change Management for the Post
Implementation Review (PIR)

Deployed Release package ready for Post Implementation Review in Change

Outputs
Management

Exit Deployed Release package ready for Post Implementation Review in Change
Criteria Management

14.5 Release Management Deployment Risks

Risk Impact

Post install verification not performed Release fails to work as expected in the
production environment

Mitigation plan not enacted Release fails to work as expected in the

production environment and costs downtime
to the customer

Communication Plan not enacted  Customer is unaware of completed

Release package.
 Customer is unaware of mitigation
plan implementation

Page 36 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

14.5 Release Management Deployment Risks

Risk Impact

 Customer does not realize that the

Release has been put into production

RELEASE MANAGEMENT CRITICAL SUCCESS FACTORS

 Obtain Management support, budget and resources
 Acceptance from staff to new procedures
 Ensure the business need is identified and understood
 Effective and efficient repeatable processes for implementing releases
 Effective and efficient build and testing facilities
 Effective test plans
 Effective use of technology and tools for release automation
 Secure Definitive Software Library (DSL) and Definitive Hardware Store (DHS)

SUPPORTING DOCUMENTS
Document Name Description Relationship

Fermilab Service Management

Policy Policy
Policy

Fermilab Change Management Change Management

Process
Process and Procedures Process and Procedures

Fermilab Configuration
Configuration Management
Management Process and Process
Process and Procedures
Procedures

Page 37 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved
 Release Management Process and Procedure
Fermilab Computing Division-RM-1.0 Docdb#3737

Page 38 of 38
Fermi National Accelerator Lab Private / Proprietary
Copyright © 2010 All Rights Reserved