Change Control and Release Management Standards

1. ICT systems and infrastructure play a critical role in the efficient operation of the organization.
Changes to ICT systems and infrastructure may arise reactively as a response to problems or
externally imposed requirements, or they may be proactive changes to reflect business
improvement initiatives.

2. This document specifies how changes to ICT systems and infrastructure can be implemented and
released into a live environment. This process exists in the context of ICT governance, as defined
in the IM Strategy.

3. A comprehensive Change Control and Release Management process minimizes risks involved in
introducing changes and it becomes a method of communication between stakeholders – the
person requesting a change, the Change Control Manager and the team building the change.
Other benefits associated with a structured Change Control and Release Management
procedure include:

a) Protect the existing infrastructure and architecture

b) Ensuring that changes are aligned with a corporate strategy
c) Ensuring that proper business cases exist and that the solution is based on user
requirements
d) Ensuring that changes are evaluated and benefits are measured
e) Avoiding duplication of system development
f) Ensuring changes are sufficiently tested
g) Ensuring communication and end user support and training is planned
h) Ensuring that rollback plans and contingency measures are planned
i) Identifying application release frequency
j) Validating decommission of a system
k) Communicating within and outside the effected group based on events
l) Providing a central repository for tracking all changes (including impact and risk
analyses)

4. In addition, the Change Control and Release Management process assures metrics and reports
that help Management judge the quality and effectiveness of Change Control and Release
Management processes and tools. A

5. The goal of the Change Control Management process is to ensure that standardized methods
and procedures are used for efficient and prompt handling of all changes – in form of change
requests utilizing PHIRE— in order to minimize the impact of change-related incidents upon
service quality, and consequently, to improve day-to-day operations of the organization. This
includes an ICT Governance structure to ensure proper project management and quality control,
as well as Performance and Risk Indicators that are mechanisms to measure the success and
associated risk of changes.

Page 1 of 14 Effective Date: 30/12/1899 Version #: 4

Scope

6. The scope of Change Control and Release Management procedures involves planning, design,
build, configuration, and testing of hardware and software to create a set of release
components for the live environment. ICT infrastructure components subject to change control
and release management include:

a) Application programs developed or customized in-house

b) Externally developed software Utility software
c) Internet domain names managed by UNDP
d) Supplier-provided systems software
e) Hardware, and hardware specifications
f) Instructions and user manuals

7. Change Control and Release Management procedures are mandatory for:

a) All ICT infrastructure items managed through UNDP Office of Information Management
and Technology (ITM/BMS)
b) All corporate ICT Infrastructure items, i.e. ICT infrastructure items used by more than
one Country Office or business unit.

Flow Chart

Page 2 of 14 Effective Date: 30/12/1899 Version #: 4

Use PHIRE for All Change Requests

8. Change control planning involves generation of change requests, which should be submitted in
UNall or using PHIRE software. Nevertheless, please note that ATLAS is being frozen until the
implementation of the NextGen ERP.

9. Include in the Change Request, for high-risk changes, all testing and validation, vendor review,
peer review, and detailed configuration and design documentation. Refer to a Non-Atlas Testing
Guide for testing protocols, and the Project Manager is responsible to ensure:

a) All resources are identified and in place for the change

b) A clear goal has been set and met for the change
c) Create solution templates for deployments affecting multiple sites. Include information
about physical layout, logical design, configuration, software versions, and deployment
guidelines
d) Document the standards for configuration, software version, supported hardware,
Domain Name System (DNS), device naming, design, and services supported
e) Ensure the change conforms to all organizational standards for design, configuration,
version, naming conventions, and management
f) Create rollback procedures (before each release, the Project Manager and the
implementation team should identify a rollback strategy depending on the project and
create one for unforeseen incidents. The Change Release specialist checks if this action
is taken and in place
g) Define escalation paths
h) Define affected users and downtimes for notification purpose

10. Change planning documentation, such as maps, detailed implementation procedures, testing
procedures and rollback procedures, should be created and included in the PHIRE ticket. The
level of planning is usually directly proportional to the risk level of the change.

Caveat: The technical description of a change is an important aspect of the change request and
should reference any standards within the organization that apply to the change. This will help
to ensure that the change conforms to the current architecture and any engineering design
guidelines or constraints.

11. Complete one of the following change request type procedures:

a) Planned Changes
b) Emergency Changes
c) Standard Changes
d) Atlas Changes

Planned Changes

12. Planned changes are normally scheduled changes, whether big or small. They involve the
following steps, including the completion and submission of a PHIRE change request:

Page 3 of 14 Effective Date: 30/12/1899 Version #: 4

 Step Action

1. Business case and requirements document

The scope of a proposed change should include a complete technical definition and the
purpose of the change. In addition, the requestor should include information describing
who will be affected, both during the change period and after deployment. This may include
business units, user groups, servers, and applications. The estimated cost and benefits must
be documented in the form of a business case.

Required templates for business case and requirements reside at the ITM PMO site.

2. Change request submission

Once the business case and requirements document are completed and signed off by the
project manager, the change request will be submitted using PHIRE software.

3. Review change request

The Track Lead or Product Lead will evaluate the business case and classify the category.

The Track Lead or Product Lead has the right to reject invalid business cases or give
suggestions.

4. Approve or reject

The Track Lead or Product Lead has a right to approve, reject, or request further
clarification. Changes are defined in the Change Magnitude Criteria below:

Minor changes are approved by Track Lead or Product Lead

Significant changes are approved by Change Control Board (CCB)

Major changes are approved by the ICT Board (IIAG for Atlas changes)

5. Build change

Change is developed in accordance with current architecture and any engineering design
guidelines or constraints. Significant and major changes are managed as separate projects,
as defined in the Change Magnitude Criteria below. Minor changes are built under the
supervision of respective Development Leads.

If there is a need to significantly change the approved requirement document or

specifications during the implementation period, the Development Lead or Project Manager
is responsible for highlighting the change to the CCB Secretary and obtaining a new
approval.

Page 4 of 14 Effective Date: 30/12/1899 Version #: 4

 6. Unit system and integration test

When the development work is completed, the development team will conduct unit,
system and integration testing. The Development Lead or the Project Manager will sign-off
for UAT readiness prior to the UAT session (see Testing Guide for detailed testing phases).

7. User acceptance testing

The Change Release and Testing Specialist will facilitate the User Acceptance Test with
testers from the business community. The Development Lead or Project Manager will
provide the UAT test scripts and support the testing process on giving guidance to testers
and resolving issues. Track Lead or Product Lead will record the UAT sign-off provided by
End Users or Business Owners.

8. Release

The Change Release & Testing Specialist will review the test results, complete the Release
Process Checklist and submit to the Development Lead and Production Manager for the
migration to the Production environment. The Change Release & Testing Specialist will also
provide input to the CCB on the overall statistics and exceptions for the monthly release.
The Change Release & Testing Specialist will notify the Communication team for release
communication and Help Desk for support. All training materials and other relevant
documents need to be updated at this phase. The final cost and benefit will also be
captured by the PMO.

9. Change review

The change manager will compile statistics on change requests and note if the change was
successful or not (including lessons learned and any best practices).

10. Communication

The change request owner or Track Lead / Product Lead will ensure communication of any
change through Release Notes, with automated email sent through PHIRE, and, for major
changes, through a user-community email detailing the change and its implications for
users.

Emergency Changes

13. Emergency changes are urgent changes required to address problems. The number of urgent,
proposed changes should be kept to an absolute minimum, because they are generally more
disruptive and prone to failure. The procedures that are in place to handle emergency changes
should be flexible enough to facilitate rapid resolution of Production problems, including
documentation of who is authorized to make emergency changes, and how to contact these
individuals.

Page 5 of 14 Effective Date: 30/12/1899 Version #: 4

 14. A sufficient number of people should be available who can resolve emergencies, or those people
should be easily accessible at all times to prevent a roadblock in the problem resolution process.

15. It is imperative that both communication and the integrity of documentation are maintained
through an emergency change. This is the time when documentation is needed most, so
documenting the steps taken to resolve the problem is of utmost importance.

16. Finally, when considering changes, it must be determined if the change will resolve the existing
problem and if it will cause other problems. Below are the steps that are critical for an
emergency change process, including the completion and submission of a PHIRE change
request:

Step Action

1. Requirements document

The scope of a proposed emergency change needs to be created and submitted, indicating the
technical definition and the purpose of the change as well as the reason for the urgency and
the impact of delaying the change.

Required templates for business case and requirements reside at the Project Portfolio Server
Project Management site.

2. Change request submission

Once the business case and requirements documents are completed and signed off by the
project manager, the change request will be submitted using PHIRE software.

3. Approve or reject

The Track Lead or Product Lead approves the changes based on the following points:

a) An immediate Severity 1 or Severity 2 Production problem (see Severity Level

Definitions)
b) An evolving problem that would rapidly lead to a Severity 1 or Severity 2 production
problem if it is not immediately prevented

The Development Lead will analyze the change and give clearance to the Development team
for implementation of the changes.

4. Change is built

Change is developed in accordance with current architecture and any engineering design
guidelines or constraints.

5. End-to-end testing by Implementation Team

When the development of the new change is complete, testing of urgent changes should be

Page 6 of 14 Effective Date: 30/12/1899 Version #: 4

 carried out by the Implementation Team. Completely untested changes should not be
implemented. The Development Lead or Project Manager is responsible for this task (see
Testing Guide for detailed testing phases).

6. Release

Change Release and Testing Specialist will approve the change.

7. Post-emergency documentation

Updating documentation is critical to ensure valid, up-to-date information. During emergency

changes, it is easy to forget to make updates because of the frantic nature of emergencies.
However, undocumented changes often result in increased downtime if the solution is
unsuccessful. Therefore, the respective project manager is responsible for all emergency
changes documents update after the fact. The Change Release & Testing Specialist will also
inform CCB on monthly reports.

8. Change review

The Change Release & Testing Specialist will compile statistics on change requests and note if
the change was successful or not (including lessons learned and any best practices).

10. Communication

The change request owner or Track Lead/Product Lead will ensure communication of any
change through Release Notes, with automated email sent through PHIRE, and, for major
changes, through a user-community email detailing the change and its implications for users.

Emergency Changes: Severity Level Definitions

Severity
Definition Scope Examples
Levels

1. Critical A major production Many or most users are unable to E-mail not
business outage, major function. Mission critical system available
impact performance down.
Impossible to
degradation, or
Mission critical application down. process payment
instability causing
significant impact to Mission critical server or
many users and/or time- connection down.
critical data processing.

2. Major Large number of users is Multiple users unable to function. Application not
business impacted. Entire group or working
Major performance issues.
impact department is
experiencing a similar Multiple users running on
problem. Small number

Page 7 of 14 Effective Date: 30/12/1899 Version #: 4

 of customers can’t use a contingencies or workarounds.
mission critical
application.

3. Minor Individual unable to use Customer can work with minimal Bug in application
business application(s). impact to their productivity.
impact
Customer having difficulty, but
basically operational.

Customer unable to carry out their

necessary tasks.

4. Normal Individual request or Customer needs information or a Minor bugs

request problem that does not standard service.
Questions
block usage of systems.
Customer has simple question or
problem.

How-to's or Procedural questions.

Standard Changes

17. A standard change is an accepted solution to an identifiable and relatively common set of
requirements, where authority is effectively given in advance of implementation. For each ICT
infrastructure item, the change request owner can approve a list of defined standard changes.
These changes can be initiated and executed without requiring explicit approval in each
individual case.

18. The Change Control Board (CCB) will review the approved list of standard changes at regular
intervals.

Atlas Changes

19. Please refer to the Atlas Change Control Manual for specific protocol procedures, and details on
Atlas-specific release management.

Testing

20. Please see System Testing Guides for detailed testing information, as follows:

Page 8 of 14 Effective Date: 30/12/1899 Version #: 4

Non-Atlas Testing Guide

Testing Phases Objectives Test & Sign-off

1. Unit Testing Verify the business transactions related to a specific Developer

development object (e.g., a program) function as designed
and meet the specified business requirements accurately

2. System Validate functionality within the module, or across a subset of Functional &
Testing modules Technical Team

3. Integration Verify the major functions of the system operate together in a Functional &
Testing cohesive manner Technical Team

4. Users Confirm the applications and processes meet end-user End Users
Acceptance requirements, and operate within the applicable technical
Testing environment

5. Performance Validate the technical components of the system and Technical Team
Testing infrastructure while providing acceptable performance for
typical production loads

Communications

21. Once a change has been approved, details of the change must be communicated, by setting
expectations, aligning support resources, detailing operational requirements, and informing
users. PHIRE has e-mail notification and the Change Release Specialist will trigger the e-mail
notification (sending Release Notes). See the Authority section for all communications points
within the process. The risk level and potential impact to affected groups as well as scheduled
downtime as a result of the change will dictate the communication requirements.

Conclusion

22. Change Control and Release Management is an essential part of all organization’s overall
security posture. Failure to properly manage change can result in vulnerabilities as well as lost
time from a lack of control for both planned and unplanned changes can lead to opportunity for
hackers and/or people with ill intent to damage or gain unauthorized access to systems. The
process of defining/implementing control policies/procedures for change is continuous, like the
changes to an environment.

Page 9 of 14 Effective Date: 30/12/1899 Version #: 4

Structure Element - Inputs

23. Resources, especially PHIRE, and sites are available at the following location: Change Release
and Testing Management
Note: The IAGG may depute to the CCB requests on changes in Atlas where it feels the purview
of IAGG is not necessary.

Change Request Process – User Definitions and Process Steps

Process Steps
Initiating - Running – Running Running – Closing – Closing -
Build the – Release the Communicate Change
Approval
change Approve change into to users Review
before work
Authority (Change the built live
starts
Request change environment
(Project Tracking)
Portfolio
Oversight)

ICT Board - Oversee

Transitional implementati
Committee (for on of IM
Atlas: Inter-Agency strategy,
Governance Group Inter-Agency
- IAGG) coordination
PMO Office Assist with Oversee
project project
documents management
preparation process
Project Manager Finalize Oversee Monitor Review in
business case change Follow up project PHIRE
and request on progress
requirements implementati project Draft Release
documents on impleme Notes
ntation
and
testing
Implementation Implement Raise
Team under and test Release
supervision of change Service
Development Lead request. Request
Update
PHIRE

Page 10 of 14 Effective Date: 30/12/1899 Version #: 4

Change Request Draft business Submit Follow up Close change
Owner (sponsor) case and change on request
requirements request impleme
documents ntation
Track Lead or Reviews Signs off
Product Lead change on
request and testing
assigns (UAT)
change size and
category follows
up on
build
impleme
ntation

Change Control Review Approval Assess-

Board (CCB)* overall of ment and
portfolio of Monthly quality of
initiatives and release recent
recommend schedule; changes
prioritization post
facto
review of
emergen
cy
approval.

Change Release & Review Perform all Coordinat Prepare and Implementatio Review in
Testing Specialist Change CCB e testing validate n Team (PHIRE) PHIRE
Request Secretariat (UAT) Release
Inform
functions and sign- Process
appropriate
off Checklist
Help Desk (e-
mail)

Email and post

Release Notes

Development Lead Emergenc Approve

and Production y change release in
Manager sign-off PHIRE

Page 11 of 14 Effective Date: 30/12/1899 Version #: 4

 24. Change Magnitude Criteria

25. Change Request Risk Assessment - Each change request should be assigned one of the following
categories:

a) High – Changes have the highest impact on user groups or particular environments and
may even affect an entire site. Change rollback is time-consuming or difficult.
Management must be aware of the change and its implications, and all users must be
notified
b) Moderate – Changes can critically impact user environments or affect an entire site, but
change rollback is a reasonably attainable scenario. Users may be notified of a
moderate-risk change
c) Low – Changes have minor impact on user environments and change rollback is easy.
Low-risk changes rarely require more than minimal documentation and User notification
may be unnecessary

Page 12 of 14 Effective Date: 30/12/1899 Version #: 4

 Level Definition Testing and Validation Requirements
1 High potential impact to large Lab validation of new solution, including
number of users or business-critical documented testing, validation, and what-if
service because of the introduction of analysis showing impact to existing
new product, software, topology, or infrastructure
feature; change involves expected
Completion of an operations support
network downtime
document
Rollback plan, implementation plan, and
adherence to the change process
Recommend solution pilots and a preliminary
design review prior to testing
2 High potential impact to large What-if analysis performed in lab to
number of users or business-critical determine the impact to existing environment
service, because of a large increase ofin regards to capacity and performance
traffic or users, backbone changes, or Test and review of all routing changes
routing changes; change may require Rollback plan
some network downtime Implementation plan
Adherence to change process
Design review for major routing changes or
backbone changes
3 Moderate potential impact to smaller Requires engineering analysis of new solution,
number of users or business service, which may require lab validation
because of any non-standard change, Implementation plan
such as new product, software, Adherence to change process
topology, features, or the addition of
new users, increased traffic, or non-
standard topology, change may
require some network downtime

4 Low potential impact, including Implementation plan

adding new standard template Adherence to change process
network modules (building or server
switches, hubs, or routers), bringing
up new WAN sites or additional
proven access services, and all Risk
Level 3 changes that have been
tested in the production
environment. Change may require
some network downtime.
5 No user or service impact, including Optional adherence to change process
adding individual users to the
network, and standard changes. No
expected downtime

Page 13 of 14 Effective Date: 30/12/1899 Version #: 4

 26. Performance indicators provide the mechanism to measure the success of the change control
management process. The indicators should be reviewed periodically to ensure that change
planning and change control management are working well.

Performance Indicators

Change Control Includes the percentage and quantity of change success by functional group and
Management risk level. Emergency changes should be identified separately in the metrics by
metrics by functional group, including the success rate for attempted fixes. Functionality
functional groups include any ICT teams making changes, possibly including server
group administration, network administration, database groups, application teams, and
facilities. Risk level is important because generally higher risk changes fail or
create incidents. A change failure may be defined as any change that is rolled
back or causes a problem incident resulting in user downtime.

Change success To target change success, start with a baseline of Change Management metrics.
rate The CMA can then identify potential issues and set overall goals. A reasonable
overall goal for change success should be 99 percent across all functional groups.
If the organization is experiencing a higher rate of change failure, it should be
targeted for improvement.

Change rate The Change Release Library will be used for archiving the change history. The
per system information can be used to investigate change rates in general for overall
planning purposes.

The Change Release Manager should archive the Change Planning history in the
Change Release Library.

Undocumented The quantity and risk level of undocumented changes should be investigated.
change audit Undocumented change is a common problem in almost all organizations. It must
be continually reiterated that team members are required to use the required
change control process, even though it adds time and efforts.

Page 14 of 14 Effective Date: 30/12/1899 Version #: 4