Measurement matters

The role of metrics in ISO 22301

A BSI whitepaper for business

Measurement matters – A BSI whitepaper for business

Executive summary
• The business continuity standard ISO 22301 recognizes the • This BSI whitepaper clarifies the requirements around
importance of having accurate performance information measurement in ISO 22301
• The standard lays down requirements for “monitoring, • Three BSI clients describe how they have approached these
measurement, analysis and evaluation” requirements
• The emphasis on monitoring performance, measurement
and metrics in ISO 22301 has caused confusion in some
organizations

Measure for measure

“You can’t manage what you can’t measure” – or, more specifically, without the key value of the business continuity programme
has become a cliché in business. Many performance data that monitoring and to the organization cannot be assessed.
managers will have this well-worn phrase measurement provide – businesses cannot The international BCMS standard ISO 22301
pasted to a noticeboard or taped to their hope to evaluate their Business Continuity recognizes the critical importance of having
PC. But, in common with many clichés, the Management system (BCMS) effectively. accurate performance information and,
saying is fundamentally true – and valuable. Having the right information is vital, as unlike its predecessor BS 25999, this
Nowhere is measurement more important without it informed judgements cannot relatively new standard lays down
than in the discipline of Business Continuity be made when managing or improving an requirements for “monitoring, measurement,
Management (BCM). Without measurement organization’s BCMS. Above all, the overall analysis and evaluation.”

Clarifying confusion
The greater emphasis on monitoring performance, measurement and ISO 22301 tasks Business Continuity (BC) professionals to:
metrics in ISO 22301 has been accompanied by a flurry of confusion
• Monitor the extent to which their business continuity policy,
in some organizations, as they ask:
objectives and targets are met
• How do we assess whether our BCMS is performing as we want
it to?
• Measure the performance of processes, procedures and functions
that protect its prioritized activities
• What sort of metrics should we adopt and why?
• Monitor compliance with the ISO 22301 standard and the
• How are other organizations approaching the issues of business continuity objectives
monitoring, measurement, analysis and evaluation?
• Review historical evidence of deficient BCMS performance
• What challenges have other organizations encountered when
implementing the new requirements in ISO 22301 and how have • Conduct internal audits at planned intervals
these been overcome?
• Evaluate all this in the management review at planned intervals
Lorna Anderson, Global Business Continuity Technical Manager at
Anderson puts it simply: “You need to determine what needs to be
BSI, says, “In general, measurement within BCM systems is not done
monitored and measured, what metrics you’ll use to do it, when to do
well.” She suggests the problem stems from two main factors. First,
it, and what you’ll do with the information.”
business continuity professionals brought up on BS 25999 have not
been schooled in the discipline. “It wasn’t a requirement, so people When the results are analysed, she says, the key is for organizations
simply didn’t do it,” says Anderson. Second, and more fundamentally, to evaluate the performance and effectiveness of their BCMS, take
“They don’t know what to do. They see the word ‘metrics’ and shout action to address adverse trends before nonconformities occur, and
‘help!’ We see organizations struggle, especially if they aren’t heavily ensure they retain relevant documentation of results.
into standards – it comes more easily where the standards’ mentality
of plan-do-check-act is more ingrained.”

2
 bsigroup.com

Setting goals
Julian Thrussell, Senior Consultant at Ultima action and improve the effectiveness of their successfully contacted, say, 90% of their
Risk Management (URM), a consultancy BCMS and ultimately enhance organizational interested parties is not sufficient. What if
specializing in business resilience standards, performance. a key customer was amongst the missing
agrees that organizations do not do metrics 10%? What would be the impact if this key
“Conducting a business impact analysis
well – and adds a third reason for it. “They customer then received information about
will provide a clear assessment of the
don’t really know what their end goal is and the incident from social media, rumours
organization’s most important activities and
therefore what they should be measuring. I and the press?” Thrussell suggests that the
will provide the basis of the BC plan. Just as
often see metrics that are not as meaningful organization needs to break the metric
a BC plan is unique to every organization, the
as they could be, such as how often the figure down into different types of interested
things they are looking to protect and the
BC plan was updated, or how often it was parties and ensure all interested parties
importance of these activities are going to
reviewed. Whilst it’s important to update BC are identified. Each of these categories will
be unique too,” says Thrussell.
plans, this is not their raison d’etre.” require input and information from different
“Too often though, there is just one overall people within the business to ensure the
He continues, “Before organizations measure
objective in BC plans, so when organizations accuracy of records. Furthermore, if these
anything they need to work out what
conduct an exercise they simply ask ‘did records are not accurate there needs to
success looks like, so they have a benchmark
the plan work as expected? Did it pass or be a corrective action process in place. If
and an objective to aim for. Before they can
fail?’ What organizations should be doing for example, a customer contact detail has
score anything as say, 5 out of 10, they first
is breaking down the plan into its key not been updated and this has resulted in
need to define the characteristics of 10 out
component parts, so during the exercise a ‘non-contact’, it will require a corrective
of 10.”
they record metrics around how the key action from the sales director to 1) correct it
Experts agree that many organizations need component parts have worked, ” suggests and 2) ensure it does not recur. Likewise, if
to take a big step forward to a situation Thrussell. there is an error in the shareholder register,
where they can benefit from using the the company secretary will need to address
He cites the example of how well a
most suitable metrics, to measure the most it. “Passing or failing the plan as a whole is
company performs, after an incident or a
important variables, at the most appropriate not terribly valuable when some aspects
business continuity exercise, in contacting
times – providing them with valuable data have worked very well and others require
interested parties, such as customers,
that they can analyse and gain useful improvement,” says Thrussell. “Success is
key suppliers, staff and investors. “Simply
insights. They are then in a position to take about accuracy and attention to detail.”
having a statistic saying the organization

Unknown unknowns
Business continuity experts agree with Donald Rumsfeld, in the sense
“There are known knowns; there that simply auditing your plan and putting metrics against what you
already have only goes so far – because you will not find things that
are things that we know that we know. are entirely missing from the plan.

We also know there are known

“To discover the unknowns, run an exercise with a business unit,
stressing that it’s not a pass/fail test, but an exercise that is designed
unknowns; that is to say we know to improve the organization’s response to an incident and ensure
nothing falls between the gaps,” suggests URM’s Thrussell. “An
there are some things we do not know. exercise will show you very quickly and publicly what works and what
doesn’t. That understanding is invaluable to a BC manager, who can
But there are also unknown use objective metrics that other people have fed into, for improving
the plans and validating the time and money being spent on business
unknowns, the ones we don’t continuity. It is better to discover a problem during an exercise than
during an incident. Would you rather identify the out-of-date contact
know we don’t know.” details in an exercise or when that key customer is on the phone to
you demanding a response to rumours circulating on social media?”
Donald Rumsfeld, US Defense Secretary, 2002

3
Measurement matters – A BSI whitepaper for business

Beyond the numbers

While the requirement for effective objectives that are ‘SMART’– Specific, Anderson summarizes, “People have
measurement suggests a wider range of Measurable, Achievable, Relevant, and done 101 different things and we need
metrics and collecting more information Time-based. They also need to be ‘DUMB’ – to bring some sense to this area. Metrics
from more places, BSI’s Anderson stresses Doable, Understandable, Manageable and are simply agreed objective, actionable
that the quality of data is more important Beneficial.” measurements that reflect your critical
than the quantity. “It comes back to having success factors.”

Performance evaluation 9.1 Clause for thought

Clause 9 of ISO 22301 brings together the maintaining and Clause 9.1 of ISO 22301 specifically states that, “a compliant
reviewing of a BCMS. organization shall determine:

Clause 9.1 Monitoring, measurement, analysis and evaluation a What needs to be monitored and measured
is a set of requirements designed to ensure that appropriate b The methods for monitoring, measurement, analysis and
metrics are in place to effectively manage the BCMS and evaluation, as applicable, to ensure valid results
provides the input to management reviews.
c When the monitoring and measuring shall be performed,
Clause 9.2 Internal audit – includes a requirement that and
the management responsible for the area being audited
d When the results from monitoring and measurement shall
must “ensure that any necessary corrections and corrective
be analysed and evaluated.
actions are taken without undue delay to eliminate detected
nonconformities and their causes. Follow-up activities shall The organization shall retain appropriate documented
include the verification of the actions taken and the reporting information as evidence of the results.
of verification results.”
The organization shall evaluate the BCMS performance and the
Clause 9.3 Management review – includes a new requirement effectiveness of the BCMS.
to provide information for the review on the trends in: Additionally, the organization shall:
1 Nonconformities and corrective actions • Take action when necessary to address adverse trends or
2 Monitoring and measurement evaluation results results before a nonconformity occurs, and

3 Auditing results • Retain relevant documented information as evidence of the

results.
Additionally, when considering the output from the
management review, changes may be required to risk The procedures for monitoring performance shall provide for:
reduction and security arrangements and operational • The setting of performance metrics appropriate to the needs
conditions and processes, if appropriate. It may also be of the organization
appropriate to change the measures for “how the effectiveness
• Monitoring the extent to which the organization’s business
of controls are measured.”
continuity policy, objectives and targets are met
This clause concludes with a requirement for the organization • Performance of the processes, procedures and functions that
to “communicate the results of management review to relevant protect its prioritized activities
interested parties, and take appropriate action relating to those
results.” • Monitoring compliance with this International Standard and
the business continuity objectives
• Monitoring historical evidence of deficient BCMS’
performance*
• Recording data and results of monitoring and measurement
to facilitate subsequent corrective actions.
(* Deficient performance could include nonconformity, near
misses, false alarms, and actual incidents).

4
 bsigroup.com

Case study 1: Allen & Overy

Not just numbers
UK law firm Allen & Overy LLP employs Restall’s high-level metrics consists of: “We don’t leave it there,” says Restall. “We
over 5,000 staff, including more than 500 sometimes carry out additional activities to
• Management review – twice a year
partners. Founded over 80 years ago, it has add confidence. For instance, we recently
built a global network spanning 45 offices • BC plan review – set of review meetings invited an external agency to do a review
in 31 countries and is the only firm to have and plan reissue process – twice a year of our BC plan arrangements. It wasn’t
been ranked in the top three of the FT Law 50 • Gold team exercise – once every two something required by the BCMS or by
since it began in 2006. years auditors, but we decided we wanted an
independent report that went more into our
As a leading legal player, Allen & Overy’s ability • Business recovery team exercise – every
BCMS’s fitness for purpose.”
to serve its clients at all times is paramount. two years
It maintains certification to ISO 22301 for its • Automated cascade exercise – annually He concludes, “You can apply some numbers
London and Belfast offices, audited by BSI, to to the high-level metrics I’ve cited, but
ensure its business continuity management • Recovery test at our professional work numbers alone don’t go far enough. It’s
system is always robust. area site – annually all very well to say, ‘yes, we’ve had two

Clive Restall, Senior Manager Global

• Internal audit – three-yearly rolling management review meetings’ and put a tick
programme in the box, but what was the quality of them?”
Resilience, says, “When I started out with
ISO 22301 and its requirement for metrics, I • ISO audits – three-yearly rolling audit and He concludes, “What we need to be asking
struggled with it. Metrics is a word that needs recertification programme is: what are the things we should do? Have
careful interpretation and application. We • Other documents are reviewed, as we done them? And, what is their quality
could measure many numbers and values required and value? The management review process
– for example, how many people can log on exists to ensure we’re satisfied with the
remotely to our systems at any one time;
• Staff training and awareness – three-
answers.”
yearly rolling programme and annual
or how many seats we have at our remote
census
recovery site – but I wouldn’t necessarily
relate these statistics to how well our BCMS The list creates a calendar of regular
is performing. They don’t tell me whether our monitoring events, creating a picture of
plan is good, bad or indifferent.” how the BCMS is performing and providing
all the information required to exercise
Business continuity planning is not a
management control.
function that produces tangible outputs
such as motor cars or widgets, and Allen & “At management review meetings, I expect
Overy is a service business with no physical to report that all these things I’ve undertaken
products either. “The issue of metrics posed to do are on track,” says Restall. “But clearly
a difficult question for us as to what to count there are going to be issues raised that will
or measure,” says Restall. “I also wanted to lead us to identify actions for improvement.
keep the chosen measures straightforward For example, when remote working was
so they would be understandable to all my first introduced at A&O, we assessed our
stakeholders.” ICT capacity for staff to work offsite. It was
fine for our day-to-day needs, but in the
Restall’s starting point was to list all the
event of an emergency we might have been
things he undertook as part of maintaining
lacking. We’ve now increased that capacity
an effective BCMS – “things that we could
to a high level. If we fail to follow through on
look back on at management review and say
the actions, this will be picked up either at
whether we’d done these things or not. Some
management review or by one or more of the
of them had a quality element, where I don’t
auditors.”
just want a tick in the box, I want to make
sure I’ve got something worthwhile”.

5
Measurement matters – A BSI whitepaper for business

Case study 2: ScottishPower

Metrics maturity
ScottishPower, part of global utilities group Iberdrola, supplies The objectives align loosely to a plan-do-check-act methodology and
electricity and gas to millions of homes and businesses around the also to the requirements of ISO 22301.
UK. Headquartered in Glasgow, its operations include electricity
McNamara explains, “Having set the objectives, we define the actions
generation, transmission, distribution and retail.
that will allow us to complete each one, breaking them down into a
Ben Woodall, Business Continuity & Communications Manager, manageable annual operating plan, with monthly checks and controls
describes the company’s BCMS as “pretty mature in terms of knowing – which include a number of metrics and measurements we use to
our business and knowing ISO 22301, and bringing the two together.” ensure we’re on track.”

He continues, “We’ve always had business continuity in some form at No metric is required for objective 1 and fairly obvious financial
ScottishPower, but the process was reinvigorated in 2007. We went budgets are applied to objective 2, but, says McNamara, “The way
on to become certified to BS 25999 in 2008, before transitioning to staff have performed in training, for example, is a relevant metric for
ISO 22301 in 2012.” objective 6.”

Prior to implementing ISO 22301, Woodall and fellow Business Woodall describes how objective 3 involves multiple activities and
Continuity Manager, Katherine McNamara, brought together the metrics. Objective 3 is delivered by four key actions: define the BCMS
BCM systems in ScottishPower’s Retail and Generation businesses, strategy; complete the business impact analyses (BIA); create the
which were previously certified separately. With office and power recovery plans; create pandemic or people-impact plans. Against each
station environments differing significantly, the process revealed a one is a metric. In the case of the BIA, for example, ScottishPower
host of different tasks and metrics that needed to be rationalized and has identified 25 BIAs it plans to review in 2014. Logically, there are
recorded more efficiently. also 25 recovery plans. Due to the nature of pandemics, there are
16 site-related (as opposed to department-related) pandemic plans,
The starting point was to redefine the objectives from two
corresponding with 16 company sites.
management systems into a single BCMS and these have been further
refined since, so there are now 10 objectives: Woodall and McNamara explain that, for many of the other objectives,
ScottishPower’s BCMS draws upon numerous metrics, from the
1 Align and certify to the most relevant or beneficial standard strategic to the tactical level, to inform management, drive corrective
for business continuity actions and maintain compliance with ISO 22301.

2 Deliver the Retail and Generation BCMS within the agreed and The standard does not dictate what metrics ScottishPower should
allocated budget use. It gives the flexibility for the company to select measures,
3 Fully understand the organization and develop a robust and scoring systems and benchmarks that are both easily accessible and
enduring continuity response in critical areas and activities useful. In reporting to management, for example, the company’s BC
professionals use a simple traffic light system – red, amber and green
4 Exercise and test continuity arrangements and plans to – to flag up issues relating to its 10 BCMS objectives.
ensure suitability
Woodall concludes: “It gives the senior management team a really
5 Work with our internal IT providers to ensure IT disaster
quick and clear understanding of where our management system
recovery arrangements are appropriate for the Retail and
is and what we’re doing about it, when throwing numbers at them
Generation businesses
probably isn’t going to help them.”
6 Provide appropriate training and awareness of the BCMS to
further develop the continuity culture within the Retail and
Generation businesses
7 Maintain and continually improve the BCMS to ensure it
remains current, appropriate, effective and aligned to industry
standards and best practice
8 Develop and maintain relationships with national government,
devolved government and local emergency planning groups
9 Manage existing and emerging external continuity
considerations (in essence, understanding customers and
suppliers)
10 Review and maintain continuity-related risks and threats to
the Retail and Generation businesses

6
 bsigroup.com

Case study 3: Telefónica

Cutting complexity
Telefónica is one of the largest This is key, according to Clarke, because “But it’s so easy for people to do. You don’t
telecommunications companies in the world internally any company maintaining have to invent new measures – clause 9.3
in terms of market capitalization and number certification to ISO 22301 will manage itself within the standard on management review
of customers. The Spanish multinational effectively, but often they are dependent on tells you everything you’ve got to look at.”
employs around 120,000 people and has external third parties.
For O2, the issue of metrics involves a host
a significant presence in 24 countries, and
“In reality, we can’t always track every of numbers – for example, the number of
is better known in the UK as the mobile
small suppliers’ performance against incidents, the scoring of those incidents, the
network operator, O2.
this requirement, but we can monitor number of nonconformities, CSI scores and
David Clarke, O2 Business Continuity key suppliers and partners, such as the so on. But, says Clarke, “the heart of the issue
Manager for Telefónica UK Limited, says outsourced provider of our sales and service goes beyond numeric values and measures”.
he has found ISO 22301 much more operation. It provides us with external data
He concludes, “The terminology in the
explicit in its reporting requirements than where we would otherwise be in the dark,”
standard uses the terms ‘metrics’ and
its predecessor, BS 25999. “For example, he adds.
‘measurement’ and some firms have got
you have to report to senior management
An important external measure is O2’s really hung up about these words, creating
annually on how many nonconformities with
customer satisfaction index (CSI) score, excessively complex processes. But the
the standard you’ve had.”
which is independently validated by a third standard allows companies to define what
He adds, “The value in frequent tracking and party. “We track our CSI score across every ‘metrics’ means to them. We’ve tried to
trending is that you identify risks and take part of our business, so it highlights service stick to plain English and straightforward
steps to mitigate them sooner.” interruptions, recurring incidents and measures that we were already doing.”
business continuity problem areas,” says
Clarke says the company’s starting point for
Clarke.
business continuity metrics was to “look at
what we do”. “We’ve thought through what “Another metric we know is the number of
an incident is and have gone on to categorize times a BC incident is declared, and again
all the different types of incidents that we require our suppliers to report to us how
might affect us from ‘major’ to ‘minor’, using many times they’ve had an incident. With this
a scoring system that runs P0, P1, P2, P3 data, we can analyse the history and see if we
etc,” says Clarke. “The benefit of this is that are hitting a particular problem at a particular
people across the organization are familiar time.”
with these categories, so they immediately
The company can also draw on data from its
understand the level of seriousness of any
internal audit function and from an external
incident being referred to.”
company that provides it with quality checks.
He continues, “We know how many incidents
O2 is monitoring compliance with
we have in each category and we also know
ISO 22301 too. For every incident it has a
how many system failures there are, so
post-incident review and uses an internal
we have metrics internally that give us a
accredited body to review compliance
clear picture of what’s happening. But the
against the standard. The company aims to
people that businesses tend to forget are
correct nonconformities within six months
partners and suppliers. So we’ve built it into
and keep documentation up to date.
our contractual requirement of suppliers
that they have to keep track of, and tell us, “We have to be monitoring and measuring for
how many incidents have impacted on their a six-monthly report to the board, as well as
service to us.” for BSI to audit us to ISO 22301,” says Clarke.

7
Measurement matters – A BSI whitepaper for business

Taking action
BSI’s Anderson leaves organizations with the following questions 4 Does each metric include a clear statement of the expected
to help them start on their approach to metrics. “Organizations results?
must remember that if you’re going to invest in BCM and wish 5 Does each metric focus on effectiveness and/or efficiency of the
to have a clear view on the health of your BCMS, then you need element being measured?
to track its performance. In other words, your BCM metrics
6 Does each metric allow for meaningful trend or statistical
constitute your BCM scorecard, the way you figure out where
analysis and include milestones and/or indicators to provide
you are. To use another term, they form your dashboard. So,
qualitative feedback?
when considering your metrics please ask yourself the basic 10
questions:” 7 Are your metrics challenging, but at the same time attainable?
8 Have assumptions and definitions been specified for what
1 Do your metrics link directly back to your BCMS and its
constitutes satisfactory performance? Is it clear what ‘good’ or
objectives?
compliance actually looks like?
2 Will the metrics drive improvement and progress?
9 Have those who are responsible for measuring performance been
3 Do your metrics follow the SMART principle: fully involved in the development of the metrics?
S = Specific: clear and focused to avoid misinterpretation. 10 Do your metrics allow for clear reporting to their intended
Should include measurement assumptions and definitions, audience?
and be easily interpreted.
M = Measurable: can be quantified and compared to other data.
It should allow for meaningful statistical analysis. Avoid “yes/
no” measures except in limited cases.
A = Attainable: achievable, reasonable, and credible under
Find out more about
conditions expected. ISO 22301 with BSI
R = Realistic: fits into the organization’s constraints and is cost-
effective.
Call: 020 346 0780
T = Timely: doable within the time frame given. or visit: bsigroup.nl

© BSI Group BS/UK/518/SC/1014/en/BLD

BSI Netherlands
Adam Smith Building
Thomas R. Malthusstraat 3c
1066JR Amsterdam
Nederland 2014
2013

T: +31 20 346 0780

E: [email protected]
bsigroup.nl