Finance and Economics Discussion Series

Divisions of Research & Statistics and Monetary Affairs

Federal Reserve Board, Washington, D.C.

Benchmarking Operational Risk Models

Filippo Curti, Ibrahim Ergen, Minh Le, Marco Migueis, and

Robert Stewart

2016-070

Please cite this paper as:

Curti, Filippo, Ibrahim Ergen, Minh Le, Marco Migueis, and Robert Stewart (2016).
“Benchmarking Operational Risk Models,” Finance and Economics Discussion Se-
ries 2016-070. Washington: Board of Governors of the Federal Reserve System,
http://dx.doi.org/10.17016/FEDS.2016.070.

NOTE: Staff working papers in the Finance and Economics Discussion Series (FEDS) are preliminary
materials circulated to stimulate discussion and critical comment. The analysis and conclusions set forth
are those of the authors and do not indicate concurrence by other members of the research staff or the
Board of Governors. References in publications to the Finance and Economics Discussion Series (other than
acknowledgement) should be cleared with the author(s) to protect the tentative character of these papers.
 Benchmarking Operational Risk Models

∗
Filippo Curti, Ibrahim Ergen, Minh Le, Marco Migueis, and Robert Stewart

March 2, 2016
Abstract
The 2004 Basel II accord requires internationally active banks to hold regulatory
capital for operational risk, and the Federal Reserve’s Comprehensive Capital
Analysis and Review (CCAR) requires banks to project operational risk losses under
stressed scenarios. As a result, banks subject to these rules have measured and
managed operational risk more rigorously. But some types of operational risk -
particularly legal risk - are challenging to model because such exposures tend to be
fat-tailed. Tail operational risk losses have significantly impacted banks’ balance
sheets and income statements, even post crisis. So, operational risk practitioners,
bank analysts, and regulators must develop reasonable methods to assess the efficacy
of operational risk models and associated equity financing. We believe benchmarks
should be used extensively to justify model outputs, improve model stability, and
maintain capital reasonableness. Since any individual benchmark can be misleading,
we outline a set of principles for using benchmarks effectively and describe how these
principles can be applied to operational risk models. Also, we provide some examples
of the benchmarks that have been used by US regulators in assessing Advanced
Measurement Approach (AMA) capital reasonableness and that can be used in
CCAR to assess the reasonableness of operational risk loss projections. We believe no
single model’s output and no single benchmark offers a comprehensive view, but that
practitioners, analysts, and regulators must use models combined with rigorous
benchmarks to determine operational risk capital and assess its adequacy.
JEL Classification: G21, G28
Keywords: Banking Regulation, Risk Management, Operational Risk, Benchmarking

∗
Filippo Curti and Ibrahim Ergen are from the Federal Reserve Bank of Richmond and can be reached
at [email protected] and [email protected], respectively. Minh Le is from the Federal
Reserve Bank of Cleveland and can be reached at [email protected]. Marco Migueis is from the Board
of Governors of the Federal Reserve System and can be reached at [email protected]. Robert Stewart
is from the Federal Reserve Bank of Chicago and can be reached at [email protected]. We thank
Azamat Abdymomunov, Bakhodir Ergashev, Mike Gibson, David Jones, Paul Rallo, and participants at the
Governance, Compliance, and Operational Risk (GCOR) Conference of the Risk Management Association
(2015) for helpful comments. The views expressed in the paper are those of the authors and do not necessarily
represent those of the Board of Governors of the Federal Reserve System, the Federal Reserve Bank of
Chicago, the Federal Reserve Bank of Cleveland, or the Federal Reserve Bank of Richmond.
1 Introduction

US banks with total assets greater than $250 billion or foreign exposure greater than
$10 billion are required to use the advanced measurement approach (AMA) to compute
operational risk loss exposure at the 99.9% confidence level according to the final Basel
rule.1 To estimate operational risk loss exposure, most US banks subject to AMA use the
loss distribution approach (LDA). With the LDA, loss frequency and severity are modeled
separately and then convoluted through Monte Carlo simulation or analytical techniques,
such as Panjer recursion.2 Modeling high quantiles of a distribution is challenging even
when data is plentiful and the distribution is well behaved; the challenges increase when
modeling large losses that occur infrequently as is often the case with operational risk
losses - particularly legal losses - as documented by Danielsson (2002) and Danielsson
(2008). Many researchers have documented the uncertainty around operational risk capital
estimates measured at the 99.9th quantile. Mignola and Ugoccioni (2006) find that LDA
estimates of the 99.9th quantile suffer from considerable uncertainty due to data scarcity
and the fat-tailed nature of operational losses. Similarly, Cope et al. (2009) find that AMA
estimates suffer from significant uncertainty and criticize the Basel standard for operational
risk due to challenges in extrapolating beyond the empirical data to the 99.9th quantile.
The challenges of estimating the 99.9th quantile were also addressed by Nešlehová et al.
(2006), who highlighted the potential pitfalls with the application of extreme value theory
(EVT) to operational risk when parameter estimates result in infinite mean models.
Finally, Opdyke and Cavallo (2012) detail some of the challenges of modeling the 99.9th
quantile when faced with truncated data and using maximum likelihood estimation.
In addition to Basel regulatory capital requirements, the Federal Reserve requires bank
1
The rule can be found at 12 CFR part 3 for the Office of the Comptroller of the Currency (OCC) and
12 CFR part 217 for the Board of Governors of the Federal Reserve System.
2
Panjer (1981).

1
holding companies (BHCs) with assets exceeding $50 billion to conduct annual stress tests:
the Comprehensive Capital Analysis and Review (CCAR). As part of CCAR stress testing,
BHCs are required to project operational risk losses over a nine-quarter time horizon
assuming various economic scenarios. But BHCs have struggled to find meaningful
relationships between operational risk losses and the macroeconomy. The uncertainty
associated with estimating links between the macroeconomy and operational risk is
inherently high due to large and infrequent loss events dominating operational risk
exposure. This uncertainty is compounded by the short length of the datasets used for
estimation and by the difficulty in defining appropriate dates for operational loss events.
Some operational loss event types, such as internal fraud, occur over extended periods of
time; while other operational loss event types, such as legal cases, may result in payouts
years after the occurrence date. Therefore, the operational risk models used in stress
testing remain immature and will likely be significantly enhanced as practitioners and
regulators refine the process.
The above challenges are not the only issue operational risk models face: models are
also affected by banks’ incentives to take on tail risk as discussed by Rajan (2006), Acharya
et al. (2010) and Haldane (2010). Judgment is inevitable in statistical modeling, and
identical datasets can lead different modelers to different conclusions; even in cases when
modelers face identical incentives, reasonable and honest people often disagree on what
model to use. But when modelers face skewed incentives, modeling frameworks are
inevitably skewed. Due to government guarantees that create subsidies, banks have
incentives to minimize equity financing including operational risk capital as documented by
Admati (2014) and the International Monetary Fund (2014). Therefore, bank modelers
have incentives to adopt models that underestimate operational risk exposure.
To overcome these modeling issues and skewed incentives, we believe extensive
benchmarking should be employed by banks, regulators, and analysts. The Federal Reserve

2
and the Office of the Comptroller of the Currency (OCC) define benchmarking as ”the
comparison of a given model’s inputs and outputs to estimates from alternative internal or
external data or models.”3 Effective benchmarking of different models requires unique
techniques. Therefore, we propose the following five principles for developing operational
risk benchmarks: 1) benchmarks should be conceptually meaningful, 2) benchmarks should
be simple, 3) multiple benchmarks should be used, 4) discrepancies between model output
and benchmarks should be explained, and 5) benchmarks cannot replace models.

1. Benchmarks should be conceptually meaningful.

Gabaix and Laibson (2008) argue that models should be ”conceptually insightful”;
similarly, we believe that operational risk benchmarks should be conceptually
meaningful. In other words, the link between the benchmark and the model output
should be intuitive. Furthermore, benchmarks should facilitate comparisons of model
results and capital figures with peer banks, between subdivisions of banks, and
through time.

2. Benchmarks should be simple.

Benchmarks should be simple to compute, explain, and understand. A complicated
benchmark is not useful because benchmarks are often used to explain model results
to managers, regulators, and non-practitioners. In addition, a benchmark requiring
complicated estimation will likely be more prone to estimation error.

3. Multiple benchmarks should be used.

No single benchmark can track a complex model. Modelers should use a series of
benchmarks that address different aspects of the modeling framework.

4. Discrepancies between model output and benchmarks should be explained.

3
See OCC Bulletin 2011-12, ”Supervisory Guidance on Model Risk Management,” and Board SR letter
11-7, ”Supervisory Guidance on Model Risk Management.”

3
 When benchmark results do not match model output reasonably well, the validity of
results should be doubted; moreover, the disparity should be explainable or changes
should be made to the model.

5. Benchmarks cannot replace models.

Benchmarks do not aim to replace models but rather to allow for simple comparisons
that inform model developers and users regarding the robustness of the model and,
thus, enhance confidence in the model output.

These five principles may sometimes conflict, and practitioners will face tradeoffs when
applying them. For example, a more sophisticated benchmark may be necessary to address
certain model assumptions; therefore, practitioners will face tradeoffs between keeping the
benchmark simple and providing a conceptually meaningful benchmark. Furthermore, the
five principles are likely not exhaustive and should be adjusted as operational risk
modeling develops. Still, we believe these principles lay out a reasonable approach for
appropriate use of benchmarks in operational risk modeling frameworks.
The remaining sections illustrate the use of benchmarks with operational risk modeling
at US banks. Section II describes the data used in the analysis. Section III explains the
AMA and CCAR benchmarks considered by the Federal Reserve. Section III.a) focuses on
benchmarks driven by financial statement information, Section III.b) discusses benchmarks
derived from banks’ internal loss data, and Section III.c) discusses benchmarks that use
both financial statement information and internal loss data. Finally, Section IV concludes.

2 Data

As part of CCAR, BHCs with assets over $50 billion are required to provide operational
loss data to the Federal Reserve, and this data is used for our analysis. At the time of

4
analysis, data was submitted by thirty-one BHCs. The BHCs must report information on
all their operational loss events above an appropriate collection threshold including: dollar
amount, occurrence date, discovery date, accounting date, Basel II event type, and Basel II
business line.4 In addition, the analysis uses financial statement variables collected from the
FR Y-9C reports, including total assets, risk-weighted assets, total deposits, gross income,
and number of employees. The AMA capital figures used in our analysis are taken from the
FFIEC 101 Schedule S, and fifteen BHCs were reporting AMA capital numbers at the time
of the analysis.
Pooling data from various institutions leads to unique data challenges. BHCs report data
with different start dates and reporting lags. Some BHCs have more than ten years of data,
while others have less than five. In addition, BHCs have different loss collection thresholds.
Finally, missing data is inevitable. To address these challenges, our analysis uses a universal
internal threshold of $20,000, and banks with missing data for any particular benchmark
were excluded from the results for that benchmark. Furthermore, the analysis uses data
from 2005Q1 to 2014Q3 to balance reporting quality with a sufficient time period. Finally,
gross loss amounts, adjusted for inflation, are used.5

3 Benchmarks

3.1 Benchmarks Based on Financial Statement Information

For at least a century, capital ratios have been used to assess bank safety; therefore,
practitioners are well versed in the simple logic behind capital ratios.6 The numerator acts
as a measure of soundness, the denominator as a proxy for risk, and larger ratios signal a
4
For a detailed description of the operational risk data provided by BHCs to the Federal Reserve, see the
FR Y-14Q reporting form and instructions at www.federalreserve.gov/apps/reportforms.
5
The inflation rates we use are obtained from FRED at: http://research.stlouisfed.org/fred2/series/PCEPILFE#
and represent the increase in personal consumption expenditures excluding food and energy.
6
Mitchell (1909).

5
safer bank. Historically, equity financing usually makes up the numerator, while metrics
such as total assets and total deposits have been used as denominators. In the following
benchmarks, we use the AMA operational risk capital and the BHC stressed operational
loss projections from CCAR as the numerator or measure of soundness.7
In this section, ratios of AMA operational risk capital to five financial statement
variables are presented for the fifteen BHCs subject to operational risk AMA requirements
and for which Schedule S data was reported in 2014Q3. Similarly, ratios of the BHC
stressed projections from CCAR to five financial statement variables are presented for the
thirty-one BHCs subject to CCAR requirements. The reporting banks included in this
sample were at varying stages of operational risk model development. Specifically, only
eight of the thirty-one banks had AMA models approved by regulators. The following five
capital ratio benchmarks are in-line with the principles outlined in the introduction, as
they are simple and provide multiple perspectives on exposure. Also, these capital ratios
can be used for comparisons between banks and across time.

3.1.1 AMA Operational Risk Capital / Total Assets

& BHC Stress Projection / Total Assets

Abdymomunov (2014) and Abdymomunov and Curti (2015) showed that total
operational risk losses are significantly correlated with the asset size of banks; also, total
assets are a simple and transparent benchmark. Therefore, we believe total assets are a
useful operational risk benchmark. However, some banks have significant off-balance sheet
exposures due to derivatives and securitization that are not represented in the total assets
figure, while other banks do not. This discrepancy should lead practitioners to exercise
caution when using this benchmark for horizontal comparisons.
Table 1 displays various percentiles and the average of the AMA capital estimates
7
Stressed loss projections are a sensible measure of soundness because BHCs are required to hold a capital
buffer equivalent to these loss projections on top of their minimum regulatory capital.

6
divided by total assets for the fifteen banks subject to AMA requirements.

Table 1: AMA Operational Risk Capital / Total Assets

Percentile Ratio
10th 0.75%
25th 0.93%
50th 1.08%
75th 1.24%
90th 1.55%
Average 1.16%
Sample Size 15

Table 2 displays various percentiles and the average of the BHC stressed projections
divided by the total assets for the thirty-one banks subject to CCAR.

Table 2: BHC Stress Projection / Total Assets

Percentile Ratio
10th 0.24%
25th 0.46%
50th 0.63%
75th 0.84%
90th 1.20%
Average 0.69%
Sample Size 31

Ratios between AMA capital and total assets range from .7% to 1.6%, with an industry
median of 1.2%; while ratios between BHC stressed operational losses and total assets range
from .2% to 1.2%, with an industry median of 0.7%. Banks with less developed operational
risk modeling tend toward the lower ratios.

3.1.2 AMA Operational Risk Capital / Risk Weighted Assets

& BHC Stress Projection / Risk Weighted Assets

Total assets are limited as a measure of exposure because they do not account for
differences in asset riskiness. To address this limitation, the Basel accords created

7
”risk-weighted assets” (RWA) - a measure of exposure in which assets of different riskiness
are weighted differently. Table 3 and Table 4 below display this benchmark:

Table 3: AMA Operational Risk Capital / Risk Weighted Assets

Percentile Ratio
10th 1.21%
25th 1.37%
50th 1.98%
75th 2.17%
90th 2.88%
Average 1.91%
Sample Size 15

Table 4: BHC Stress Projection / Risk Weighted Assets

Percentile Ratio
10th 0.31%
25th 0.59%
50th 0.86%
75th 1.23%
90th 2.39%
Average 1.13%
Sample Size 31

Ratios between AMA capital and total RWAs range from 1.2% to 2.9%, with an industry
median of 2.0%; while ratios between BHC stressed operational losses and total RWAs range
from .3% to 2.4%, with an industry median of .9%.
To account for operational risk exposure, the Basel framework requires banks to multiply
AMA operational risk capital estimates by 12.5 to obtain operational risk RWAs, which are
then summed to credit and market RWAs to calculate total RWA. Total RWA are then used
for regulatory capital ratios. A table presenting operational risk RWAs as a percentage of
total RWAs is included in Appendix A. Operational risk RWAs as a percentage of total
RWAs range from 15% to 36% with a median of 25% for US AMA BHCs in 2014Q3. These
percentages demonstrate that operational risk is a significant risk for US BHCs.

8
3.1.3 AMA Operational Risk Capital / Basic Indicator Approach Capital
& BHC Stress Projection / Basic Indicator Approach Capital

Gross income (GI) is the proxy used in the Basel standardized approaches for operational
risk capital calculations. We calculate gross income for a year by summing net interest
income (item 3) and total noninterest income (item 5.m) from the schedule HI of the FR
Y-9C report. Then, to calculate the Basic Indicator Approach (BIA) capital for 2014Q3, we
average gross income for the three previous years (2011, 2012, and 2013), excluding negative
values, and multiply this average by 15%. Tables 5 and 6 display this benchmark.

Table 5: AMA Operational Risk Capital / Basic Indicator Approach Capital

Percentile Ratio
10th 1.2
25th 1.4
50th 1.7
75th 2.0
90th 2.2
Average 1.9
Sample Size 15

Table 6: BHC Stress Projection / Basic Indicator Approach Capital

Percentile Ratio
10th 0.5
25th 0.6
50th 0.9
75th 1.2
90th 1.9
Average 1.1
Sample Size 31

Ratios between AMA capital and BIA capital range from 1.2 to 2.2, with an industry
median of 1.7; while ratios between BHC stressed operational losses and BIA capital range
from .5 to 1.9, with an industry median of 0.9. In 2014Q3, fourteen of the fifteen US AMA
BHCs reported operational risk capital greater than the capital they would be required to

9
hold under the BIA, which indicates that the BIA likely underestimates the operational risk
exposure of large US BHCs. Although gross income has been used across the industry to
proxy operational risk, its reliability as a proxy has not been established.8

3.1.4 AMA Operational Risk Capital / Total Deposits

& BHC Stress Projection / Total Deposits

This benchmark is simple and difficult to game. However, it should not be used to
compare banks with different funding models. For example, despite facing operational risk
comparable or greater than traditional banks, investment banks rely much less on deposit
funding than traditional banks. Tables 7 and 8 display this benchmark.

Table 7: AMA Operational Risk Capital / Total Deposits

Percentile Ratio
10th 1.01%
25th 1.36%
50th 1.69%
75th 3.55%
90th 7.50%
Average 3.07%
Sample Size 15

Table 8: BHC Stress Projection / Total Deposits

Percentile Ratio
10th 0.32%
25th 0.65%
50th 0.96%
75th 1.95%
90th 3.58%
Average 1.51%
Sample Size 31

8
Recently, the Basel Committee proposed revisions to the standardized approaches for operational risk -
partly aimed at improving the risk sensitivity - which replace gross income as a proxy for operational risk.
Basel Committee on Banking Supervision (2014).

10
 Ratios between AMA capital and total deposits range from 1.0% to 7.5%, with an industry
median of 1.7%; while ratios between BHC stressed operational losses and total deposits
range from .3% to 3.6%, with an industry median of 1.0%.

3.1.5 AMA Operational Risk Capital / Number of Employees

& BHC Stress Projection / Number of Employees

The number of employees is a useful operational risk benchmark because employees are
likely a driver of operational risk. The more employees a firm has, the more opportunities
for an employee to cause losses intentionally or unintentionally. Tables 9 and 10 display this
benchmark:
Table 9: AMA Operational Risk Capital / Number of Employees

Percentile Ratio
10th 60,997
25th 70,004
50th 79,768
75th 126,803
90th 201,584
Average 114,326
Sample Size 15

Table 10: BHC Stress Projection / Number of Employees

Percentile Ratio
10th 19,131
25th 31,776
50th 44,926
75th 75,140
90th 114,656
Average 68,299
Sample Size 31

Ratios between AMA capital and number of employees range from 61.0k to 201.6k,
with an industry median of 79.8k; while ratios between BHC stressed operational losses

11
and total deposits range from 19.1k to 114.7k, with an industry median of 44.9k.
The benchmark ratios presented above compare banks’ capital estimates to financial
statement information and, thus, can be used to rank banks. However, these benchmarks
offer limited insight into whether the industry ratios are reasonable. For example, a ratio
above the industry median may not be conservative because the reasonableness of the
industry median ratio as a benchmark depends on the soundness of models across the
industry. To assess a safe level of capital, further analysis is required using banks’ loss
histories.

3.2 Benchmarks Based on Banks Internal Losses

To understand and compare banks’ operational risk loss exposure, loss experience
should be considered. Past loss experience is likely a good proxy for operational risk loss
exposure going forward, as past losses relate to the quality of risk controls and the riskiness
of the business environment. However, caution should be applied when making
comparisons based on internal loss benchmarks because banks are at different levels of data
collection. Some banks have collected up to twelve years of data and have a mature
collection process; while others have collected only a few years of data, with questionable
quality in the collection process. Still, the following five internal loss-based benchmarks can
be used to better understand banks’ capital adequacy.

3.2.1 AMA Operational Risk Capital / Maximum Loss

& BHC Stress Projection / Maximum Loss

To begin, a simple comparison of the banks’ capital projections divided by their largest
recorded loss is computed. Tables 11 and 12 display this benchmark.

12
 Table 11: AMA Operational Risk Capital / Maximum Loss

Percentile Ratio
10th 2.6
25th 3.0
50th 4.2
75th 9.2
90th 12.0
Average 6.0
Sample Size 15

Table 12: BHC Stress Projection / Maximum Loss

Percentile Ratio
10th 0.8
25th 2.2
50th 3.2
75th 6.2
90th 9.5
Average 4.4
Sample Size 31

Ratios between AMA capital and the largest historical loss range from 2.6 to 12.0, with an
industry median of 4.2; while ratios between BHC stressed operational losses and the largest
historical loss range from .8 to 9.5, with an industry median of 3.2. Estimates resulting from
prudent AMA and CCAR models should exceed the firm’s largest loss, but some banks are
still in the early stages of developing credible models.

3.2.2 AMA Operational Risk Capital / Maximum 4 Quarters Loss

& BHC Stress Projection / Maximum 9 Quarters Loss

AMA capital is designed to provide a buffer for operational risk losses over a four quarter
horizon, while BHC stressed operational loss projections are designed to provide an estimate
of losses under stressed scenarios over a nine quarter horizon. Therefore, comparing the
operational risk capital model outputs to actual worse case scenarios experienced by banks

13
over those time periods provides a valuable insight into the efficacy of the models. Tables
13 and 14 display these benchmarks:

Table 13: AMA Operational Risk Capital / Maximum 4 Quarters Loss

Percentile Ratio
10th 1.5
25th 1.9
50th 3.5
75th 4.7
90th 5.6
Average 3.6
Sample Size 15

Table 14: BHC Stress Projection / Maximum 9 Quarters Loss

Percentile Ratio
10th 0.6
25th 1.1
50th 2.0
75th 3.7
90th 5.5
Average 2.8
Sample Size 31

Ratios between AMA capital and largest consecutive four quarter losses range from 1.5
to 5.6, with an industry median of 3.5; while ratios between BHC stressed operational losses
and largest consecutive nine quarters losses range from .6 to 5.5, with an industry median of
2.0. Given that most banks have less than fifteen years of data, safety and soundness should
likely lead to ratios well above one for both AMA and CCAR models.

3.2.3 AMA Operational Risk Capital / Average 4 Quarters Loss

& BHC Stress Projection / Average 9 Quarters Loss

Comparing AMA capital and stressed loss projections to average losses is also useful.
Tables 15 and 16 display these benchmarks.

14
 Table 15: AMA Operational Risk Capital / Average 4 Quarters Loss

Percentile Ratio
10th 5.6
25th 7.4
50th 11.6
75th 13.8
90th 16.5
Average 11.4
Sample Size 15

Table 16: BHC Stress Projection / Average 9 Quarters Loss

Percentile Ratio
10th 1.6
25th 2.2
50th 3.5
75th 6.4
90th 9.6
Average 4.6
Sample Size 30

Ratios between AMA capital and average four quarter losses range from 5.6 to 16.5,
with an industry median of 11.6; while ratios between BHC stressed operational losses and
average nine quarters losses range from 1.6 to 9.6, with an industry median of 3.5. Given
that AMA capital is supposed to cover tail losses and BHC stressed projections are supposed
to cover stressed losses, both of these ratios should be significantly above one.

3.2.4 AMA Operational Risk Capital / Most Recent 4 Quarters Loss

& BHC Stress Projection / Most Recent 9 Quarters Loss

Recent losses may be more representative of a firm’s current risk exposure. Therefore, a
benchmark that highlights recent losses may be useful to identify concerning trends, but this
benchmark is more volatile than other benchmarks (see the coefficient of variation statistics
in Appendix B). Tables 17 and 18 display these benchmarks.

15
 Table 17: AMA Operational Risk Capital / Most Recent 4 Quarters Loss

Percentile Ratio
10th 16.0
25th 23.3
50th 39.8
75th 47.5
90th 63.8
Average 38.3
Sample Size 15

Table 18: BHC Stress Projection / Most Recent 9 Quarters Loss

Percentile Ratio
10th 2.1
25th 3.0
50th 5.2
75th 11.1
90th 13.2
Average 7.1
Sample Size 30

Ratios between AMA capital and most recent four quarter losses range from 16.0 to 63.8,
with an industry median of 39.8; while ratios between BHC stressed operational losses and
average nine quarters losses range from 2.1 to 13.2, with an industry median of 7.1.

3.2.5 AMA Operational Risk Capital / 99.9 Empirical Bootstrap (4 quarters)

& BHC Stress Projection / 99.9 Empirical Bootstrap (9 quarters)

This benchmark uses a LDA framework, but with some unique assumptions to ensure
comparability. The process to compute this benchmark begins with separating each bank’s
losses into the seven Basel event types which are used as units of measure. Then, like a
traditional LDA, loss frequency and severity are modeled separately and assumed
independent. To model frequency, let N(t) be a Poisson random variable with intensity λ,
which represents the number of losses in the time interval [0,t] (assumed to be four

16
quarters for the AMA benchmark and nine quarters for the CCAR benchmark):

(λt)n
P (N (t) = n) = expλt , (1)
n!

To model severity, assume Xi is a positive random variable describing the magnitude

of losses. While traditional LDA models assume that loss severity follows a parametric
distribution, this benchmark uses the empirical distribution of observed loss severities to
define the severity distribution. To combine the frequency and severity distributions, we
calculate event type aggregate losses, S(t):

N (t)
X
S(t) = Xi (2)
i=1

Monte Carlo simulation is used to obtain the distribution of aggregate losses for each
event type. In each simulation path, first, a loss count is drawn from the Poisson distribution;
then, the corresponding number of loss severities are drawn from the empirical distribution
and these severities are added up. To obtain the 99.9th quantile of the total annual loss
distribution, the 99.9th quantile of the aggregate annual loss distributions of the seven event
types are summed (i.e., we assume full dependency between event types). Tables 19 and 20
display these benchmarks:

Table 19: AMA Operational Risk Capital / 99.9 Empirical Bootstrap (4 quarters)

Percentile Ratio
10th 0.8
25th 1.0
50th 1.5
75th 2.7
90th 3.0
Average 1.7
Sample Size 15

17
 Table 20: BHC Stress Projection / 99.9 Empirical Bootstrap (9 quarters)

Percentile Ratio
10th 0.2
25th 0.6
50th 1.0
75th 1.5
90th 2.6
Average 1.3
Sample Size 31

Ratios between AMA capital and the historical simulation benchmark range from .8 to
3.0, with an industry median of 1.5; while ratios between BHC stressed operational losses
and the historical simulation benchmark range from .2 to 2.6, with an industry median of
1.0. By using the empirical distribution of severities as the severity distribution, this
benchmark assumes that a bank will never experience a loss event bigger than the largest
loss event the bank has experienced in the past. Therefore, this benchmark should likely be
a floor for AMA models.
The historical simulation benchmark is more complicated than the other benchmarks
explored making results susceptible to estimation error, but the benchmark is conceptually
meaningful as it provides for easy comparison between banks, while including many of the
assumptions of the more sophisticated AMA models used by banks. Furthermore, the
historical simulation benchmark can be useful to assess whether changes in model
estimates are sensible. For example, the historical simulation benchmark can provide a
robustness check for data updates. If the historical simulation estimate increases by 100%
for a particular unit of measure, an increase of 80% on the model estimate is likely
reasonable; while if the historical simulation estimate barely changes after a data update,
then a 80% change in model estimates is likely the result of an unstable model.

18
3.3 Benchmarks Based on Financial Statement Information and

Internal Loss Data

As discussed in Section I, the 99.9th quantile standard for AMA raises modeling
challenges. The following benchmark, which combines financial statement information and
internal loss data, offers insight into reasonable AMA capital numbers given past industry
losses. Specifically, we use the distribution of ratios of cumulative four consecutive quarters
of operational losses to total assets in the quarter prior to the loss window to derive a
benchmark for AMA capital. To estimate this benchmark we use data for all AMA banks
from 2000Q1 to 2014Q3, which includes more than 600 ratios. The distribution of these
ratios is used to identify the ratio number between AMA capital and total assets that
would reflect the 99.9 confidence level, if exposure was homogenous across the industry and
unchanged from what was observed in the past. Figure 1 displays the distribution of ratios:

Figure 1: Distribution: One Year OpRisk Loss / Total Assets

The 99.9 quantile for this distribution slightly exceeds 1.3%. Using similar logic, in Figure
2, the distribution of ratios is presented using gross income instead of total assets:

19
 Figure 2: Distribution: One Year OpRisk Loss / Gross Income

The 99.9 quantile for this distribution of ratios slightly exceeds 34%. Similar combined
benchmarks can be computed for CCAR, using nine-quarter losses instead of the four quarter
losses used for AMA. The distribution of ratios for CCAR is displayed in Figures 3 and 4.

Figure 3: Distribution: Nine Year OpRisk Loss / Total Assets

20
 Figure 4: Distribution: Nine Quarter OpRisk Loss / Total Assets

These distributions show the range of operational losses experienced in the industry in
the last fifteen years. However, restraint should be used when evaluating the high quantiles of
these distributions to assess the credibility of AMA and CCAR estimates. Despite using the
full history of industry losses reported to the Federal Reserve, the distribution of observed
ratios still only includes around 600 ratios and, thus, the estimate of the 99.9th quantile
suffers from significant uncertainty, as it results from interpolation of the largest two data
points. Moreover, operational loss exposure varies significantly across banks and is not
always proportional to size.

4 Conclusion

This paper provides guidelines for using benchmarks in conjunction with operational
risk models. Operational risk is difficult to model at high quantiles and difficult to link to
macroeconomic factors. Furthermore, banks’ incentives to minimize capital lead banks’
operational risk modelers to make less conservative assumptions in their models. Therefore,

21
benchmarks should be an integral part of any operational risk modeling framework.
Benchmarks should be used by modelers, regulators, and analysts to gauge the efficacy of
the models and assess the reasonableness of results.
Still, misuse of operational risk benchmarks may be more detrimental than failing to
use benchmarks at all. Therefore, we offer five principles to ensure robustness for
operational risk benchmarks. Benchmarks should be conceptually meaningful. Benchmarks
should be simple. Multiple benchmarks should be used. Benchmarks should explain
discrepancies. And benchmarks cannot replace models. Operational risk benchmarks must
be practical, easily understood, and lead to meaningful changes. But benchmarks cannot
replace the detail found in models; rather, operational risk benchmarks allow for
comparability across banks and over time. Furthermore, practitioners will have to balance
the different principles within the modeling framework when tracking and recalibrating
their operational risk models. These simple principles offer direction around what role
operational risk benchmarks should play in an operational risk modeling framework.
The benchmarks presented in this paper illustrate the range of outcomes in the
industry. Thus, this paper allows practitioners to compare their models estimates with the
estimates from other large US BHCs. These benchmarks are examples of the benchmarks
currently being used by US banks and regulators, but the examples should not be
considered complete. Rather, benchmarks should constantly be re-evaluated and updated
as models change, regulations change, and economic factors change. With these inevitable
changes, new operational risk benchmarks should be developed and poorly performing
benchmarks should be retired. The process must be dynamic. But a sound operational risk
modeling framework combined with ample use of operational risk benchmarks offers the
best way forward for measuring and managing operational risk.

22
References
Abdymomunov, A. (2014). Banking sector operational losses and macroeconomic environment. Working
Paper,Federal Reserve Bank of Richmond.

Abdymomunov, A. and Curti, F. (2015). Improving the robustness of operational risk modeling through
external data scaling. Working Paper,Federal Reserve Bank of Richmond.

Acharya, V., Cooley, T., Richardson, M., and I., W. (2010). Manufacturing tail risk: A perspective on the
financial crisis of 2007-09. Foundations and Trends in Finance, 4:247–325.

Admati, A. (2014). Statement for senate committee on banking, housing and urban affairs subcommittee on
financial institutions and consumer protection. Hearing on ”Examining the GAO report on expectations
of government support for bank holding companies”, July 31, 2014.

Basel Committee on Banking Supervision (2014). Operational risk - revisions to the simpler approaches.
Bank of International Settlements.

Cope, E., Mignola, G., Antonini, G., and Ugoccioni, R. (2009). Challenges and pitfalls in measuring
operational risk from loss data. The Journal of Operational Risk, 4(4):3–27.

Danielsson, J. (2002). The emperor has no clothes: Limits to risk modeling. Journal of Banking & Finance,
26(7):1273–1296.

Danielsson, J. (2008). Blame the models. Journal of Financial Stability, 4(4):321–328.

Gabaix, X. and Laibson, D. (2008). The Seven Properties of Good Models, The Foundations of Positive and
Normative Economics: A Handbook. Oxford University Press.

Haldane, A. (2010). The $100 billion question. Comments by Andrew G Haldane, Executive Director,
Financial Stability, Bank of England, at the Institute of Regulation & Risk, Hong Kong, 30 March 2010.

International Monetary Fund (2014). Global financial stability report. Washington: International Monetary
Fund, April 2014.

Mignola, G. and Ugoccioni, R. (2006). Sources of uncertainty in modeling operational risk losses. The
Journal of Operational Risk, 1(2):33–50.

Mitchell, W. C. (1909). The decline in the ratio of banking capital to liabilities. Quarterly Journal of
Economics, 23(4).

Nešlehová, J., Embrechts, P., and Chavez-Demoulin, V. (2006). Infinite mean models and the lda for
operational risk. The Journal of Operational Risk, 1(1):3–25.

Opdyke, J. and Cavallo, A. (2012). Estimating operational risk capital: The challenges of truncation,
the hazards of maximum likelihood estimation, and the promise of robust statistics. The Journal of
Operational Risk, 7(3):3–90.

Panjer, H. (1981). Recursive evaluation of a family of compound distributions. ASTIN Bulletin, 12:22–26.

Rajan, R. (2006). Has financial development made the world riskier? European Financial Mangement,
12(4):499–533.

23
Appendix A

AMA Operational Risk RWA / Total RWA

Percentile Ratio
10th 15.1%
25th 17.1%
50th 24.7%
75th 27.2%
90th 36.0%
Average 23.9%
Sample Size 15

24
 Appendix B Descriptive Statistics
Benchmarks based on financial statement information:
AMA - Financials

Ratio N Mean SD CV 10% 25% 50% 75% 90%

OpRisk Capital / Total Assets 15 1.2% 0.4% 36.8% 0.7% 0.9% 1.1% 1.2% 1.7%
OpRisk Capital / Gross Revenue 15 28.2% 16.0% 56.9% 17.0% 21.3% 24.3% 30.7% 33.5%
OpRisk Capital / Total RWA 15 1.9% 0.7% 34.2% 1.2% 1.3% 2.0% 2.2% 3.0%
OpRisk Capital / Total Deposits 15 3.1% 2.7% 87.8% 0.9% 1.3% 1.7% 4.5% 8.2%
OpRisk Capital / BIA 15 1.9 1.0 50.5% 1.1 1.4 1.7 2.0 2.2
OpRisk Capital / Number of Employees 15 114,325 76,280 66.7% 60,564 67,098 79,767 137,119 232,117

CCAR - Financials

Ratio N Mean SD CV 10% 25% 50% 75% 90%

BHC Stress / Total Assets 31 0.7% 0.4% 52.4% 0.2% 0.4% 0.6% 0.9% 1.2%
BHC Stress / Gross Revenue 31 15.9% 11.2% 70.4% 6.9% 9.4% 13.2% 17.3% 27.9%
BHC Stress / Total RWA 31 1.1% 0.9% 79.6% 0.3% 0.5% 0.9% 1.2% 2.4%
BHC Stress / Total Deposits 30 1.5% 1.5% 97.6% 0.2% 0.5% 0.8% 1.3% 2.3%
BHC Stress / BIA 31 1.1 0.7 70.4% 0.5 0.6 0.9 1.2 1.9
BHC Stress / Number of Employees 31 68,298 70,782 103.6% 19,131 31,710 44,925 82,004 114,655

25
 Benchmarks based on banks’ internal losses:
AMA - Internal Losses

Ratio N Mean SD CV 10% 25% 50% 75% 90%

OpRisk Capital / Max Loss 15 6.0 3.8 64.6% 2.6 3.0 4.2 9.2 12.0
OpRisk Capital / Max 4q 15 3.6 1.9 54.0% 1.5 1.9 3.5 4.7 5.6
OpRisk Capital / Average 4q 15 11.4 5.0 44.0% 5.6 7.4 11.6 13.8 16.5
OpRisk Capital / 99.9 Emp. Bootstrap 15 1.7 0.9 54.3% 0.8 1.0 1.5 2.7 3.0
OpRisk Capital / Most Recent 4q 15 38.2 20.7 54.1% 16.0 23.3 39.8 47.5 63.8
OpRisk Capital / Historic 99.9 TA Ratio 15 90.5% 33.3% 36.8% 19.5% 68.9% 81.9% 94.3% 108.7%
OpRisk Capital / Historic 99.9 GI Ratio 15 84.4% 43.4% 51.5% 9.9% 54.1% 69.6% 89.0% 98.0%

CCAR - Internal Losses

Ratio N Mean SD CV 10% 25% 50% 75% 90%

BHC Stress / Max Loss 31 12.6 16.4 130.0% 1.2 2.2 5.6 15.2 39.5
BHC Stress / Max 9q 31 2.8 2.3 82.1% 0.6 1.1 2.0 3.7 5.5
BHC Stress / Average 9q 30 4.6 3.2 69.3% 1.7 2.2 3.8 6.9 10.3
BHC Stress / 99.9 Emp. Bootstrap 31 1.3 1.0 78.1% 0.2 0.6 1.0 1.6 2.6
BHC Stress / Most Recent 9q 30 7.1 4.9 68.4% 2.2 2.9 5.7 12.1 12.9
BHC Stress / Historic 99.9 TA Ratio 31 37.1% 19.6% 53.0% 12.7% 24.8% 33.6% 47.9% 65.6%
BHC Stress / Historic 99.9 GI Ratio 31 30.6% 21.5% 70.4% 13.2% 18.1% 25.5% 33.4% 53.9%

26