Applied Economics and d Finance

Vool. 2, No. 2; May

M 2015
ISSN 2332--7294 E-ISSN 2332-7308
Published by Redfame Publishing
P
URLL: http://aef.redffame.com

Modelinng Banks’’ Probabillity of Deefault

Xiaooming Tong1
1
Global Crediit Market, Citi Group, New Y
York, USA
Correspondennce: Xiaoming Tong, Global Credit Markett, Citi Group, N
New York, USA
Received: Febbruary 3, 2015 Accepted: Febbruary 26, 2015 Available onliine: March 23, 2015
doi:10.11114/aaef.v2i2.739 URL: http://dxx.doi.org/10.111114/aef.v2i2.739

Abstract
The unpreceddented financiaal crisis of 20008-2009 has caalled attentionn to limitationss of existing m methods for esstimating
the default risk of financial intuitions. Over the past deccade, we have had consideraable success at predicting deffault and
credit relativee value usingg Merton-typee structural m models and H Hybrid Probabiility of Defauult models. However, H
generating accurate model--based estimattes of default probabilities ((PDs) for finaancial firms haas proven diffficult. To
address this neeed, I built andd tested a timee-adaptive statiistical model tthat predicts thhe default probbabilities of banks. The
model is a loggistic regression whose inpuut variables arre selected baased on their ppast effectivenness at predictiing bank
failures and w
whose inclusionn in the model and weights aare to be updatted quarterly. M Model perform mance at discrim minating
between defaaults and non--defaults was evaluated forr horizons off one to five years using a sequence off annual
walk-forward out-of-samplee tests from 19992 to 2012. I tested the abiility of the moodel to predictt absolute defa ault rates
out to five yeaars and, exceptt for underestiimating the higgh bank defaullt rates during the credit crisis, the models perform
well at estimaating the annuaal bank defaultt rates. Becausse most defaullt models provvide little benefit over agency y ratings
for low-rated credits, I exammined the perfoormance of thee model to Krooll agency ratinngs only for thhose banks rate ed above
single-B-minuus or above siingle-C-minus.. Although default predictioons from agenncy ratings falll off rapidly fo or banks
rated at or aboove single-B and
a single-C, tthe time-adaptiive statistical m model predictiions deterioratte far less. Acc curacy at
predicting bannk defaults ussing agency raatings decreasees to near chaance at a prediiction horizon of five years, but the
time-adaptive statistical model continues tto perform weell above chancce at all horizoons. I also pressent a detailed analysis
of the contribuutions of finanncial variabless to model outpputs by year (22000-2012) annd tenor (1-5 yyears) and evaluate the
consistency of variable conntributions oveer time. The m model performss favorably at predicting deffaults, even re elative to
the best non-financial corpporate default models, with a 97% accurracy ratio (AR R) at one yearr prior to defa ault, and
decreasing, buut still above-cchance predicttive power outt to five years. I find that baanks’ quality oof assets and return
r on
equity are moost important for
fo predicting nnear term defaaults, giving w way at longer hhorizons to opeerating income e and the
yield on earninng assets.
Keywords: baank default, crredit risk, defauult risk
1. Introductioon: The Bank
k Default Mod
del
Over the passt decade, wee have had cconsiderable ssuccess at preedicting defauult and creditt relative valu ue using
Merton-type sstructural moddels, such as Moody’s/KM MV model (Vassicek, 1988; K Kealhofer, 19999) and Citi’ss Hybrid
Probability off Default (HPD D) model (Sobbehart and Keeenan, 2002; 22003). Howeveer, generating accurate mod del-based
estimates of ddefault probabbilities (PDs) ffor financial fi
firms has provven difficult. S Some reasons for this are fin nancials’
high levels off leverage, the relative opaciity of their asssets and liabilities, potential support from governments, extreme
risk of “tail evvents” and reggulatory changees. The numerrous bank failuures amid the ffinancial crisiss of 2008-2009 9 and the
subsequent rattings downgraades of many fi financial firms have highlighhted limitationss of agency creedit ratings and
d current
credit models to anticipate defaults for financial firms. During the crrisis, many ban anks went fromm apparent solv vency to
default in a vvery short perriod of time ppresumably reeflecting the pparticular senssitivity of finaancial institutiions and
insurance com mpanies to suudden decliness in investor cconfidence. A Although the ccredit ratings of financial firms
f are
m Vazza and Krraemer (2012) in Figure 1 deemonstrate tha
concentrated iin the investmeent grade rangge, results from at despite
their higher crredit ratings, financial
fi firms have a faster and steeper paath to default tthan their non--financial coun
nterparts.
Investors are becoming inccreasingly inteerested in bettter assessing and managingg their credit exposure to financial
institutions. A
Also, the U.S. Office
O Comptroller of the Currencyy (OCC), in acccordance withh the Dodd-Frank Act,
of the C
has published final rules (Deepartment of thhe Treasury, 2012) that remoove referencess to credit ratinngs from its reggulations

29
Applied Economics and Finance Vol. 2, No. 2; 2015

pertaining to investment securities, securities offerings, and foreign bank capital equivalency deposits.1 Amid this
backdrop, the development of accurate models for assessing bank credit risk appears critical both for managing
exposure to financial firms and for compliance with Federal regulations.

Figure 1. Paths into Default for Financial and Non-Financial Firms from Vazza and Kraemer (2012)
Source: Standard & Poor’s
For the development of the time-adaptive statistical model that predicts PDs of banks, I used information contained in
banks’ financial statements as published by the U.S. Federal Deposit Insurance Corporation (FDIC). As of March 2013,
there were 7,019 depository institutions in the US reporting to the FDIC with total liabilities of $12.8 trillion.2 Potential
inputs to the models are financial ratios found, in preliminary analyses, to be effective in forecasting future bank failures.
A series of models predicting default at one- to five-year horizons are computed annually, and the outputs are
predictions of annual marginal default probabilities for each bank from one to 30 years. I back-tested the model’s ability
to predict defaults of US depository institutions between 2000 and 2012 using bank data since 1992. For those studies, I
evaluated model performance using a walk-forward procedure. That is, to estimate default risk in any test year, I use
only information before that year to select model variables and calibrate the model coefficients.
The bank default model is represented in Figure 2. The left panel shows the functional form of the model, a logistic
regression, and the most recent coefficients for the one-year model, last updated in 2013 using data up to the end of
2012. As described further below, the variables for each model are chosen based on their relative Bayesian Information
Coefficients (BIC), the measure of information contribution, and criterion for inclusion in the model.3 The middle panel
shows how the individual variables are fit to linear regressions and summed prior to input into the non-linear function,
Ø(z). I built models to predict defaults over yearly horizons from one to five years, each assuming survival (i.e.,

1
Section 939A of the Dodd–Frank Act requires federal agencies to review regulations that require the use of an
assessment of creditworthiness of a security or money market instrument and any references to, or requirements in,
those regulations regarding credit ratings. Section 939A then requires the agencies to modify the regulations identified
during the review to substitute any references to, or requirements of, reliance on credit ratings with such standards of
creditworthiness that each agency determines to be appropriate.
2
Information about aggregate bank sector size obtained from the FDIC “Statistics on Banking”, which is accessible
online at http://www2.fdic.gov/sdi/sob/.
3
That is, as described below, I used the BIC as a criterion for variable selection, with those variables with the highest
individual BIC chosen first.

30
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

non-default) uup to the year of prediction. The panel at tthe right in Figgure 2 shows C Cumulative A Accuracy Profille (CAP)
curves for thee five annual prediction
p horizons. The CA AP curves meassure the extentt to which the model scores serve to
separate defauulters (tendenccy to have highh values of Ø(zz)) from non-deefaulters (likellihood of loweer values of Ø(z)).
Figure 2. Loggistic Regressioon Model for B Bank Defaultss. Left: Functioonal form of L
Logistic Regresssion and Mosst Recent
Set of Input V
Variables; Midddle: Illustratioon of How Inddividual Variabbles Feed the NNon-Linear Regression; Rigght: CAP
Curves for Moodels to Predicct 1- to 5-Year Defaults

The model perrforms well at identifying thhe riskiest bankks. For examplle, the right paanel of Figure 2 demonstratess that the
10% of bankss with the larggest values off Ø(z) includde 94% of the banks that deefaulted withinn one year.4 Although
A
performance ddrops monotonnically when ppredicting defaaults for each subsequent yeear from years two to five (a
assuming
survival to thhe start of eachh year), the m
models perform
m significantlyy above chancce for all yearrs. That is, forr models
predicting deffault in years tw
wo, three, fourr, and five, thee10% of banks with the largeest values of Ø
Ø(z) include 80%, 68%,
55%, and 40% % of the defaullting banks in tthe sample.
t constructioon of the model including the method oof variable seelection, walk-forward
In this reportt, I describe the
validation, andd a detailed diiscussion of m
model performaance. I also desscribe further ffeatures of thee model. In parrticular, I
show how vallues of Ø(z), ouutput from the bank model, aare mapped to historical defaault rates to esttimate physica al default
probabilities. I also describe further validdation studies and compare model perform mance with esstimations of bank
b risk
derived from agency creditt ratings. Finaally, I present detailed studiies of the conntributions of vvariables, high hlighting
which variablees are most higghly predictivee of bank defauults and the peeriods over whhich they are mmost effective.
2. Constructiing the Bank PD
P Model
2.1 Financial Variables as Predictors
P of D
Default
Since the pionneering work of o Beaver (19666) and Altmaan (1968), finaancial modelerrs have realizedd that certain financial
ratios are highhly predictive of a firm’s futture default. T For instance, I found that banks with
The same is truue for banks. F
low, especiallyy negative, retturn on equity (ROE) are muuch more likelyy to default. Inntuitively, bankks with low or negative
profitability w
will likely struuggle to pay thheir liabilities on time and wwill have difficulty finding additional funding. To

4
If the modell performed at chance, only 10% of the deffaulters wouldd be included inn the 10% of thhe sample with
h the
highest valuess of Ø(z).

31
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

illustrate this eeffect, each paanel in Figure 3 displays norrmalized distribbutions of ROOEs for defaultiing and non-de efaulting
banks. Distribbutions are shoown for one-, tw wo-, three-, annd four-year hoorizons in succcessive panels..5 Inspection of
o Figure
3 reveals thatt banks with loow ROE are m much more likkely to defaultt than those w with high ROE Es. Also, the prredictive
power of the ROE as regarrds default deccreases with inncreases in thee time horizonn. That is, the distributions of o ROEs
from defaultinng and non-deefaulting bankks are clearly aapart from eacch other at one- and two-yeear horizons, but b those
differences naarrow, becominng very small at four years out. I ran t-tessts on the diffferences betweeen the distribu utions of
ROEs for defaaulting and nonn-defaulting banks cease to bbe significant oover four yearrs out.

Figure 3. D
Distributions of Normalized R
ROEs for Defa
faulting and Noon-Defaulting Banks at One--, Two-, Three--, and
Four-Year Horizonss
Source: FDIC
C
A similar testiing procedure as illustrated in Figure 3 foor ROE revealeed other finanncial ratios thatt are useful for default
prediction. Thhese include firms’
fi leveragee ratios, ratios of non-perforrming to perfoorming loans, and net loans to bank
capital, to nam
me a few. A chhallenge in preedicting defauult is to select an appropriatee set of variabbles and combiine them
appropriately in a multivariaate model. To do this, I empployed a walk-forward logisttic regression ttechnique. The e logistic
regression funnction (describbed in the following section) is commonly uused for prediccting variabless with binary outcomes,
o
when the inputss are non-lineaarly related to the desired ouutput. The walk forward meethod constructs a new
particularly w
model each yyear from the candidate varriable set, whiile adding the data from thee previous yeaar to the deve elopment
sample. For vvariable selecttion in each nnew model, I uuse an automaated proceduree called forwaard stepwise selection,
which is explaained in detail below.
2.2 Logistic R
Regression
For those unffamiliar with logistic regresssion, I descriibe that methood briefly in tthis section. L Logistic regression has
similarities to the more fam miliar multiple linear regresssion method, bbut involves ann extra step, thhe logistic tran nsform. I
illustrate this ggraphically forr a set of hypoothetical input variables in F
Figure 4. The aapplication beggins with selection of a
set of candidaate financial vaariables, denotted xi, i=1, … …, n. The inpuuts, xi, could bee financial ratiios or other qu
uantities.
The lower portion of Figurre 3 depicts hoow values of hhypothetical innput variables (the circles inn each plot) are a fit by
functions, of tthe form

f (xi )   i   i xi (1)

to derive consstants, i, and coefficients,

c i, for each inpuut variable.
form in Equatioon 1. For variaable x1for the example
Then, for a giiven set of inpuuts, each xi is put through itts linear transfo
in Figure 3, the constant 1=0 and thee coefficient β1=-3. Thus, if 1=0.5 as shown in thee figure, f (x x1 ) =-1.5.
Hypothetical ffunctions and outputs
o for x2 and xn are allso shown in Figure 4.
The resulting outputs of thee first stage of the logistic reggression, the vvalues f (xi ) , are summed att an intermediate stage
whose output z can be represented as

z   0    i xi (2)
i1

where

5
Because financial ratios suuch as ROEs ccan have very ddispersed distrributions, I connverted firms’ ROEs into sta
andard
normal distribbutions before plotting. This transformationn does not channge the orderinng of firms onn the ROE axis.

32
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

0    i (3)
i1

For the exampple in Figure 3,3 the resultingg value of z is assumed to bee -1.2.6 The vaalue of z from Equation 2 is then put
through the loogistic transfoorm that serves to constrainn the output off the regressioon to a value between 0 and 1. For
example, for tthe default moddel, the resultiing PD is givenn as:
1
D
PD (4)
1  e z
where the resuulting value off PD for z=-1.22 is 0.26 or a 226% probabilityy of default ovver the time fraame in question.
Figure 4. Logistic Regressioon Function. L
Linear Transforrmations of Fiinancial Variabbles (Lower Pllots) are Summ
med at an
Intermediate S
Stage and Put Through
T Logistic Transfform (Top Grapph) Which Connverts the Outtput to a Value between
the L
0 and 1 (0% aand 100%)

2.3 Automatedd Selection of Input

I Variablees
Note that the ooverall plan iss to derive a neew model eachh year, incorpoorating into thee learning sammple the data frrom each
successive yeear’s defaultingg and non-deffaulting firms.. Because the factors that iinfluence defaaults and theirr relative
contributions may change over time, I choose to use an aadaptive proceddure for selectting variables ffor each annua al model.
I first assembbled a set of 20
2 candidate ffinancial ratioss that have beeen shown to bbe predictive of subsequentt default.
Because the ddistributions ofo different finnancial ratios can vary wideely, I chose too standardize all input varia ables via
transformationn into standardd normal distribbutions beforee testing their uusefulness as innputs to each aannual model.
The process oof model consttruction beginss with only thhe logistic funcction and no vvariables choseen for inclusio
on. Then,
for each canddidate input vaariable, I builld a logistic ddefault model by selecting vvalues of  i and  i for each
variable that eenables the beest prediction of default on the developmment sample. T That is for eachh input variabble xi , I
solve for  i and  i in thhe following equation for PD D:
1
PD   (  i   i xi )
(5)
1 e

6
Note that thhis value of z cannot be deduced from the vvalues shown iin Figure 4 as iit is assumed to have contributions
from variabless x3 to xn-1thaat are not givenn in the figure.

33
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

The variable w with the greateest predictive ppower with resspect to defaullt is chosen as the first input variable. As described
d
in further detaail below, I choose the Bayesiian Information Criterion (BBIC) developedd by Schwartz (1978) as the measure
of predictive power. The BIC B measures hhow well the model fits thee data, but alsso imposes a ppenalty for having too
many variablees, thereby guaarding against overfitting thhe data. After sselection of the first variablee, the process repeated
to select a seccond variable, and so on, unttil model perfoormance ceasees to improve. Once all the vvariables for th he model
are selected, the value of the constant  0 and coeffficients  i (i  1,..., n) forr each of the variables are e refit to
minimize the error in the loggistic regressioon equation:
1
PD  n (6)
 ( 0    x ) i i

1 e i1

Figure 5. Leftt: One-Year Baank PD Model Equation, Varriables, and Coorresponding C Coefficients (G
Green: List Risky; Red:
More Riskky), With Variaables Listed inn the Order Selected. Right: BBayesian In forrmation Criterrion (BIC) for Each
Successivee Variable Seleected.
An illustrationn of the resultss of variable seelection is presented in Figuure 5. The top portion of the left panel disp plays the
logistic regresssion equation,, with the tablee below it listinng the input vaariables to the model in the oorder in which they are
selected. Thatt is, variables are listed in descending orrder of their ppredictive pow wer. The BIC values resultiing from
inclusion of eeach variable are
a also displayyed. The rightt portion of Figgure 5 is a ploot of the BIC vvalues that ressult from
the inclusion of each variabble. For instancce, the model starts with onnly a constant tterm whose BIIC value is 7,4 459. The
variable selecttion proceduree determined thhat banks’ retuurn on equity ((ROE) providees the largest ppredictive pow wer of all
candidate variiables, and its inclusion in thhe model achieeves a BIC of 44,092. After seelection of the ROE, the proc cedure is
run again, piccking the Liabiility/Asset ratiio as the best oof the remainiing candidate vvariables, brinnging the BIC down to
3,568. This pprocedure conttinued until thhe BIC could no longer be decreased. At that point, ssix variables had h been
selected and thheir corresponnding coefficiennts appear in tthe left table off Figure 5.
2.4 The Term Structure of Baank PDs
The method I have descriibed can be uused to prediict defaults ovver one- to ffive-year horizzons. Howeve er, some
applications (ee.g., long-term Ds over longer periods.
m investment pportfolios) requuire estimationn of the term sstructure of PD
My approach to extending the term struucture of bankk PDs for term ms beyond fivve years is to use long-term m annual
average margiinal default rattes determinedd from historicaal data on bankk defaults.
Construction of PD term sttructures beginns by using the set of five logistic regresssion models, eeach developed d for the
marginal defaault rate betweeen successivee years over a period from one to five yyears. That is, let PDt den notes the
model designeed to predict bank
b defaults t years from noow, conditionaal on the given banks surviviing to year t-1.. That is,
for years t =1,,..., 5, PDt iss the conditionnal logistic regression model where
1
PDt   n

(7)
   t ,0    t , i xt , i 
 
1 e i1

Then, for eachh bank j, the prrobability of default

d in year t assuming surrvival to year tt-1 is given by

34
Applied Economics and Finance Vol. 2, No. 2; 2015

1
PDt , j  (8)
 n

   t ,0    t ,i xt ,i , j 
 
1 e i1

Note that because I fit a separate model for each year, the variables selected and the coefficients  j,t will, in general,

be different for each year. Let, CPDt , j be the cumulative probability of default for bank j from time t=0 to t years.

Then, the cumulative probabilities for bank j over horizons from t=1 to T years can be determined from their annual PDs
as:
CPD1, j  P1, j
CPD2, j  CPD1, j  (1  CPDi ,1 )  P2, j
(9)
....
CPDT , j  CPDT 1, j  (1  CPDT 1, j )  PT , j
The procedure for calculating marginal PDs beyond five years is illustrated in Figure 6. First, I construct a map between
one-year PDs and Standard & Poor’s rating categories. This is made possible using a map that I derived between
average probabilities of default for commercial and industrial firms from HPD model (Sobehart and Keenan, 2003) and
their corresponding agency ratings.7 For example, the left panel of Figure 6 illustrates a mapping between one-year
PDs from the HPD model to rating categories calibrated using data of all U.S. banks between 1982 and 2012. Using this
map, I can assign an implied rating to each bank that corresponds to its current one-year PD from the logistic regression
model. Then, for a given bank, I combine its term structure of cumulative default rates from one to five years with the
marginal annual default rates reported by Moody’s from its imputed credit rating from six to thirty years. That is, I
assume each bank’s conditional PD beyond five years follows the long-term historical values for its implied rating
category. A resulting set of stylized bank annual cumulative default rates by implied whole letter rating categories
appear in the right panels of Figure 6. The top panel shows cumulative default rates on a linear PD scale, whereas the
lower plot shows those same data in logarithmic PD units. Notice that, as expected, average cumulative default rates for
any given tenor increase with decreasing rating categories.

7
That rating map is constructed using PDs from the HPD model for non-bank corporate firms. Then firms are ranked
with respect to their model PDs and assigned to rating categories that replicate the number of firms in each rating
category in the sample. Finally, implied ratings for U.S. banks are assigned based on their inclusion within PD
boundaries determined for each rating category.

35
Applied Economics and Finance Vol. 2, No. 2; 2015

Figure 6. Left: Mapping From One-Year PD to Imputed Risk Category; Right: Term Structure of Average Cumulative
Bank PD For Each Implied-Rating Category on Linear (Top) and Logarithmic (Bottom) Scales
3. Walk-Forward Backtesting
I back-tested the model by constructing an annual series of models of the bank models using all available US bank data
from 1992 to 2012. The number of non-defaulting banks and defaulting banks in the sample by year is given by the
green bars (left axis) and red bars (right axis) in Figure 7. Notice that there were roughly 14,000 banks in the sample in
1993, but that number declined to around 7,000 by 2012. Also, there are three apparent waves of defaults: one in the
early 90s, a small one around the year 2000, and a surge of bank failures during the recent financial crisis.

Figure 7. Number of Non-Defaulting Banks (Green Bars, Left Axis) and Defaulting Banks (Red Bards, Right Axis)
Banks by Year in the Development Dataset
In order to determine out-of-sample performance of the model, I used a walk-forward procedure as illustrated in Figure
8 for the one-year model. The test set is sufficiently large, with a total of 499 defaulters out of 11,114 distinct banks, to
provide a strong test of model performance. Because the model needs a minimum number of years of data for
development, data from the years 1992 through 1999 were used to construct the first annual model (select variables and
calibrate the weights) for each horizon for one to five years. The one-year model for 1999 was then used to score all
non-defaulting banks at the beginning of 2000 and its ability to predict defaults in 2000 was determined. Models for
year two through five used only banks that had survived to the model year to score for prediction. Thus for the two-year
model, firms surviving until 2001 were scored with its 1999 model, and so forth for the longer horizons. To generate the
set of models for year 2000 (i.e., used to predict defaults in 2001 to 2005 for one- to five-year models), I added the data
from year 2000 to the set from 1992 to 1999. Variables were selected and coefficients determined and the model was
tested on the corresponding test sample for the given horizon. That procedure was repeated annually until 2012. Of
course, from models at horizons longer than one year, testing could only be done to year 2012 minus the horizon year. I
adopted the walk forward procedure because it most realistically estimates the performance of the model as it will be
deployed in practice.
Figure 8. Illustration of the Walk-Forward Development and Testing Procedure for the One-Year Models: A New Model
is Developed Each Year from 1999 to 2011 Using Data From All Previous Years and Tested on Defaulted and
Non-Defaulted Bank in Each Subsequent Year from 2000 to 2012. For Models with Two- to Five-Year Horizons, Test
Samples Consisted of Firms Surviving Until Year X+2 to X+5, Respectively.

To evaluate model performance at separating banks that will default from non-defaulters, I generated Cumulative

36
Applied Economics and Finance Vol. 2, No. 2; 2015

Accuracy Profile (CAP) Curves for the one- to five-year model horizons. The cumulative resulting CAP curves for test
years 1999-2012 are displayed in Figure 9. For example, to generate the one year-curve (blue line), I first rank all banks
over the entire 13-year test period from highest to lowest by their one-year PDs from the models. Then, for successive
intervals in the ranked population I calculate the cumulative fraction of defaulting banks contained within that interval.
The interpretation of CAP curves is straightforward; for any criterion, the fraction of defaulters caught above the
population percentile is measures the discriminatory power of the model. For example, the CAP curve for the one-year
model at the 10% population criterion caught 94% of the banks that defaulted within the following year over the period
from 1999-2012. The higher and steeper the CAP curve over the diagonal chance line, the better the model is at
discriminating defaulters from non-defaulters. The table at the right in Figure 9 displays values of the CAP curves for
each of the model horizons for various values of the population cut-off. The left-most values in the table show that the
10% of banks ranked riskiest by the one- to five-year models capture 94%, 80%, 68%, 55%, and 40% of the defaulting
banks, respectively. Not surprisingly, those data reveal that the power of the models decline as the horizon extends
beyond one year, but even the five-year model is performing well above chance, capturing 40% of the banks that default
in the fifth year after model development and scoring. Finally, it is important to note that even though the models are
only regenerated on an annual basis, the financial data from the banks is available to update bank default scores on a
quarterly basis and that is how the model will be used in practice.

Figure 9. Left: CAP Curves for Predictions of Bank Defaults for One- to Five-Year Models using Walk-Forward Testing
from 1999 Through 2012; Right: Values of the One- to Five-Year CAP Curves at Critical Thresholds, with
Corresponding Values from the Chance Line Also Shown
From a risk management perspective, the most relevant horizon for prediction is at one year. Thus, if a bank survives for
that one year, the next year’s model can be used to assess its subsequent risk. Still, there are applications for which
multi-year estimates of losses and portfolio relative value are of interest. These include buy-and-hold portfolios of bank
obligations, such as structured products. For example, if one holds a portfolio of bank TRUPS (trust preferred securities)
with five years of remaining maturity, they may wish to estimate-five year portfolio losses. For this type of application,
it is important that the absolute PD levels be accurate. The CAP curves, because they rank PDs, assess only the relative
accuracy of the models.8 Indeed, the models do specify absolute PD levels and I can assess their accuracy using the
reliability plots in Figure 10. To construct the plots in Figure 10 I separated all banks into bins by 5% PD increments,
and plot each bin’s average predicted PDs on the horizontal axis and the realized rate of defaults on the vertical axis.
The interpretation of reliability plots is as follows. For example, the one-year plot includes the point (27% predicted, 31%
obtained), which means for all the banks assigned one-year PDs between 25% and 30%, 31% of them actually defaulted
within the following year. A perfect model would have all points falling on the diagonal line for which predicted PD and
realized default rates match exactly. Error bars at two standard deviations for the realized default rates are also shown in
each plot.

8
For example, if one multiplies all PDs by 10 the CAP curves will not change, but the absolute PD levels implied by
the models will be too large.

37
Applied Economics and Finance Vol. 2, No. 2; 2015

Figure 10. Plots of Predicted and Obtained Bank Annual Default Rates from Models for One to Five Years Out. Error
Bars are Two Standard Deviation Bands
The plots in Figure 10 indicate that the default probabilities generated by the model are reasonably accurate at
predicting default rates for banks over multi-year horizons. With respect to the two standard deviation bars, most data
predictions do not differ significantly from the diagonal “perfect model” line. However, a notable exception is that the
bank model typically underestimates the default rates for the second- and third-highest bins (i.e., the high default
60%-70% bins). Further analysis revealed that the model under-predicted the sudden surge of defaults during the
financial crisis of 2008 and 2009. Consider the left panel Figure 11 which displays the historical annual high yield
corporate default rates (left axis) and U.S. bank default rates (right axis) from 1993 through 2012. Notice that the high
yield default rates varied substantially over the period, with high rates early in the century. The banks had been
relatively safe before 2008, with an average annual default rate of only 0.06% and even the maximum during that period
is only 0.34%. The right panel of Figure 11 plots average predicted and realized annual default rates from the one-year
bank model. The bank default models that are constructed annually did not predict well the overall bank default rate in
2008 and 2009, the years of high bank defaults. More generally, the plot reveals that PD levels from the bank model
tend to trail observed annual PD rates by one year. Note that the financial data for U.S. banks are published quarterly by
the U.S. Federal Deposit Insurance Corporation (FDIC). Thus, in practice, I plan to update the model quarterly,
potentially minimizing the lag in accurately predicting annual default rates.

Figure 11. Left: Historical Annual High Yield Default Rate and Bank Default Rates From 1993 to 2012. Right: Model
Predicted Annual Bank Default Rate and Realized Bank Default Rates, 2001–2012 with Two Standard Deviation Error
Bars
3.1 Converting Model Scores to Default Probabilities
I previously showed that values of Ø(z) from the model are highly correlated with default probability (see right panel of
Figure 12). That is, the model appears to perform well at ranking the relative default risk of U.S. banks. Although I
attempted to link outputs of the model (i.e., values of Ø(z)) to actual physical default probabilities, the resulting values
proved less than satisfactory. Accordingly, in this section, I link values of Ø(z) from the bank model to default
probabilities from Hybrid Probability of Default (HPD).
My approach to transforming values of Ø(z), for i  1,..., 5, where i indicates model for a given default year
contingent upon survival to year i  1, is straightforward. For those banks that have PDs from HPD model, I plot HPD
PDs versus values of ln  (z)i from the bank model as shown in the left panel of Figure 12 for the one-year model
(i.e., i  1). Then I fit the points with a second- order polynomial of the form

38
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

Di  ai   bi  lln  (z)i    ci  ln  (z)i2 

PD (10)
(

as shown by the red line in the figure for which i  1 ,and a1  5.5 , b1  0.68 , and c1  0.0 02 . It is
important to note that I immpose monotoonicity on thee function in E Equation 10 tto ensure thatt the conversion from
ln  (z) to PPD does noot change the ordering of bbanks as regarrds their defauult risk. Thus, the transform
mation in
Equation 10 m merely serves to transform model outputss to physical PPDs and does not alter the CAP curves showns in
Figure 2.
Figure 12. Maappings Betweeen Bank Moddel Outputs, ln i (z) annd HPD PDs. LLeft: Best Fit Order 2 Polynnomial of
HPD PDs to ln 1 (z) ; Right: Best Fit Orrder Polynomiaal 2 to ln i (zz) for i  1,..., 5 Models

The right pannel of Figure 121 shows the m mapping from m ln  (z) to PD function ffor the one-yeear model (i.e., i  1)
w the functioons for years two to five. T
using the red line, along with The coefficiennts of ai ,bi annd ci for eacch of the
curves are insset in the grapph. Finally, noote that these m
mapping functtions are only used for the current set off models.
Although the mapping in Fiigure 12 is forr December 5thh, 2013, this m mapping can be updated dailly to reflect ch
hanges in
banks’ PDs froom the HPD model.
m
4. Performan
nce of the Ban
nk Model Relaative to Agenccy Credit Ratiings
In this sectionn I performed a series of studdies to addresss the extent to which my moddels offer any advantages ov ver using
agency ratings for assessingg bank risk. Innvestors are innterested in thee ability to preedict defaults for credits nott already
recognized byy the agencies as risky. Thatt is, “How welll does the moodel do at preddicting defaultss for banks wiith credit
ratings above triple-C, singlle-B, and so onn?”
Of particular iinterest to inveestors is the riiskiness of smaaller and/or loower-rated finaancial firms, paarticularly savings and
loans and bannk holding com mpanies. This is because deebt from thosee financial firm ms is often placed in TRUP Ps (Trust
Preferred seccurities). Becaause returns aand payouts from TRUPs are highly ddependent on defaults and d ratings
downgrades, tthose investorrs are particulaarly interestedd in accurate aassessments off default probaabilities and siignals of
deteriorating credit quality. To test this, I examined thhe relative preedictive poweer of ratings bby Kroll, who are best
known in this space, and myy bank default models for savvings and loanns and bank hoolding companiies.

Figure 13. F
Financial Firms Having Bothh Kroll Ratingss and My Bankk Model Scorees: Left: Bank H
Holding Comp
panies;
Right: Saavings and Loans
Source: Kroll Rating Agency
To test the preedictive powerr of my modell versus agenccy ratings, I firrst determined those financiaal firms that have both
model scores and Kroll ageency ratings. K
Kroll has threee categories off financial firmms: bank holdiing companiess (BHC),

39
Applied Economics and Finance Vol. 2, No. 2; 2015

Savings and Loans (SNL), and Banks (not identified). I obtained Kroll ratings for as many financial institutions as
possible over the period from 2000-2012. The number of banks having Kroll ratings and my model scores appear in
Figure 13, broken out by bank holding companies and savings and loans. Clearly, it appears that Kroll rates significantly
less financial firms than are scored by my bank model. Part of this is because Kroll does not rate new banks within the
first three years of their existence. It is also possible that I received only partial data on Kroll bank ratings. Nevertheless,
as shown in Figure 13, there are roughly 1,000 firms each year having both Kroll ratings and my model scores, and
these are typically the lower rated portion of the financial services firms. Importantly, as shown in Figure 14, there are
at least a reasonable number of defaults for testing model, at least when results are aggregated over the 13-year test
period.9

Figure 4. Number of Defaults for Testing

One- to Five-Year Models

Figure 14. Number of Defaults for Testing One- to Five-Year Models

To test the predictive power of my bank PD model versus agency ratings, I first determined those financial firms that
have both model scores and agency ratings. The left panel of Figure 15 shows the Kroll rating scale, where ratings range
from single-A-plus to default (D). The middle and right panels of Figure 15 display the distributions of bank holding
companies and savings and loans by Kroll credit ratings, respectively, for financial firms having both Kroll credit
ratings and my bank model scores. Notice that Kroll does not rate many banks or savings and loans at A+ or A-. Of
course, there are relatively few very low rated (single-C-plus to single-C-minus) financial institutions as it is very
difficult for low-rated financial institutions to survive for long. Notice also that there are fewer savings and loans than
bank holding companies in the sample.

Figure 15. Kroll Rating Scale for Financial Firms (Left) and Distributions of Bank Holding Companies (Middle) and
Savings and Loans (Right) with Both My Bank Model Scores and Kroll Credit Ratings
Source: Kroll Rating Agency
Default is necessarily a probabilistic event. That is, one is rarely certain that an obligor will default until its actual
occurrence. A useful method for evaluating models’ predictive accuracy of probabilistic events is by constructing
cumulative accuracy profile (CAP) curves such as that shown in the upper panel of Figure 2. Construction of
cumulative accuracy profiles are described in detail in many places (Sobehart and Stein, 2000) and a short description
appears in Appendix A. Briefly, to construct a CAP curve for a bank default model, values of estimated risk are first
ranked from largest to smallest, with information whether each score is associated with a subsequently defaulted bank

9
Notice in Figure 14 that numbers of defaults increase with model horizon. This is because I use overlapping windows
in counting multi-year defaults.

40
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

or a solvent one. Then, I staart from the baanks with the rriskiest scores, say the top 11%. In that topp 1% of banks with the
largest estimaates of defaultt, I calculate the percentage of defaulterrs in that sam mple. For exam mple, if a moddel were
assigning scorres at random,, one would obbserve only 1% % of the defauulters in the topp 1% of the poopulation. Anyy percent
of defaulters aabove 1% wouuld be indicativve of the modeel’s predictive ppower.
I computed CAAP curves for predicting bannk defaults usiing Kroll agenncy ratings andd my bank moddel. Figure 16 displays
CAP curves foor bank holdinng companies ((top) and savinngs and loans (bottom) for ppredicting defaault in year one
e to four.
The curves foor Kroll ratinggs and my bannk model are presented in each graph foor comparison. A useful me easure of
predictive pow
wer from CAP P analysis is thee Area under tthe Curve (AU
UC), which is thhe percentage of the area un
nder each
CAP curve. Thhe AUCs for Kroll
K ratings annd my bank model are inset iin each plot.
Several featurres of the data in Figure 16 aare of interest. First, it is cleear that both m
my bank defauult model and the
t Kroll
agency ratings order banks’ risk at betterr than chance levels, even oout to four yeaars. It is also eevident that prredictive
accuracy for bboth models deecreases as thee year of predicction gets farthher out in time.10 Visual insppection of AUC
Cs in the
top and bottomm panels is suffficient to connclude that eachh model’s perfformance for bbond holding ccompanies and d savings
and loans are similar. This is
i confirmed bby values of AU UCs listed in ttabular form inn Figure 17 foor each model and year
(these same vvalues are insett in each plot in Figure 16).. That is, AUC Cs for BHCs aand SNLs withhin each model vary at
most by 4% annd often only byb 1%.

Figure 16. Coomparison of Cumulative

C Acccuracy Profilee (CAP) Curvees for Bank Hoolding Compannies (Top) and Savings
and Loaans Between K
Kroll Agency RRatings and Myy Bank Model from 2000-20012
Source: Krroll Rating Aggency

Figure 17. A
Areas Under thee CAP Curves for Kroll Agenncy Ratings annd My Bank M
Model for Bankk Holding Com
mpanies
and Savvings and Loanns
Source: Kroll Rating Agency
Figure 16 andd Figure 17 alsso allow compaarison of perfoormance betweeen Kroll ratinngs and my bannk model at prredicting
bank defaults. Again, visuaal inspection oof the CAP currves in Figuree 16 is sufficieent to concludde that my ban
nk model

10
Recall that for each modeel the predictioon for each yeaar is dependennt on the firm ssurviving up too the year of prrediction.
Thus, when prredicting defauults for year tw
wo, all firms thhat defaulted inn the first year after the date of prediction are
a
excluded from
m the sample.

41
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

performs better than Kroll ratings, particcularly as the time horizon increases. Nootably, for onee-year predictiions, the
models are booth quite good;; AUCs for myy bank model aare 98% and 999% for BHCss and SNLs, reespectively, witth AUCs
for Kroll ratinngs are 96% annd 97%. Still, as shown in thhe last row of Figure 17, myy model edges out Kroll ratin ngs even
with that advanntage tending tto increase witth prediction hhorizon. In fact, for all eight CAP curves in Figure
at one-year, w
16. AUCs for my bank defauult model are ggreater than thhose for Kroll rratings.
Despite the suuccess of my bankb model inn predicting deefaults, both innvestors and trraders have reemarked the prredicting
bank defaults at short horizoons is not diffiicult. That is, tthey claim thaat bank failuress tend to be rappid and fairly obvious.
Some suggestt that deterioraation of banks’’ credit is refleected in high leevels of non-pperforming loanns and loss off investor
confidence as evidenced byy rapid withdraawal of banks’ necessary shhort term fundiing. Furthermoore, agency ratings are
sufficient to capture those aspects
a of bankk performance.. That the Kroll ratings and m my bank modeel have extrem mely high
predictive powwer at the one--year horizon iis consistent w with that view. However, the predictive pow wer of my ban nk model
at longer horizzons relative to
t agency ratinngs suggests thhat my bank m model is addingg value. I evaluuate these issuues in the
remaining secction of this stuudy.
5. Predicting Default for Less
L Risky Ban
nks
One way to asssess the addedd value of bankk default modeels, given that low-rated finaancials have allready been rec cognized
by agencies annd investors asa risky, is to eexclude the risskiest obligors from the anallysis of perform mance. To thaat end, in
separate analyyses, I eliminatted all those obbligors rated bby Kroll’s below single single-C (i.e., singlle-C-minus andd below)
and all those bbanks Kroll raates below singgle-B (single-B B-minus and bbelow).11 I theen computed C CAP curves an nd AUCs
on those sub-ssamples as in Figure
F 16 and Figure 17, resppectively. The resulting CAP P curves and taable of AUCs for
f Bank
Holding Comppanies appear in Figure 18 aand the left tabble in Figure 19, respectivelyy.12 Consider ffirst the CAP curves
c in
Figure 18. CA AP curves exclluding BHCs rrated below siingle-C appearr by horizon inn the top panels and those ex xcluding
BHCs rated bbelow single-B B appear in thee lower panelss. First, notice that only onee firm rated beelow single-C and four
BHCs rated bbelow single-B B default withinn one year. Thhis confirms thhe intuition, sttated above, thhat the predictiing bank
defaults at shoort horizons is not difficult ggiven their low
w agency ratinggs. That is, thee defaults that occur within one-year
of risk scoringg occur almostt exclusively foor banks rated below single-C by the Krolll agency.
Figure 18. Cuumulative Accuuracy Profile (CAP) Curvess for Predictingg Defaults on Bank Holdingg Companies for
f Kroll
Ratings and M
My Bank Defauult Model. Topp: CAP Curvess Excluding Crredits Rated byy Kroll Below
w Single-C; and
d Bottom:
CAP Curves E Excluding Firm
ms Rated Below w Single-B, 20002-2012

11
It is importtant to keep thee distinction inn mind between the Kroll ratting scale in thhe left portion oof Figure 5 and
d the
more familiar scales of Stanndard and Poorr’s and Moodyy’s. As a rule oof thumb, a Krooll rating of sinngle-C is roughly
equivalent to a rating by Staandard and Pooor’s of triple-C
C.
12
Because PD D score can takke any value bbetween 0 and 1, while ratinggs can only takke one of a smaall number of discrete
d
values, my scoores also alloww for finer disccrimination bettween institutioons than ratinggs do. This diffference also ex
xplains
why the CAP curves for PD D model is a steep-function whhile the CAP cuurve for ratinggs is a piecewisse linear function.
This is particuularly evident as
a the default ssample is smalll as for the onne-year horizonn model.

42
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

Figure 19. A
Areas Under thhe CAP Curvess (AUCs) Exclluding Firms R
Rated by Kroll as Single-C-MMinus and Belo
ow and
Single-B-Minuss and Below foor Bank Holdinng Companiess (Left) and Saavings and Loaans (Right)
The pattern oof CAP curvees in Figure 18 is similarr for banks rrated by Krolll at or below w single-C-miinus and
single-B-minuus. First, for all
a horizons greater than onee-year, my bannk model outpperforms Krolll agency rating gs. Also,
performance appears to deecrease as the time horizoon increases from two to four years, bbut then decreases in
performance aare greater forr Kroll ratingss than for my bank model. I note that forr predicting deefaults at the four-year
f
horizon, the CCAP curves indicated that KKroll ratings arre nearly at chhance, whereass my bank model is perform ming well
above chancee. Interestinglyy, performancee of both myy bank PD moodel and Kroll ratings do not appear to o change
significantly bbetween BHC samples that eexclude firms bbelow single-C C or single-B.
The features oof the CAP currves mentionedd above for Baank Holding C Companies are presented quaantitatively as AUCs
A in
the left portioon of Figure 19. That is, exxcept for one-yyear, already ddiscussed abovve, AUCs from m my bank model
m are
greater than thhose for Kroll ratings at all teenors, by 8% tto 14%. The paattern is similaar when excludding either BH
HCs rated
below C or B B. That is, peerformance apppears to drop off for both models as prediction horizzon increases, but that
performance ddrops off moree dramatically for Kroll ratinngs, being closse to chance aat the four-yearr horizon. Also o, AUCs
for cases wheere BHCs bellow B and C are excluded are remarkabbly similar; m model performaances do not drop off
appreciably wwhen the criteriion for inclusioon in the sampple is raised to B and above ffrom C and aboove.
The right porttion of Figure 19 lists similarr measures forr Savings and L
Loans. The patttern of resultss mimics well those
t for
the BHCs in thhe left panel:
 My bank PD modeel outperformss Kroll ratings at all tenors w
when SNLs rateed below C annd B are exclud
ded;
 AUCss tend to decreaase with increaases in year off default predicction, but thosee decreases, w
while slight for my
bank P
PD model, are greater for Krroll agency ratiings;
 Perforrmance of bothh my bank PD model and Krroll's ratings arre similar whenn either SNLs below C and B are
excludded from the analysis.
The CAP currves for SNLs are remarkabbly similar to those for BHCs. The CAP curves for Saavings and Lo oans that
correspond to those presented in for Bankk Holding Com mpanies in Figgure 18 are preesented for commparison in Fiigure 20.
Note that the ppattern of CAP
P curves for SN
NLs in Figure 20 is highly siimilar to those for BHCs in F
Figure 18.

43
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

Savings and Looans for Kroll Ratings

Figure 20. Cuumulative Acccuracy Profile ((CAP) Curves for Predictingg Defaults on S
and My Bankk Default Moddel. Top: CAP Curves Excludding Credits R Rated by Kroll Below Single--C; and Bottomm: CAP
Curves Exccluding Firms RRated Below S Single-B, 20022-2012
6. Analysis off Variable Con
ntributions
From the anallysis of the CA AP curves andd AUCs abovee, it is clear thhat my bank PD model conttinues to perform well,
even as the yeear of default prediction mooves out. One of the reasonss for this is thaat the bank PD D model uses different
sets of variablles to predict default
d at eachh successive hoorizon. Accorddingly, in this section, I detaail the contribu
utions of
the various caandidate variabbles to the moddels, both overr the time fram
me of testing frrom 2000-20122 and for tenorrs of one
to five years.
As described iin detail previoously, for eachh test year in thhe walk-forwarrd model deveelopment proceedure from 20000-2012,
I built modelss to predict deefaults for one--, two-, three-, four- and fivve-year horizonns. To construuct each modell in each
year, I began by selecting first
f the variabble having thee largest Bayessian Informatiion Criterion ((BIC) when prredicting
default alone.13 Then I iterrated this proccedure, addingg variables onee-by-one, untill no further im mprovement inn overall

13
The Bayesiaan Information Criterion
C was deeveloped by Schhwartz (1978).
Appendix A: C
Cumulative Acccuracy Profile ((CAP) Curves aand Receiver O
Operating Characteristics (RO
OCs)
The cumulativee accuracy profille (CAP) curve and its close relative, the receivver operating chaaracteristic (RO
OC)
Appendix A: C
Cumulative Acccuracy Profile ((CAP) Curves aand Receiver O
Operating Characteristics (RO
OCs)
The cumulativee accuracy profile (CAP) curve and its close reelative, the receiiver operating chharacteristic (RO OC) are two-dim mensional
plots where thee variable of inteerest, say the likkelihood that thhe bank defaulteed in the followiing year is plottted against the predictive
p
variable, the rannked magnitude of the bank’s riisk of default as estimated by a given model.
Figure 29: Top: CAP Curve witth Points to Illusstrate Various Feeatures; Bottom
m: Descriptions oof Points on the CAP curve

44
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

The CAP curvee at the top of Figgure 29 illustrattes some importaant features:
 The nnegative diagonaal shown by dashhes is the “channce line” where tthe likelihood off detecting a subbsequent defaultt is
unrelaated to the risk rating
r from the ggiven model; thee “hit” and “falsse positive” ratees are the same;
 The ““CAP curve”, the solid curved liine in Figure 277, the plot of “hitts” versus “falsee positives” as oone goes through
h the
entiree set of banks rannked by their prredicted credit riisk from largest to smallest; andd
 The ddashed area undeer the CAP curvve, called the “arrea under the cuurve” or AUC, iss an important m
measure of perforrmance,
whose value is the arrea under the CA
AP curve.
The bottom porrtion of Figure 292 provides desscriptions of the points labeled in the CAP curvve. For examplee, the “star” in the plot at
(hit, false alarm
m) = (0.0, 1.0) is
i indicative off a perfect prediictor; all target cases are rankeed above the firrst “false positiv ve” in the
sample. That is all defaulted baanks larger risk rratings than thosse that did not ddefault. A point aalong the chance line, “E,” is allso shown.
The point “A” is said to plot at
a a “conservativve” criterion, ass the false positiive rate is very low. Points “B”” and “C” illustrate more
permissive criteeria than “A.” Finally,
F the poinnt “D” is on the ROC curve forr a given level oof discriminationn. Note that poiints A, B,
and C could inttersect the same ROC, which woould indicate supperior predictabbility to that show wn through poinnt “D.”
Appendix B: D
Description of Candidate
C Variaables
For clarificationn, I present moree detailed descriiptions of some of the candidatee variables.
ROA: Net incoome after taxes and
a extraordinarry items (annualiized) as a percennt of average tottal assets.
Non-Current L Loans / Loan LossL Allowance:: Non-current looans are defined as assets past duue 90 days or mmore, plus assets placed in
nonaccrual statuus. This quantitty is divided byy loan loss allow
wance. Loan losss allowance is the amount eacch bank must maintain in
reserve for loann and lease lossses to absorb estimated credit llosses associatedd with its loan aand lease portfoolio (which also
o includes
off-balance- sheeet credit instrum
ments).
Earning Assetss / Assets: Thiss is the ratio of aall loans and otther investmentss that earn intereest or dividend income to the sum
s of all
assets owned bby the institutioon including caash, loans, secuurities, bank prremises, and othher assets. Thiss total does no ot include
off-balance-sheeet accounts.
Net Interest M
Margin: Total intterest income lesss total interest eexpense (annuallized) as a perceent of average eaarning assets.
Yield on Earniing Assets: Totaal interest incom
me (annualized) aas a percent of aaverage earning assets.
Net Operating Income to Assets: Net operatinng income (annuualized) as a perrcent of averagee assets.

45
Applied Economics and Finance Vol. 2, No. 2; 2015

BIC occurred. A list of the candidate variable set used for model construction appears in Figure 21 and a more detailed
description of each candidate variable appears in Appendix B.

Figure 21. Candidate Variables for My Bank Default Models for One- to Five-Year Horizon

Figure 22. Left: Potential Model Variables and Their Order of Selection for the One-Year Model for 2000 to 2012; Right: Summary
of Variable Selection, Displaying Probability of Selection and Average Order if Selected for the One-Year Model

Consider first variables selected for the one-year model. The left panel of Figure 22 lists the variables selected each year
from 2000 to 2012 for the one-year model using my walk-forward development and testing procedure. The number
associated with each variable is the order in which the variable was chosen based on its BIC. Those variables having no
value associated with them in a given year were not selected for that year’s model. The table shows that the most
important variable for predicting one-year default is firms’ return on equity (ROE). The ROE was the first variable
chosen for one-year models in every year. The ratio of liabilities to assets is also important for short-term default
predictions, being chosen second for all years except 2010. Contributions from other variables are less consistent over
time. For example, prior to 2009, earning assets to total assets is the third variable selected for all models, but was not
even selected in 2010 or 2011, being selected last in 2012. Conversely, the ratio of non-current loans to loans was not
selected at all in 2000, gradually increasing in its importance, such that from 2009 onward, it was either the second or
third most important predictor of default. Other variables having contributions to one-year default prediction are the
yield on earning assets over the period from 2004 to 2009, and in recent years, the return on assets (ROA), the total
assets, and the ratio of non-current loans to allowance for loan losses. Finally, notice that there is a tendency for the
number of important variables to increase over the testing period, with early models having only four or five variables,
expanding to seven variables by 2011 (see last row of Figure 22).
The right panel of Figure 22 summarizes the consistency of variable contributions to the one-year models over time,
both in terms of what percentage of the 13 yearly models each variable was included, but also its average place in the
hierarchy of contributions if selected. As mentioned above, ROE was selected as the first variable using the BIC 100%

46
Applied Economics and Finance Vol. 2, No. 2; 2015

of the time, with the ratio of Liabilities to Assets, also chosen in all models, but having an average in the order of 2.1
owing to its third position in the 2010 model. The percentage of non-current loans and net operating income to assets
are in 92% of the annual models at roughly fourth and fifth rank, whereas the fraction of earning assets is in 85% of the
models, but when included has an average rank of 3.5. After those variables, contributions drop off rapidly, with the
yield on earning assets in 46% of the models, but only at a rank of 5.8, followed by ROA and total assets at 28%.
Finally, non-current loans to loan loss allowance is in 15% of the annual one-year models at a rank of fifth. Variables
never included in any of the annual one-year models are: net loans to bank equity capital, annual default rate, ratio of
assets 90 days past due to those 30-60 days past due, net interest income to earning assets, and net interest margin.

Figure 23. Summary of Variable Selection, Displaying Probability of Selection and Average Order if Selected for
Annual Two-, Three-, Four- and Five-Year Bank Default Models
Figure 23 presents summaries of the consistency of variable contributions to the two-, three-, four- and five-year models
over time, analogous to that shown for the one-year model at the right in Figure 22. (The lists of variables selected and
their orders for the two- to five-year models analogous to that shown for the one-year model at the left in Figure 22
appear in Figure 24.) The figures reveal shifts in the importance of various predictive variables over time. For example,
the ROE, most important for the one-year model, becomes successively unimportant for predicting default in later years,
not even being included on any of the annual four- or five-year models. Liabilities to assets, also important at one year,
declines immediately to around 30% at two-years and remains at about that frequency, but never of greater importance
than a rank of third. Conversely, net loans to bank equity capital, not included in any one-year models, is in every model
at two and three years, maintaining its contribution, albeit in lesser amounts, out to five years. Meanwhile, the yield on
earning assets, only marginally important at one year, becomes more important at longer horizons, being one of the
most important at three to four years.

47
Applied Economics and Finance Vol. 2, No. 2; 2015

Figure 24. Left: Potential Model Variables and Their Order of Selection for the Two Through Five-Year Models for
2000 to 2012
A more detailed summary of the changes in variables as the annual prediction horizon increases from one to five years
appears in Figure 25. The most important variables for each annual prediction horizon are listed followed by a
description of the changes that occurred from the previous years’ model. For example, in predicting default from one to
two years, ROE has become less important and the ratio of liabilities to assets is no longer in the model, whereas the
fraction of non-current loans has become important along with the ratio of net loans to bank equity capital. In moving
from the two- to three-year horizon, ROE is no longer in the model, whereas net loans to bank equity capital, which
entered the model in at two years, is now most important. When predicting default between three and four years, the
value of non-current loans is no longer in the model, whereas the current yield on earning assets has become most
important. Finally, at five years, the current yield on earning assets remains most important along with net interest
margin, with the fraction of current loans and operating income to assets no longer included. In general, quality of assets
is most important in near-term default predictions, giving way to operating income and the yield on earning assets as the
important determinants of default at longer horizons. It is important to remember that our models are for marginal
annual defaults. Thus, variables important for modeling defaults in early years remain important for cumulative default
prediction, just not for predicting marginal defaults pending survival to later years.

Figure 25. Description and Comparison of Important Variables for Bank Defaults at One- to Five-Year Horizons
A quantitative analysis of the variable contributions over time is presented in Figure 26. The left table in the figure
shows the likelihood of each variable being selected for the default models by horizon in each year over the period from
2000 to 2012. The table at the right displays the average order of selection if the variable was included in the model at
the listed horizons. Consider first, the probabilities of variable selection. Those probabilities have been color coded for
convenience, with variables included in 76% to 100% of the annual models for a given horizon coded in red, those
included between 11% and 75% in green, and those in 10% or less in blue. Although ROE and the ratio of liabilities to

48
Applied Economics and Finance Vol. 2, No. 2; 2015

assets are included in all one-year models, their contributions drop off rapidly at two to five years. Also, performance on
earning assets (earning assets to assets) is important at one- and two-year horizons, but is not included in three- and
four-year models, with only moderate contributions to five-year models. The color coding in Figure 26 helps to reveal
some features not easily distinguished in the data. For example, the most consistently important variable is the
percentage of non-current loans, included in all annual models from one to three years and dropping off to 69% at four
years and 15% at five years. Also, asset size, while not in all annual models is in 15% and 46% of models in all years.
For two- to four-year horizons, the yield on earning assets and net loans to bank equity capital become important, while
having less influence in one- and five-year models. Notably, the five-year models appear to have the most diversity of
variable contributions with no model in more than 54% of annual models. Finally, net interest margin appears relatively
unimportant, except at the five-year horizon, with the ratio of non-current loans to loan loss allowance only included in
a small fraction of models at one- and three-year horizons.

Figure 26. Probabilities of Variable Selection by Model Horizon (Left) and Average Order of Inclusion If Selected at
Given Horizon (Right) For Annual Models Over the Period from 2000-2012
The average order of selection of each variable for each model horizon appears at the right in Figure 26. Variables
selected early in the process (averaging from 1 to 2.5) are coded in red, those in the middle set, averaging between 2.5
and 5.0 are in green, and those selected above fifth on average in blue. The table shows the importance of ROE in early
year models; ROE was selected first in all one-year models from 2000 to 2012, with non-current liabilities second in all
but one year. Again, for mid-year horizons, net operating income becomes selected in most models, averaging between
1.1 and 2.4 in selection order. Although the annual default rate is never included in more than 31% of models at any
horizon, when selected at four- and five-year horizons, its average order is second. The table also confirms the relative
unimportance of net interest income to earning assets and net interest margin.
The data at the right in Figure 26 seem to suggest that a variable's probability of being selected is related to the order in
which is it selected. This is confirmed, at least in general, by Figure 27, which demonstrates that the average order of
selection (first to last) increases with the probability of selection. That is, frequently selected variables tend to be
selected earliest and vice versa. Examples of frequently selected variables with early selection include ROE, the ratio of
liabilities to assets, and percentage of non-current loans. Conversely, rarely selected variables such as net interest
income to earning assets and net interest margin are typically near the last ones to be selected in the few models in
which they are present.

Figure 27. Relationship between Variables’ Probability of Selection and Order of Importance

49
Applied Economics and Finance Vol. 2, No. 2; 2015

I conclude my analysis of variable contributions with an analysis of the consistency of the signs of the variables input
into each model. That is, I measure the extent to which a variable, if selected maintains the same sign in different years
and tenors. For example, variables having positive signs are indicative of higher default risk, whereas negative inputs
signal lower risk of default. Figure 28 shows for each variable the likelihood that it is positive if selected. Again, color
coding is used to highlight variables; variables having 67% or greater likelihood of being positive highlighted in red (for
higher risk), those between 0% and 33% positive in green (for lower risk), and gray for those in between. If a variable is
not included in a model (e.g., for ROE in year four and year five models) the corresponding cell is blank. Clearly,
almost all variables, when selected for a model in a given year are of the same sign. That is, nearly all cells with values
are either 0% or 100%. Exceptions to this include assets at the five-year horizon and net operating income to assets at
one year. Although some variables are of the same sign when included in models at all horizons (e.g., ROE, annual
default rate, and net interest income to earning assets), most variables change sign when included at different horizons.
However, excluding their inputs to the five-year model, all variables except percent of non-current loans are of the same
sign if included in models at shorter horizons.

Figure 28. Analysis of Variable Consistency by Model Tenor: Left: Probabilities of Variables Having Positive Signs
(Higher Default Risk; Red) If Selected and Right: Average Values of Variable Contributions If Selected
A clue to the changes in signs of variables at five years is the large average coefficients, both positive and negative
assigned to variables in the five-year models. Many of those values (e.g., fraction of earning assets, assets 90 days past
due, net interest margin, yield on earning assets are four to eight times the average values of variables in models at
shorter tenors. The results in Figure 26 show that the consistency of variables selected for the models decrease with
tenor and these instabilities likely underlie the large values selected for input variables when chosen for fifth-year
default predictions.
7. Conclusion
In this study, I develop a dynamic measure to overcome limitations of the Merton-type structural models in predicting
default probabilities for financial firms. I build and test adaptive statistical models to estimate default probabilities for
U.S. banks. As described in detail, the models are logistic regression whose input variables are selected and calibrated
based on their past effectiveness at predicting bank failures. Selection of variables in the model and their weights were
updated yearly using a “walk-forward” procedure.
I presented a detailed analysis of the contributions of financial variables to model outputs by year (2000-2012) and
tenor (1-5 years ahead). For a given prediction horizon, I find great consistency among variables selected for each
annual model over the period from 2000 to 2012. For predictions of marginal defaults in years one and two, return on
equity and percentage of non-performing loans are major determinants of default. Those variables give way in
importance at intermediate tenors (i.e., three- to four-years) to current yield on earning assets and net loans to bank
equity capital. Finally, at the five-year horizon, yield on earning assets, asset size, and net interest margin become most
important predictors of default.
I also analyzed the consistency of the order of variable selection as well as the signs of the variables as they reflect
increases and decreases in credit quality. I find a rough, but positive, relationship between the probability of a variable
being selected in each annual model and its average importance in the predictive selection hierarchy. Also, I find that
variables, when selected for a given predictive tenor, are nearly always of the same sign across the years of annual
model development. However, I do find that many variables change sign as the year of marginal default prediction

50
Applied Econom
mics and Financce Vol. 2, No
o. 2; 2015

changes, but tthis mostly occcurs for a singlle year; predictting defaults bbetween years ffour and five.
I further derivved estimates of physical deefault probabillities from riskk scores from the bank PD m model. These PDs can
then be used tto estimate exppected losses ffrom default oon bank portfolios. Because most models ooffer little ben nefit over
agency ratingss for already loow-rated crediits, I examinedd the performannce of the bannk PD model reelative to Krolll agency
ratings for creedits rated abovve single-B-m
minus and for thhose rated abovve single-C-m
minus. I find thaat the predictiv
ve power
of agency ratiings drops off dramatically aas credit qualitty of the scoriing sample inccreases, with m
much less deterioration
in default preddiction using thhe bank PD mmodel.
The model prredicts defaults at annual hoorizons from oone to five yeears. Performannce of the moodels at discrim minating
between defaaults and nonn-defaults is eevaluated for horizons of one to five yyears using a sequence off annual
walk-forward out-of-samplee tests from 19992 to 2012. I aalso measure tthe ability of thhe models to ppredict absolute default
rates from onee to five yearss and, except ffor underestim
mating the highh bank default rates during tthe financial crisis, the
models perforrmed well at estimating thhe annual bannk default ratees. In generall, the models perform favo orably at
predicting deffaults, with a 97%
9 accuracy ratio (AR) at one year priorr to default, annd decreasing, but still above
e-chance
predictive powwer out to fivee years. The moodels are desiggned to be upddated on an annnual basis, butt updated finan
ncials for
inputs to the m
model are avaailable from thhe FDIC on a qquarterly basiss. Not only doo my results prrovide evidenc ce of the
advantage of tthe adaptive sttatistical modeeling approachh over agency ratings, but allso they providde insight to th
he short-
and long-term
m determinants of bank failurre.
References
Altman, E. (11968). Financiial Ratios, Disscriminant Anaalysis, and thee Prediction oof Corporate B
Bankruptcy, Jo
ournal of
Finance, 23, 589-609. http://dx.doi.o
h org/10.1111/j.1540-6261.1968.tb00843.x
Beaver, W. H H. (1966). Fiinancial Ratioos as Predictoors of Failuree, Journal off Accounting Research, 4, 71-111.
http://dx..doi.org/ 10.23307/2490171
Department oof the Treasuryy, Office of thhe Comptroller of the Curreency (2012). 112 CFR Parts 1, 5, 16, 28, and 160
[Docket ID OCC–20112–0005] RIN N 1557–AD366, Alternativess to the Use of External C Credit Ratings in the
Regulatioons of the OCCC, Federal Reggister, 77(114). http://www.ggpo.gov/fdsys/ggranule/FR-20012-06-13/2012-14169
Eom, Y., Hellwege, J., & Huang,
H J. (20003). Structuraal Models of C
Corporate Bonnd Pricing: A
An Empirical Analysis,
A
Forthcomming: Review of
o Financial Sttudies.
Huang, J., & Huang, M. (22000). How MMuch of the Coorporate-Treassury Yield Sprread is Due too Credit Risk?
? A New
Calibration Approach, Working Papeer, Cornell University.
Kealhofer, S S. (1999). Credit Riskk and Risk Managemennt, AIMR Conference Proceedings, 80-94.
http://dx..doi.org/10.41772/2151-6219..1000126
Moody’s Invvestors Service (2010). Corporate Default and R
Recovery Raates, 1920-20009. http://dx
x.doi.org/
10.3905/j
/jfi.2011.21.2.0015
Schwarz, G. E
E. (1978). Estim
mating the Dim
mension of a M
Model, Annals of Statistics, 66(2), 461–464.
Sobehart, J., & Stein, R. (20000). Moody’s Public Firm R
Risk Model: A Hybrid Approoach to Modeliing Short Term
m Default
Risk, Mooody’s Investorrs Service, Gloobal Credit Reesearch.
Sobehart, J., & Keenan, S. (2002).
( Hybridd Contingent C
Claims Models: A Practical A
Approach to Moodeling Default Risk.
Sobehart, J., & Keenan, S. (2003). Hybriid Probability of Default M
Models: A Practtical Approachh to Modeling
g Default
Risk, Quuantitative Creddit Analyst, 3, 5-29.
Vasicek, O. (11984). Credit Valuation,
V KMV
V White Paperr, KMV LLC.

This work is licensed under a Creative Coommons Attribution 3.0 Licennse.

51