Scorecard Formula Guide
Scorecard Formula Guide
Scorecard Formula Guide
Formula Guide
STATISTICA Scorecard
STATISTICA Scorecard is a comprehensive tool dedicated for developing, evaluating, and monitoring
scorecard models. For more information see TUTORIAL Developing Scorecards Using STATISTICA
Scorecard [4]. STATISTICA Scorecard is an add-in for STATISTICA Data Miner and on the computation
level is based on native STATISTICA algorithms such as: Logistic regression, Decision trees (CART and
CHAID), Factor analysis, Random forest and Cox regression. The document contains formulas and
algorithms that are beyond of the scope of the implemented natively in STATISTICA.
PAGE 1 OF 32
Contents
Feature selection ........................................................................................................................................ 5
Feature selection Cramers V ............................................................................................................... 6
Notation .................................................................................................................................................. 6
Computation Details ............................................................................................................................... 6
Feature selection (IV) Information Value ............................................................................................ 7
Notation .................................................................................................................................................. 7
Computation Details ............................................................................................................................... 7
Feature selection Gini .......................................................................................................................... 8
Computation Details ............................................................................................................................... 8
Interaction and Rules .................................................................................................................................. 9
Interaction and Rules Bad rate (and Good rate).................................................................................. 9
Notation ................................................................................................................................................ 10
Interaction and Rules Lift (bad) and Lift (good) ................................................................................. 11
Notation ................................................................................................................................................ 11
Attribute building ...................................................................................................................................... 12
Attribute building (WoE) Weight of Evidence ................................................................................... 12
Notation ................................................................................................................................................ 12
Computation Details ............................................................................................................................. 12
Scorecard preparation .............................................................................................................................. 13
Scorecard preparation Scaling Factor ............................................................................................. 13
Notation ................................................................................................................................................ 13
Scorecard preparation Scaling Offset ............................................................................................. 14
Notation ................................................................................................................................................ 14
Scorecard preparation Scaling Calculating score (WoE coding) ...................................................... 15
Notation ................................................................................................................................................ 15
Computation Details ............................................................................................................................. 15
Scorecard preparation Scaling Calculating score (Dummy coding) ................................................. 16
Notation ................................................................................................................................................ 16
Computation Details ............................................................................................................................. 16
Scorecard preparation Scaling Neutral score ................................................................................. 17
STATISTICA Formula Guide
STATISTICA Scorecard
Copyright 2013 Version 1
PAGE 2 OF 32
Notation ................................................................................................................................................ 17
Scorecard preparation Scaling Intercept adjustment ..................................................................... 18
Notation ................................................................................................................................................ 18
Survival ...................................................................................................................................................... 19
Reject Inference ........................................................................................................................................ 20
Reject Inference - Parceling method .................................................................................................... 20
Model evaluation ...................................................................................................................................... 21
Model evaluation Gini ........................................................................................................................ 21
Notation ................................................................................................................................................ 21
Computation Details ............................................................................................................................. 21
Model evaluation Information Value (IV) .......................................................................................... 21
Model evaluation - Divergence ............................................................................................................. 22
Notation ................................................................................................................................................ 22
Model evaluation Hosmer-Lemeshow ............................................................................................... 23
Notation ................................................................................................................................................ 23
Computation Details ............................................................................................................................. 23
Model evaluation Kolmogorov-Smirnov statistic .............................................................................. 24
Notation ................................................................................................................................................ 24
Computation Details ............................................................................................................................. 24
Comments ............................................................................................................................................. 24
Model evaluation AUC Area Under ROC Curve .............................................................................. 25
Notation ................................................................................................................................................ 25
Model evaluation 2x2 tables measures ............................................................................................. 26
Notation ................................................................................................................................................ 26
Cut-off point selection .............................................................................................................................. 27
Cut-off point selection ROC optimal cut-off point ............................................................................. 27
Notation ................................................................................................................................................ 27
Score cases ................................................................................................................................................ 28
Score cases Adjusting probabilities.................................................................................................... 28
Notation ................................................................................................................................................ 28
Calibration tests ........................................................................................................................................ 29
PAGE 3 OF 32
PAGE 4 OF 32
Feature selection
The Feature Selection module is used to exclude unimportant or redundant variables from the initial
set of characteristics. Select representatives option enable you to identify redundancy among
numerical variables without analyzing the correlation matrix of all variables. This module creates
bundles of commonly correlated characteristics using Factor analysis with principal components
extraction method and optional factor rotation that is implemented as standard STATISTICA procedure.
Bundles of variables are created based on value of factor loadings (correlation between given variable
and particular factor score) User can set the option defining minimal absolute value of loading that
makes given variable representative of particular factor. Number of components is defined based on
eingenvalue or max factors option. If categorical predictors are selected before factor calculation
variables are recoded using WoE (log odds) transformation (described in the Attribute Building
chapter).
In each bundle, variables are highly correlated with the same factor (in other words have high absolute
value of factor loading) and often with each other, so we can easily select only a small number of
bundle representatives. After bundles are identified user can manually or automatically select
representatives of each bundle. In case of automatic selection user can select correlation option that
allows selecting variables with the highest correlation with other variables in given bundle. The other
option is IV criterion (described below).
Variable rankings can be created using three measures of overall predictive power of variables: IV
(Information Value), Cramers V, and the Gini coefficient. Based on these measures, you can identify
the characteristics that have an important impact on credit risk and select them for the next stage of
model development. For more information see TUTORIAL Developing Scorecards Using STATISTICA
Scorecard [4].
PAGE 5 OF 32
V=
2
n min(w 1, k 1)
2
n
Notation
Where:
2
Computation Details
Note: All continuous predictors are categorized (using by default 10 equipotent categories).
Missing data or value marked by user as atypical are considered as separate category.
PAGE 6 OF 32
Notation
Where:
k
gi
column-wise percentage distribution of the total good cases in the ith bin
bi
column-wise percentage distribution of the total bad cases in the ith bin
Computation Details
Note: All continuous predictors are categorized (using by default 10 equipotent categories).
Missing data or value marked by user as atypical are considered as separate category.
PAGE 7 OF 32
Computation Details
Note: All continuous predictors are categorized (using by default 10 equipotent categories).
Missing data or value marked by user as atypical are considered as separate category.
PAGE 8 OF 32
PAGE 9 OF 32
Notation
Where:
nbad
ngood
ntotal
PAGE 10 OF 32
Notation
Where:
BRRule
Bad rate calculated for a subset of cases that meet given rule
BRDataset
GRRule
Good rate calculated for a subset of cases that meet given rule
GRDataset
PAGE 11 OF 32
Attribute building
In the Attribute Building module, risk profiles for every variable can be prepared. Using an automatic
algorithm based on the standard STATISTICA CHAID, C&RT or CHAID on C&RT methods; manual mode;
percentiles or minimum frequency, we can divide variables (otherwise referred to characteristics) into
classes (attributes or bins) containing homogenous risks. Initial attributes can be adjusted manually
to fulfill business and statistical criteria such as profile smoothness or ease of interpretation. There is
also an option to build attributes automatically. To build proper risk profiles, statistical measures of the
predictive power of each attribute (Weight of Evidence (WoE) and IV Information Value) are
calculated.
If automatic creation of attributes is selected program can find optimal bins using CHAID or C&RT
algorithm. In such case tree models are built for each predictor separately (in other words, each model
contains only one predictor). Attributes are created based on terminal nodes prepared by particular
tree. For continuous predictors there is also option CHAID on C&RT which creates initial attributes
based on C&RT algorithm. Terminal nodes created by C&RT are inputs to CHAID method that tries to
merge similar categories into more overall bins. All options of C&RT and CHAID methods are described
in STATISTICA Help (Interactive Trees (C&RT, CHAID)) [8]. More information see : TUTORIAL Developing
Scorecards Using STATISTICA Scorecard [4].
Notation
Where:
g
column-wise percentage distribution of the total bad cases in the analyzed bin
Computation Details
Note: All continuous predictors are categorized (using by default 10 equipotent categories).
If there are atypical values in the variables they are considered as separate bin.
Note: If there are categories without good or bad categories WoE value is not
calculated. Such category should be merged with adjacent category to avoids errors in
calculations.
PAGE 12 OF 32
Scorecard preparation
The final stage of this process is scorecard preparation using a standard STATISTICA logistic regression
algorithm to estimate model parameters. Options of building logistic regression model like estimation
parameters or stepwise parameters are described in STATISTICA Help (Generalized Linear/Nonlinear
(GLZ) Models) [8].
There are also some scaling transformations and adjustment method that allows the user to calculate
scorecard so the points reflect the real (expected) odds in incoming population. More information see :
TUTORIAL Developing Scorecards Using STATISTICA Scorecard [4].
Notation
Where:
pdo
PAGE 13 OF 32
Notation
Where:
Score
scoring value for which you want to receive specific odds of the loan repayment parameter given by the user
Odds
odds of the loan repayment for specific scoring value - parameter given by the
user
Factor
PAGE 14 OF 32
offset
Notation
Where:
logistic regression coefficient for characteristics that owns the given attribute
WoE
factor
offset
Computation Details
Note: After computation is complete the resulting value is rounded to the nearest integer
value.
PAGE 15 OF 32
offset
Notation
Where:
factor
offset
Computation Details
Note: After computation is complete the resulting value is rounded to the nearest integer
value.
PAGE 16 OF 32
Notation
Where:
k
scorei
distri
PAGE 17 OF 32
Notation
Where:
regression
pgood
probability of sampling cases from good strata (or class that is coded in logistic
regression as 1)
pbad
probability of sampling cases from bad strata (or class that is coded in logistic
regression as 0)
PAGE 18 OF 32
Survival
The Survival module is used to build scoring models using the standard STATISTICA Cox Proportional
Hazard Model. We can estimate a scoring model using additional information about the time of
default, or when a debtor stopped paying. Based on this module, we can calculate the probability of
default (scoring) in given time (e.g., after 6 months, 9 months, etc.). Options of input parameters and
output products of Cox Proportional Hazard Model are described in STATISTICA Help (Advanced
Linear/Nonlinear Models - Survival - Regression Models) [8]. For more information see TUTORIAL
Developing Scorecards Using STATISTICA Scorecard [4].
PAGE 19 OF 32
Reject Inference
The Reject inference module allows you to take into consideration cases for which the credit
applications were rejected. Because there is no information about output class (good or bad credit) of
rejected cases, we must add this information using an algorithm. To add information about the output
class, the standard STATISTICA k-nearest neighbors method (from menu Data-Data filtering/RecodingMD Imputation) and parceling method are available. After analysis, a new data set with complete
information is produced.
PAGE 20 OF 32
Model evaluation
The Model Evaluation module is used to evaluate and compare different scorecard models. To assess
models, the comprehensive statistical measures can be selected, each with a full detailed report. More
information see : TUTORIAL Developing Scorecards Using STATISTICA Scorecard [4].
G = 1 (B ( xi ) B ( xi 1 ) ) (G ( xi ) + G ( xi 1 ) ) ; and G ( x 0 ) = B ( x 0 ) = 0
i =1
Notation
Where:
k
G(xi)
B(xi)
Computation Details
Note: There is strict relationship between Gini coefficient as AUC (Area Under ROC Curve)
coefficient. Such relationship can be expressed as G = 2 AUC 1 .
PAGE 21 OF 32
(meanG meanB ) 2
.
0,5 (varG + varB )
Notation
Where:
meanG
meanB
varG
varB
PAGE 22 OF 32
(oi ni i )2
ni i (1 i )
Notation
Where:
k
number of groups
oi
ni
Computation Details
Groups for this test are based on the values of the estimated probabilities. In STATISTICA
Scorecard implementation 10 groups are prepared. Groups have the same number of cases.
First group contains subjects having the smallest estimated probabilities and consistently
the last group contains cases having the largest estimated probabilities.
PAGE 23 OF 32
Notation
Where:
G(x)
B(x)
xj
j=1,,N
Computation Details
KS statistic is a base of formulating statistical test checking if tested distributions differs
significantly. In STATISTICA Scorecard standard KS test is performed based on standard
STATISTICA implementation.
Comments
Very often KS statistic is presented in the graphical form such as on graphs below.
For more information see TUTORIAL Developing Scorecards Using STATISTICA Scorecard [4].
PAGE 24 OF 32
Notation
Where:
G
PAGE 25 OF 32
Observed
Bad
Predicted
Good
Bad
Good
TP
, whereas
TP + FN
TN
. The other measures used in the AUC
TN + FP
TP + TN
TP
TN
SENS
report : ACC =
; PPV =
; NPV =
; LR =
.
TP + TN + FP + FN
TP + FP
TN + FN
1 SPEC
Notation
Where:
TP
FP
FN
TN
SENS
Sensitivity
SPEC
Specificity
ACC
Accuracy
PPV
NPV
LR
PAGE 26 OF 32
FN cost p
Notation
Where:
p
FP cost
cost of situation when good cases that are incorrectly predicted as bad
FN cost
cost of situation when bad cases that are incorrectly predicted as good
PAGE 27 OF 32
Score cases
The Score Cases module is used to score new cases using a selected model saved as an XML script. We
can calculate overall scoring, partial scorings for each variable, and probability of default from the
logistic regression model, adjusted by an a priori probability of default for the whole population
supplied by the user. For more information see TUTORIAL Developing Scorecards Using STATISTICA
Scorecard [4].
Notation
Where:
pi
PAGE 28 OF 32
Calibration tests
The Calibration Tests module allows banks to test whether or not the forecast probability of default
(PD) has been the PD that has actually occurred. The Binomial Distribution and Normal Distribution
tests are included to test as appropriate the rating classes. The Austrian Supervision Criterion (see [5])
can be selected allowing STATISTICA to automatically choose the appropriate distribution test.
Computation Details
Two tests for determining whether a model underestimates rating results or the PD are the
standard STATISTICA Normal Distribution Test and the standard STATISTICA Binomial Test.
When the Austrian Supervision Criterion is checked, STATISTICA automatically selects the
proper test for each rating class. (see [5]). If the sample meets the following criteria, the
Standard Normal Distribution test is appropriate. For example, if you have a maximum PD
value for a class of .10% then your minimum frequency for that class must be greater than
or equal to 9,010 cases to use the Normal Distribution test. If there are less than 9,101
cases, the Binomial Distribution test would be appropriate.
Maximum PD Value
0.10%
9,010
0.25%
3,610
0.50%
1,810
1.00%
910
2.00%
460
3.00%
310
5.00%
190
10.00%
101
20.00%
57
50.00%
37
PAGE 29 OF 32
Population stability
The Population Stability module provides analytical tools for comparing two data sets (e.g., current and
historical data sets) in order to detect any significant changes in characteristic structure or applicant
population. Significant distortion in the current data set may provide a signal to re-estimate model
parameters. This module produces reports of population and characteristic stability with respective
graphs. For more information see TUTORIAL Developing Scorecards Using STATISTICA Scorecard [4].
Population stability
Population stability index measures the magnitude of the population shift between actual
and expected applicants. You can express this index using the following formula:
k
Actuali
Population stability = ( Actuali Expectedi ) ln(
).
Expectedi
i =1
Notation
Where:
k
Actuali
percentage distribution of the total Actual cases in the ith score value or score
range
Expectedi
percentage distribution of the total Expected cases in the ith score value or
score range
PAGE 30 OF 32
Characteristic stability
Characteristic stability index provides the information on shifts of distribution of variables
used for example in the scorecard building process. You can express this index using the
k
Notation
Where:
k
Actuali
Expectedi
scorei
PAGE 31 OF 32
References
[1] Agresti, A. (2002). Categorical data analysis, 2nd ed. Hoboken, NJ: John Wiley & Sons.
[2] Hosmer, D, & Lemeshow, S. (2000). Applied logistic regression, 2nd ed. Hoboken, NJ: John Wiley &
Sons.
[3] Maddala, G. S. (2001) Introduction to Econometrics. 3rd ed. John Wiley & Sons.
[4] Migut, G. Jakubowski, J. and Stout, D. (2013) TUTORIAL Developing Scorecards Using STATISTICA
Scorecard. StatSoft Polska/StatSoft Inc.
[5] Oesterreichishe Nationalbank. (2004). Guidelines on credit risk management: Rating models and
validation. Vienna, Austria: Oesterreichishe Nationalbank.
[6] Siddiqi, N. (2006). Credit Risk Scorecards: Developing and Implementing Intelligent Credit Scoring.
Hoboken, NJ: John Wiley & Sons.
[7] StatSoft, Inc. (2013). STATISTICA (data analysis software system), version 12. www.statsoft.com.
[8] Zweig, M. H., and Campbell, G. Receiver-operating characteristic (ROC) plots: a fundamental
evaluation tool in clinical medicine. Clinical chemistry 39.4 (1993): 561-577.
PAGE 32 OF 32