Understanding

credit risk
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
What is credit risk?
The possibility that someone who has borrowed money will not repay it all

Calculated risk di erence between lending someone money and a government bond

When someone fails to repay a loan, it is said to be in default

The likelihood that someone will default on a loan is the probability of default (PD)

CREDIT RISK MODELING IN PYTHON

What is credit risk?
The possibility that someone who has borrowed money will not repay it all

Calculated risk di erence between lending someone money and a government bond

When someone fails to repay a loan, it is said to be in default

The likelihood that someone will default on a loan is the probability of default (PD)

Payment Payment Date Loan Status

$100 Jun 15 Non-Default
$100 Jul 15 Non-Default
$0 Aug 15 Default

CREDIT RISK MODELING IN PYTHON

Expected loss
The dollar amount the rm loses as a result of loan default

Three primary components:

Probability of Default (PD)

Exposure at Default (EAD)

Loss Given Default (LGD)

Formula for expected loss:

expected_loss = PD * EAD * LGD

CREDIT RISK MODELING IN PYTHON

Types of data used
Two Primary types of data used:

Application data

Behavioral data

Application Behavioral
Interest Rate Employment Length
Grade Historical Default
Amount Income

CREDIT RISK MODELING IN PYTHON

Data columns
Mix of behavioral and application Column Column
Contain columns simulating credit bureau Income Loan grade
data Age Loan amount
Home ownership Interest rate
Employment length Loan status
Loan intent Historical default
Percent Income Credit history length

CREDIT RISK MODELING IN PYTHON

Exploring with cross tables
pd.crosstab(cr_loan['person_home_ownership'], cr_loan['loan_status'],
values=cr_loan['loan_int_rate'], aggfunc='mean').round(2)

CREDIT RISK MODELING IN PYTHON

Exploring with visuals
plt.scatter(cr_loan['person_income'], cr_loan['loan_int_rate'],c='blue', alpha=0.5)
plt.xlabel("Personal Income")
plt.ylabel("Loan Interest Rate")
plt.show()

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Outliers in Credit
Data
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Data processing
Prepared data allows models to train faster

O en positively impacts model performance

CREDIT RISK MODELING IN PYTHON

Outliers and performance
Possible causes of outliers:

Problems with data entry systems (human error)

Issues with data ingestion tools

CREDIT RISK MODELING IN PYTHON

Outliers and performance
Possible causes of outliers:

Problems with data entry systems (human error)

Issues with data ingestion tools

Feature Coe cient With Outliers Coe cient Without Outliers

Interest Rate 0.2 0.01
Employment Length 0.5 0.6
Income 0.6 0.75

CREDIT RISK MODELING IN PYTHON

Detecting outliers with cross tables
Use cross tables with aggregate functions

pd.crosstab(cr_loan['person_home_ownership'], cr_loan['loan_status'],
values=cr_loan['loan_int_rate'], aggfunc='mean').round(2)

CREDIT RISK MODELING IN PYTHON

Detecting outliers visually
Detecting outliers visually

Histograms

Sca er plots

CREDIT RISK MODELING IN PYTHON

Removing outliers
Use the .drop() method within Pandas

indices = cr_loan[cr_loan['person_emp_length'] >= 60].index

cr_loan.drop(indices, inplace=True)

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Risk with missing
data in loan data
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
What is missing data?
NULLs in a row instead of an actual value

An empty string ''

Not an entirely empty row

Can occur in any column in the data

CREDIT RISK MODELING IN PYTHON

Similarities with outliers
Negatively a ect machine learning model performance

May bias models in unanticipated ways

May cause errors for some machine learning models

CREDIT RISK MODELING IN PYTHON

Similarities with outliers
Negatively a ect machine learning model performance

May bias models in unanticipated ways

May cause errors for some machine learning models

Missing Data Type Possible Result

NULL in numeric column Error
NULL in string column Error

CREDIT RISK MODELING IN PYTHON

How to handle missing data
Generally three ways to handle missing data
Replace values where the data is missing

Remove the rows containing missing data

Leave the rows with missing data unchanged

Understanding the data determines the course of action

CREDIT RISK MODELING IN PYTHON

How to handle missing data
Generally three ways to handle missing data
Replace values where the data is missing

Remove the rows containing missing data

Leave the rows with missing data unchanged

Understanding the data determines the course of action

Missing Data Interpretation Action

NULL in loan_status Loan recently approved Remove from prediction data

NULL in person_age Age not recorded or disclosed Replace with median

CREDIT RISK MODELING IN PYTHON

Finding missing data
Null values are easily found by using the isnull() function

Null records can easily be counted with the sum() function

.any() method checks all columns

null_columns = cr_loan.columns[cr_loan.isnull().any()]
cr_loan[null_columns].isnull().sum()

# Total number of null values per column

person_home_ownership 25
person_emp_length 895
loan_intent 25
loan_int_rate 3140
cb_person_default_on_file 15

CREDIT RISK MODELING IN PYTHON

Replacing Missing data
Replace the missing data using methods like .fillna() with aggregate functions and
methods

cr_loan['loan_int_rate'].fillna((cr_loan['loan_int_rate'].mean()), inplace = True)

CREDIT RISK MODELING IN PYTHON

Dropping missing data
Uses indices to identify records the same as with outliers

Remove the records entirely using the .drop() method

indices = cr_loan[cr_loan['person_emp_length'].isnull()].index
cr_loan.drop(indices, inplace=True)

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Logistic regression
for probability of
default
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Probability of default
The likelihood that someone will default on a loan is the probability of default

A probability value between 0 and 1 like 0.86

loan_status of 1 is a default or 0 for non-default

CREDIT RISK MODELING IN PYTHON

Probability of default
The likelihood that someone will default on a loan is the probability of default

A probability value between 0 and 1 like 0.86

loan_status of 1 is a default or 0 for non-default

Probability of Default Interpretation Predicted loan status

0.4 Unlikely to default 0
0.90 Very likely to default 1
0.1 Very unlikely to default 0

CREDIT RISK MODELING IN PYTHON

Predicting probabilities
Probabilities of default as an outcome from machine learning
Learn from data in columns (features)

Classi cation models (default, non-default)

Two most common models:

Logistic regression

Decision tree

CREDIT RISK MODELING IN PYTHON

Logistic regression
Similar to the linear regression, but only produces values between 0 and 1

CREDIT RISK MODELING IN PYTHON

Training a logistic regression
Logistic regression available within the scikit-learn package

from sklearn.linear_model import LogisticRegression

Called as a function with or without parameters

clf_logistic = LogisticRegression(solver='lbfgs')

Uses the method .fit() to train

clf_logistic.fit(training_columns, np.ravel(training_labels))

Training Columns: all of the columns in our data except loan_status

Labels: loan_status (0,1)

CREDIT RISK MODELING IN PYTHON

Training and testing
Entire data set is usually split into two parts

CREDIT RISK MODELING IN PYTHON

Training and testing
Entire data set is usually split into two parts

Data Subset Usage Portion

Train Learn from the data to generate predictions 60%
Test Test learning on new unseen data 40%

CREDIT RISK MODELING IN PYTHON

Creating the training and test sets
Separate the data into training columns and labels

X = cr_loan.drop('loan_status', axis = 1)
y = cr_loan[['loan_status']]

Use train_test_split() function already within sci-kit learn

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4, random_state=123)

test_size : percentage of data for test set

random_state : a random seed value for reproducibility

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Predicting the
probability of
default
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Logistic regression coefficients
# Model Intercept
array([-3.30582292e-10])
# Coefficients for ['loan_int_rate','person_emp_length','person_income']
array([[ 1.28517496e-09, -2.27622202e-09, -2.17211991e-05]])

# Calculating probability of default

int_coef_sum = -3.3e-10 +
(1.29e-09 * loan_int_rate) + (-2.28e-09 * person_emp_length) + (-2.17e-05 * person_income)
prob_default = 1 / (1 + np.exp(-int_coef_sum))
prob_nondefault = 1 - (1 / (1 + np.exp(-int_coef_sum)))

CREDIT RISK MODELING IN PYTHON

Interpreting coefficients
# Intercept
intercept = -1.02
# Coefficient for employment length
person_emp_length_coef = -0.056

For every 1 year increase in person_emp_length , the person is less likely to default

CREDIT RISK MODELING IN PYTHON

Interpreting coefficients
# Intercept
intercept = -1.02
# Coefficient for employment length
person_emp_length_coef = -0.056

For every 1 year increase in person_emp_length , the person is less likely to default

intercept person_emp_length value * coef probability of default

-1.02 10 (10 * -0.06 ) .17

-1.02 11 (11 * -0.06 ) .16

-1.02 12 (12 * -0.06 ) .15

CREDIT RISK MODELING IN PYTHON

Using non-numeric columns
Numeric: loan_int_rate , person_emp_length , person_income

Non-numeric:

cr_loan_clean['loan_intent']

EDUCATION
MEDICAL
VENTURE
PERSONAL
DEBTCONSOLIDATION
HOMEIMPROVEMENT

Will cause errors with machine learning models in Python unless processed

CREDIT RISK MODELING IN PYTHON

One-hot encoding
Represent a string with a number

CREDIT RISK MODELING IN PYTHON

One-hot encoding
Represent a string with a number

0 or 1 in a new column column_VALUE

CREDIT RISK MODELING IN PYTHON

Get dummies
Utilize the get_dummies() within pandas

# Separate the numeric columns

cred_num = cr_loan.select_dtypes(exclude=['object'])
# Separate non-numeric columns
cred_cat = cr_loan.select_dtypes(include=['object'])
# One-hot encode the non-numeric columns only
cred_cat_onehot = pd.get_dummies(cred_cat)
# Union the numeric columns with the one-hot encoded columns
cr_loan = pd.concat([cred_num, cred_cat_onehot], axis=1)

CREDIT RISK MODELING IN PYTHON

Predicting the future, probably
Use the .predict_proba() method within scikit-learn

# Train the model

clf_logistic.fit(X_train, np.ravel(y_train))
# Predict using the model
clf_logistic.predict_proba(X_test)

Creates array of probabilities of default

# Probabilities: [[non-default, default]]

array([[0.55, 0.45]])

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Credit model
performance
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Model accuracy scoring
Calculate accuracy

Use the .score() method from scikit-learn

# Check the accuracy against the test data

clf_logistic1.score(X_test,y_test)

0.81

81% of values for loan_status predicted correctly

CREDIT RISK MODELING IN PYTHON

ROC curve charts
Receiver Operating Characteristic curve
Plots true positive rate (sensitivity) against false positive rate (fall-out)

fallout, sensitivity, thresholds = roc_curve(y_test, prob_default)

plt.plot(fallout, sensitivity, color = 'darkorange')

CREDIT RISK MODELING IN PYTHON

Analyzing ROC charts
Area Under Curve (AUC): area between curve and random prediction

CREDIT RISK MODELING IN PYTHON

Default thresholds
Threshold: at what point a probability is a default

CREDIT RISK MODELING IN PYTHON

Setting the threshold
Relabel loans based on our threshold of 0.5

preds = clf_logistic.predict_proba(X_test)
preds_df = pd.DataFrame(preds[:,1], columns = ['prob_default'])
preds_df['loan_status'] = preds_df['prob_default'].apply(lambda x: 1 if x > 0.5 else 0)

CREDIT RISK MODELING IN PYTHON

Credit classification reports
classification_report() within scikit-learn

from sklearn.metrics import classification_report

classification_report(y_test, preds_df['loan_status'], target_names=target_names)

CREDIT RISK MODELING IN PYTHON

Selecting classification metrics
Select and store speci c components from the classification_report()

Use the precision_recall_fscore_support() function from scikit-learn

from sklearn.metrics import precision_recall_fscore_support

precision_recall_fscore_support(y_test,preds_df['loan_status'])[1][1]

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Model
discrimination and
impact
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Confusion matrices
Shows the number of correct and incorrect predictions for each loan_status

CREDIT RISK MODELING IN PYTHON

Default recall for loan status
Default recall (or sensitivity) is the proportion of true defaults predicted

CREDIT RISK MODELING IN PYTHON

Recall portfolio impact
Classi cation report - Underperforming Logistic Regression model

CREDIT RISK MODELING IN PYTHON

Recall portfolio impact
Classi cation report - Underperforming Logistic Regression model

Number of true defaults: 50,000

Loan Amount Defaults Predicted / Not Predicted Estimated Loss on Defaults

$50 .04 / .96 (50000 x .96) x 50 = $2,400,000

CREDIT RISK MODELING IN PYTHON

Recall, precision, and accuracy
Di cult to maximize all of them because there is a trade-o

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Gradient boosted
trees with XGBoost
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Decision trees
Creates predictions similar to logistic regression

Not structured like a regression

CREDIT RISK MODELING IN PYTHON

Decision trees for loan status
Simple decision tree for predicting loan_status probability of default

CREDIT RISK MODELING IN PYTHON

Decision tree impact

Loan True loan status Pred. Loan Status Loan payo value Selling Value Gain/Loss
1 0 1 $1,500 $250 -$1,250
2 0 1 $1,200 $250 -$950

CREDIT RISK MODELING IN PYTHON

A forest of trees
XGBoost uses many simplistic trees (ensemble)

Each tree will be slightly be er than a coin toss

CREDIT RISK MODELING IN PYTHON

Creating and training trees
Part of the xgboost Python package, called xgb here

Trains with .fit() just like the logistic regression model

# Create a logistic regression model

clf_logistic = LogisticRegression()
# Train the logistic regression
clf_logistic.fit(X_train, np.ravel(y_train))

# Create a gradient boosted tree model

clf_gbt = xgb.XGBClassifier()
# Train the gradient boosted tree
clf_gbt.fit(X_train,np.ravel(y_train))

CREDIT RISK MODELING IN PYTHON

Default predictions with XGBoost
Predicts with both .predict() and .predict_proba()
.predict_proba() produces a value between 0 and 1

.predict() produces a 1 or 0 for loan_status

# Predict probabilities of default

gbt_preds_prob = clf_gbt.predict_proba(X_test)
# Predict loan_status as a 1 or 0
gbt_preds = clf_gbt.predict(X_test)

# gbt_preds_prob
array([[0.059, 0.940], [0.121, 0.989]])
# gbt_preds
array([1, 1, 0...])

CREDIT RISK MODELING IN PYTHON

Hyperparameters of gradient boosted trees
Hyperparameters: model parameters (se ings) that cannot be learned from data

Some common hyperparameters for gradient boosted trees

learning_rate : smaller values make each step more conservative

max_depth : sets how deep each tree can go, larger means more complex

xgb.XGBClassifier(learning_rate = 0.2,
max_depth = 4)

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Column selection for
credit risk
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Choosing specific columns
We've been using all columns for predictions

# Selects a few specific columns

X_multi = cr_loan_prep[['loan_int_rate','person_emp_length']]

# Selects all data except loan_status

X = cr_loan_prep.drop('loan_status', axis = 1)

How you can tell how important each column is

Logistic Regression: column coe cients

Gradient Boosted Trees: ?

CREDIT RISK MODELING IN PYTHON

Column importances
Use the .get_booster() and .get_score() methods
Weight: the number of times the column appears in all trees

# Train the model

clf_gbt.fit(X_train,np.ravel(y_train))
# Print the feature importances
clf_gbt.get_booster().get_score(importance_type = 'weight')

{'person_home_ownership_RENT': 1, 'person_home_ownership_OWN': 2}

CREDIT RISK MODELING IN PYTHON

Column importance interpretation
# Column importances from importance_type = 'weight'
{'person_home_ownership_RENT': 1, 'person_home_ownership_OWN': 2}

CREDIT RISK MODELING IN PYTHON

Plotting column importances
Use the plot_importance() function

xgb.plot_importance(clf_gbt, importance_type = 'weight')

{'person_income': 315, 'loan_int_rate': 195, 'loan_percent_income': 146}

CREDIT RISK MODELING IN PYTHON

Choosing training columns
Column importance is used to sometimes decide which columns to use for training

Di erent sets a ect the performance of the models

Model Model Default

Columns Importances
Accuracy Recall
loan_int_rate, person_emp_length (100, 100) 0.81 0.67
loan_int_rate, person_emp_length,
(98, 70, 5) 0.84 0.52
loan_percent_income

CREDIT RISK MODELING IN PYTHON

F1 scoring for models
Thinking about accuracy and recall for di erent column groups is time consuming

F1 score is a single metric used to look at both accuracy and recall

Shows up as a part of the classification_report()

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Cross validation for
credit models
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Cross validation basics
Used to train and test the model in a way that simulates using the model on new data

Segments training data into di erent pieces to estimate future performance

Uses DMatrix , an internal structure optimized for XGBoost

Early stopping tells cross validation to stop a er a scoring metric has not improved a er a
number of iterations

CREDIT RISK MODELING IN PYTHON

 How cross validation works
Processes parts of training data as (called folds) and tests against unused part

Final testing against the actual test set

1 h ps://scikit 2 learn.org/stable/modules/cross_validation.html

CREDIT RISK MODELING IN PYTHON

Setting up cross validation within XGBoost
# Set the number of folds
n_folds = 2
# Set early stopping number
early_stop = 5
# Set any specific parameters for cross validation
params = {'objective': 'binary:logistic',
'seed': 99, 'eval_metric':'auc'}

'binary':'logistic' is used to specify classi cation for loan_status

'eval_metric':'auc' tells XGBoost to score the model's performance on AUC

CREDIT RISK MODELING IN PYTHON

Using cross validation within XGBoost
# Restructure the train data for xgboost
DTrain = xgb.DMatrix(X_train, label = y_train)
# Perform cross validation
xgb.cv(params, DTrain, num_boost_round = 5, nfold=n_folds,
early_stopping_rounds=early_stop)

DMatrix() creates a special object for xgboost optimized for training

CREDIT RISK MODELING IN PYTHON

The results of cross validation
Creates a data frame of the values from the cross validation

CREDIT RISK MODELING IN PYTHON

Cross validation scoring
Uses cross validation and scoring metrics with cross_val_score() function in scikit-learn

# Import the module

from sklearn.model_selection import cross_val_score
# Create a gbt model
xg = xgb.XGBClassifier(learning_rate = 0.4, max_depth = 10)
# Use cross valudation and accuracy scores 5 consecutive times
cross_val_score(gbt, X_train, y_train, cv = 5)

array([0.92748092, 0.92575308, 0.93975392, 0.93378608, 0.93336163])

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Class imbalance in
loan data
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Not enough defaults in the data
The values of loan_status are the classes
Non-default: 0

Default: 1

y_train['loan_status'].value_counts()

loan_status Training Data Count Percentage of Total

0 13,798 78%
1 3,877 22%

CREDIT RISK MODELING IN PYTHON

Model loss function
Gradient Boosted Trees in xgboost use a loss function of log-loss
The goal is to minimize this value

True loan status Predicted probability Log Loss

1 0.1 2.3
0 0.9 2.3
An inaccurately predicted default has more negative nancial impact

CREDIT RISK MODELING IN PYTHON

The cost of imbalance
A false negative (default predicted as non-default) is much more costly

Person Loan Amount Potential Pro t Predicted Status Actual Status Losses
A $1,000 $10 Default Non-Default -$10
B $1,000 $10 Non-Default Default -$1,000
Log-loss for the model is the same for both, our actual losses is not

CREDIT RISK MODELING IN PYTHON

Causes of imbalance
Data problems
Credit data was not sampled correctly

Data storage problems

Business processes:
Measures already in place to not accept probable defaults

Probable defaults are quickly sold to other rms

Behavioral factors:
Normally, people do not default on their loans
The less o en they default, the higher their credit rating

CREDIT RISK MODELING IN PYTHON

Dealing with class imbalance
Several ways to deal with class imbalance in data

Method Pros Cons

Increases number of
Gather more data Percentage of defaults may not change
defaults
Increases recall for Model requires more tuning and
Penalize models
defaults maintenance
Sample data Least technical
Fewer defaults in data
di erently adjustment

CREDIT RISK MODELING IN PYTHON

Undersampling strategy
Combine smaller random sample of non-defaults with defaults

CREDIT RISK MODELING IN PYTHON

Combining the split data sets
Test and training set must be put back together

Create two new sets based on actual loan_status

# Concat the training sets

X_y_train = pd.concat([X_train.reset_index(drop = True),
y_train.reset_index(drop = True)], axis = 1)
# Get the counts of defaults and non-defaults
count_nondefault, count_default = X_y_train['loan_status'].value_counts()
# Separate nondefaults and defaults
nondefaults = X_y_train[X_y_train['loan_status'] == 0]
defaults = X_y_train[X_y_train['loan_status'] == 1]

CREDIT RISK MODELING IN PYTHON

Undersampling the non-defaults
Randomly sample data set of non-defaults

Concatenate with data set of defaults

# Undersample the non-defaults using sample() in pandas

nondefaults_under = nondefaults.sample(count_default)
# Concat the undersampled non-defaults with the defaults
X_y_train_under = pd.concat([nondefaults_under.reset_index(drop = True),
defaults.reset_index(drop = True)], axis=0)

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Model evaluation
and implementation
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Comparing classification reports
Create the reports with classification_report() and compare

CREDIT RISK MODELING IN PYTHON

ROC and AUC analysis
Models with be er performance will have more li

More li means the AUC score is higher

CREDIT RISK MODELING IN PYTHON

 Model calibration
We want our probabilities of default to accurately represent the model's con dence level
The probability of default has a degree of uncertainty in it's predictions

A sample of loans and their predicted probabilities of default should be close to the
percentage of defaults in that sample

Sample of Average predicted Sample percentage of actual

Calibrated?
loans PD defaults
10 0.12 0.12 Yes
10 0.25 0.65 No

h p://datascienceassn.org/sites/default/ les/Predicting%20good%20probabilities%20with%20supervised%20lea

CREDIT RISK MODELING IN PYTHON

Calculating calibration
Shows percentage of true defaults for each predicted probability

Essentially a line plot of the results of calibration_curve()

from sklearn.calibration import calibration_curve

calibration_curve(y_test, probabilities_of_default, n_bins = 5)

# Fraction of positives
(array([0.09602649, 0.19521012, 0.62035996, 0.67361111]),
# Average probability
array([0.09543535, 0.29196742, 0.46898465, 0.65512207]))

CREDIT RISK MODELING IN PYTHON

Plotting calibration curves
plt.plot(mean_predicted_value, fraction_of_positives, label="%s" % "Example Model")

CREDIT RISK MODELING IN PYTHON

Checking calibration curves
As an example, two events selected (above and below perfect line)

CREDIT RISK MODELING IN PYTHON

Calibration curve interpretation

CREDIT RISK MODELING IN PYTHON

Calibration curve interpretation

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Credit acceptance
rates
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Thresholds and loan status
Previously we set a threshold for a range of prob_default values
This was used to change the predicted loan_status of the loan

preds_df['loan_status'] = preds_df['prob_default'].apply(lambda x: 1 if x > 0.4 else 0)

Loan prob_default threshold loan_status

1 0.25 0.4 0
2 0.42 0.4 1
3 0.75 0.4 1

CREDIT RISK MODELING IN PYTHON

Thresholds and acceptance rate
Use model predictions to set be er thresholds
Can also be used to approve or deny new loans

For all new loans, we want to deny probable defaults

Use the test data as an example of new loans

Acceptance rate: what percentage of new loans are accepted to keep the number of
defaults in a portfolio low
Accepted loans which are defaults have an impact similar to false negatives

CREDIT RISK MODELING IN PYTHON

Understanding acceptance rate
Example: Accept 85% of loans with the lowest prob_default

CREDIT RISK MODELING IN PYTHON

Calculating the threshold
Calculate the threshold value for an 85% acceptance rate

import numpy as np
# Compute the threshold for 85% acceptance rate
threshold = np.quantile(prob_default, 0.85)

0.804

Loan prob_default Threshold Predicted loan_status Accept or Reject

1 0.65 0.804 0 Accept

2 0.85 0.804 1 Reject

CREDIT RISK MODELING IN PYTHON

Implementing the calculated threshold
Reassign loan_status values using the new threshold

# Compute the quantile on the probabilities of default

preds_df['loan_status'] = preds_df['prob_default'].apply(lambda x: 1 if x > 0.804 else 0)

CREDIT RISK MODELING IN PYTHON

Bad Rate
Even with a calculated threshold, some of the accepted loans will be defaults

These are loans with prob_default values around where our model is not well calibrated

CREDIT RISK MODELING IN PYTHON

Bad rate calculation

#Calculate the bad rate

np.sum(accepted_loans['true_loan_status']) / accepted_loans['true_loan_status'].count()

If non-default is 0 , and default is 1 then the sum() is the count of defaults

The .count() of a single column is the same as the row count for the data frame

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Credit strategy and
minimum expected
loss
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Selecting acceptance rates
First acceptance rate was set to 85%, but other rates might be selected as well

Two options to test di erent rates:

Calculate the threshold, bad rate, and losses manually

Automatically create a table of these values and select an acceptance rate

The table of all the possible values is called a strategy table

CREDIT RISK MODELING IN PYTHON

Setting up the strategy table
Set up arrays or lists to store each value

# Set all the acceptance rates to test

accept_rates = [1.0, 0.95, 0.9, 0.85, 0.8, 0.75, 0.7, 0.65, 0.6, 0.55,
0.5, 0.45, 0.4, 0.35, 0.3, 0.25, 0.2, 0.15, 0.1, 0.05]
# Create lists to store thresholds and bad rates
thresholds = []
bad_rates = []

CREDIT RISK MODELING IN PYTHON

Calculating the table values
Calculate the threshold and bad rate for all acceptance rates

for rate in accept_rates:

# Calculate threshold
threshold = np.quantile(preds_df['prob_default'], rate).round(3)
# Store threshold value in a list
thresholds.append(np.quantile(preds_gbt['prob_default'], rate).round(3))
# Apply the threshold to reassign loan_status
test_pred_df['pred_loan_status'] = \
test_pred_df['prob_default'].apply(lambda x: 1 if x > thresh else 0)
# Create accepted loans set of predicted non-defaults
accepted_loans = test_pred_df[test_pred_df['pred_loan_status'] == 0]
# Calculate and store bad rate
bad_rates.append(np.sum((accepted_loans['true_loan_status'])
/ accepted_loans['true_loan_status'].count()).round(3))

CREDIT RISK MODELING IN PYTHON

Strategy table interpretation
strat_df = pd.DataFrame(zip(accept_rates, thresholds, bad_rates),
columns = ['Acceptance Rate','Threshold','Bad Rate'])

CREDIT RISK MODELING IN PYTHON

Adding accepted loans
The number of loans accepted for each acceptance rate
Can use len() or .count()

CREDIT RISK MODELING IN PYTHON

Adding average loan amount
Average loan_amnt from the test set data

CREDIT RISK MODELING IN PYTHON

Estimating portfolio value
Average value of accepted loan non-defaults minus average value of accepted defaults

Assumes each default is a loss of the loan_amnt

CREDIT RISK MODELING IN PYTHON

Total expected loss
How much we expect to lose on the defaults in our portfolio

# Probability of default (PD)

test_pred_df['prob_default']
# Exposure at default = loan amount (EAD)
test_pred_df['loan_amnt']
# Loss given default = 1.0 for total loss (LGD)
test_pred_df['loss_given_default']

CREDIT RISK MODELING IN PYTHON

 Let's practice!
CREDIT RISK MODELING IN PYTHON
 Course wrap up
CREDIT RISK MODELING IN PYTHON

Michael Crabtree
Data Scientist, Ford Motor Company
Your journey...so far
Prepare credit data for machine learning models
Important to understand the data

Improving the data allows for high performing simple models

Develop, score, and understand logistic regressions and gradient boosted trees

Analyze the performance of models by changing the data

Understand the nancial impact of results

Implement the model with an understanding of strategy

CREDIT RISK MODELING IN PYTHON

Risk modeling techniques
The models and framework in this course:
Discrete-time hazard model (point in time): the probability of default is a point-in-time
event

Stuctural model framework: the model explains the default even based on other factors

Other techniques
Through-the-cycle model (continuous time): macro-economic conditions and other e ects
are used, but the risk is seen as an independent event

Reduced-form model framework: a statistical approach estimating probability of default

as an independent Poisson-based event

CREDIT RISK MODELING IN PYTHON

Choosing models
Many machine learning models available, but logistic regression and tree models were used
These models are simple and explainable

Their performance on probabilities is acceptable

Many nancial sectors prefer model interpretability

Complex or "black-box" models are a risk because the business cannot explain their
decisions fully

Deep neural networks are o en too complex

CREDIT RISK MODELING IN PYTHON

Tips from me to you
Focus on the data
Gather as much data as possible

Use many di erent techniques to prepare and enhance the data

Learn about the business

Increase value through data

Model complexity can be a two-edged sword

Really complex models may perform well, but are seen as a "black-box"

In many cases, business users will not accept a model they cannot understand

Complex models can be very large and di cult to put into production

CREDIT RISK MODELING IN PYTHON

 Thank you!
CREDIT RISK MODELING IN PYTHON