`

IIF and EY Survey Report

on Machine Learning
Uses in Credit Risk and AML
Applications

Public Summary
December 2022

0
Table of Contents
Introduction to the IIF and EY Survey Report on Machine Learning — Uses in Credit Risk
and AML Applications ......................................................................................................................... 1
A. Executive Summary ....................................................................................................................... 2
B. Survey Methodology, Participants, and Use Cases ........................................................................ 3

Machine Learning use in Credit Risk and Anti-Money Laundering Applications — Survey
Results and Analysis ........................................................................................................................... 4
1. Global Journey of Adoption of ML in Production ............................................................................. 5
2. Realized Benefits of ML Use .......................................................................................................... 7
3. Challenges for ML Use ................................................................................................................... 8
4. Machine Learning Governance Maturity ......................................................................................... 9
5. Engagement with Regulators.......................................................................................................... 9
6. Model Validation ........................................................................................................................... 10
7. Controls against Unfairness / Bias ................................................................................................ 11
8. Model Monitoring — Feedback Mechanisms and Controls ........................................................... 12
Conclusion ....................................................................................................................................... 13
EY and IIF Contacts ......................................................................................................................... 13

Glossary ............................................................................................................................................. 14

IIF | EY
Introduction to the IIF and EY
Survey Report on Machine
Learning — Uses in Credit Risk and
AML Applications

1 IIF | EY
A. Executive Summary
As the use of machine learning (ML) has grown across the financial services industry, benefits and inherent risks
have grown as well. Risk management of ML has emerged as a primary consideration for financial institutions as
they determine the target scope and scale of machine learning use. To explore the latest machine learning risk
management practices at financial institutions, the Institute of International Finance (IIF) and EY Global Services
Limited (EY) have developed a joint publication focused specifically on machine learning risk management for (1)
credit risk management and (2) anti-money laundering (AML)1 purposes.2 This joint publication is based on IIF’s
annual survey of ML governance practices, which included 43 participating institutions across the world in 2022,
including global systemically important banks (G-SIBs), national banks, regional banks, and other financial
institutions, including insurers.

The survey and analysis

cover the following ML topics:
Global Journey of Adoption of ML
1. in Production

2. Realized Benefits of ML Use

3. Challenges for ML Use

Machine Learning Governance
4. Maturity

5. Engagement with Regulators

6. Model Validation

7. Controls against Unfairness / Bias

Model Monitoring — Feedback
8. Mechanisms and Controls

1
Formal definitions for credit risk management, anti-money laundering, and other key terminology are found in the Glossary at
the end of this report.
2
This publication is the summary report of the 2022 Survey on ML Uses in Credit Risk and AML Applications. The full report
comprehends a more extensive set of analysis and is only available to responding institutions.

2 IIF | EY
Key takeaways from this year’s survey include:

1 The majority of respondents are using machine learning techniques in production for credit risk
management and anti-money laundering;
Top realized benefits for credit risk management included increased model accuracy, identification of
2 new risk segments or patterns, and the ability to conduct broad analysis of risk management data from
different sources;

3 Top realized benefits for AML included lower false-positive numbers, identification of new patterns, and
increased predictive analysis;

4 Key challenges in adoption of ML included data quality, explainability, and IT-infrastructure;

Machine learning applications are most often being governed through existing model risk management
5 or enterprise risk frameworks;
Key considerations raised by financial institutions with regulators include the complex nature of some
6 algorithms and outcomes, bias and ethical issues related to the use of machine learning, and
transparency;
Primary techniques used in validating ML models include ongoing performance monitoring, monitoring
7 against benchmarks, in-sample / out-of-sample testing, and data quality validation;

To mitigate against unfairness/biased or discriminatory outcomes, institutions leverage a wide range

8 of controls, including excluding sensitive attributes from feature analysis/selection/engineering,
alignment with institutional codes of ethics, auditing, testing, and other controls;
From a model monitoring perspective, the majority of respondents either have feedback mechanisms
9 or controls in place for correcting the ML models to help ensure outcomes are as expected, or currently
are in the process of defining feedback mechanisms.

B. Survey Methodology, Participants, and Use Cases

IIF staff surveyed a globally diverse group of 43 financial institutions over the period from January to September
2022, with a mix of multiple-choice questions and questions that sought more expansive commentary.3 For certain
multiple-choice questions, only one option could be selected; other multiple-choice questions allowed for the
selection of multiple options from the list, as appropriate. In some instances, institutions did not respond completely
to all questions, which has impacted the distribution of responses. Overall, survey results are based on the sample
size of 43 participants and the responses of participants are not necessarily representative of the global population
of financial institutions. Furthermore, the survey was taken in a dynamically changing environment and the results
should be viewed as a snapshot in time.4
The sample of participants span multiple institution types located across six continents. Financial institutions are
categorized by region according to where they are headquartered, while acknowledging that many have operations
across multiple jurisdictions. There are nine regions represented in this study covering 43 financial institutions: Euro
Area (eight institutions), Other Europe (seven institutions), Latin America (three institutions), United States (four
institutions), Canada (two institutions), Asia-Pacific (seven institutions), Japan (four institutions), China (three
institutions), and the Middle East and Africa (five institutions). The Euro Area region consists of firms that are
headquartered in countries that use the Euro as a currency. The “Other Europe” region is composed of firms that
are headquartered in the Nordics, Switzerland, and the UK.

3
Firms’ survey responses have been collated and anonymized by the IIF in advance of sharing with EY and the joint report
writing team. The commentary presented in this report is not representative of any individual firm.
4
Financial institutions navigated expansive international sanctions imposed against Russia following its invasion of Ukraine in
2022. This may have impacted survey participation and responses.

3 IIF | EY
Machine Learning use in Credit
Risk and Anti-Money Laundering
Applications — Survey Results and
Analysis

4 IIF | EY
1. Global Journey of Adoption of ML in Production
1.A. Credit Risk Management
The majority of financial institutions surveyed apply machine learning techniques in production for credit risk
management. In particular, as shown in Figure 1.1, 52% of the respondents are applying new techniques in
production, while an additional 25% of the institutions are currently experimenting with their use. A small portion of
institutions are not applying machine learning techniques at the current stage but plan to do so in the foreseeable
future. Only a very small portion (9%) of the institutions do not have any plans to apply machine learning techniques.
Per Figure 1.2, the majority of financial institutions in the Euro Area and all in the United States are applying ML
techniques in production.

Figure 1.1: Do you apply any Machine Learning

techniques in your credit risk-related analyses?

9%

14%

52%

25%

a. Yes, we are applying new techniques in production b. Yes, but we are only experimenting (pilot projects)
c. No, but planning for the foreseeable future d. No, and we have no plans to

Figure 1.2: Do you apply any Machine Learning techniques

in your credit risk-related analyses?

0 5 10 15 20 25

a. Yes, we are applying new techniques in production

b. Yes, but we are only experimenting (pilot projects)

c. No, but planning for the foreseeable future

d. No, and we have no plans to

Euro Area Other Europe Latin America

U.S. Canada Asia Pacific (excl. Japan and China)
Japan China Middle East and Africa

5 IIF | EY
1.B. AML
The adoption of ML usage in production for AML is similar relative to credit risk management; however, a higher
percentage of respondents are experimenting with machine learning applications for AML (31%) than for credit risk
management (25%), and a negligible portion (3%) of institutions do not have any plans to apply machine learning
techniques.

Figure 1.3: Do you apply any Machine Learning

techniques in your AML-related analyses?

3%
15%

51%

31%

a. Yes, we are applying new techniques in production b. Yes, but we are only experimenting (pilot projects)
c. No, but planning for the foreseeable future d. No, and we have no plans to

6 IIF | EY
2. Realized Benefits of ML Use
2.A. Credit Risk Management
For credit risk management, increased model accuracy was overall the most improved outcome observed when
using ML techniques globally. Discovery of new risk segments/patterns and the ability to conduct a holistic analysis
of different data sources were commonly improved outcomes as well. Notably, financial institutions were more likely
to adopt ML techniques to increase productivity rather than save costs, as shown in Figure 2.1.

Figure 2.1: What are the improved outcomes of using ML

techniques in your credit risk analyses?
8
7
6
5
4
3
2
1
0
a. Discovery of b. Ability to c. Overcoming d. Increase e. More f. Cost savings g. Other
new risk conduct holistic data model accuracy efficiency in
segments or analysis of deficiencies or model
patterns that different data inconsistencies development
were previously sources process
unnoticed

Euro Area Other Europe Latin America

U.S. Canada Asia Pacific (excl. Japan and China)
Japan China Middle East and Africa

2.B. AML
For AML, institutions highlighted two key benefits: improving the efficiency of their models and discovering new risk
segments/patterns. Furthermore, firms realized a reduction in false-positives through the use of machine learning
and, in turn, a reduction in operating costs, per Figure 2.2.
`

7 IIF | EY
 Figure 2.2: What are the improved outcomes
of using ML techniques in AML? (By region)
0 5 10 15 20 25 30

a. Discovery of new risk segments or money laundering

patterns that were previously unnoticed by systems

b. Ability to conduct holistic analysis of different data

sources
c. Increased speed and/or automation of analysis allow the
AML process to respond to the latest developments in
money laundering methods

d. Lower false positive numbers

e. Increased automation efficiency

f. Increased predictive analysis based on historical data

and customer behavior

g. Higher % of SARs to Alerts

h. Higher % of SARs followed-up by FIU

i. Cost savings

Euro Area Other Europe Latin America

U.S. Canada Asia Pacific (excl. Japan and China)
Japan China Middle East and Africa

3. Challenges for ML Use

3.A. Credit Risk Management
For credit risk management, explainability, IT-infrastructure, supervisory considerations, and data quality were the
four primary challenges for ML use identified. Notably, machine learning governance was not raised as a key
challenge, which may be related to the advanced stage of model risk governance implementation across the
financial services industry (particularly in banks). Additionally, among regions, firms in China specifically raised the
key challenge of helping to ensure that businesses are aware of potential limitations to AI/ML use.
From a data perspective, multiple data sources and formats and poor data quality (e.g., lack of labeled data) were
key challenges raised. These were identified across regions and financial institution types.

8 IIF | EY
3.B. AML
For AML, data quality, explainability, and IT-infrastructure were again raised as primary challenges. Data quality
was highlighted as a greater challenge for AML than for credit risk management. Multiple respondents also
mentioned the lack of appropriately skilled staff as a challenge.

4. Machine Learning Governance Maturity

4.A. Credit Risk Management and AML
For credit risk management, the majority of institutions across regions govern machine learning applications through
existing model risk management or enterprise risk frameworks or made incremental updates to their existing model
governance frameworks to cover machine learning models. At the same time, a number of respondents indicated
that new governance mechanisms had been developed for ML applications that complement existing frameworks.
Based on the survey results, the governance for AML machine learning applications is very similar to the
governance for credit risk management applications, with slight differences noted specific to the nature of AML
governance.

5. Engagement with
Regulators
5.A. Credit Risk Management
From a credit risk management perspective, most
respondents have engaged regulators in the
application of ML techniques. Of the respondents that
have not engaged yet, over half plan to.
Regarding the key topics raised with regulators,
common observations were noted across
geographies and included the “black box” nature of
some algorithms, bias and ethical issues, regulatory
constraints for credit risk applications, and
transparency.

5.B. AML
For AML, most respondents have engaged regulators
or plan to in the future. Similar to the results for credit
risk management, the institutions that have already
engaged with regulators are also institutions that tend
to have increased regulatory requirements based on
the size and type of the institutions.
The top topics raised to regulators were the “black
box” nature of algorithms, transparency, and the lack
of previous experience of supervisors, with the “black
box” nature of algorithms raised across institution
types.

9 IIF | EY
6. Model Validation
6.A. Credit Risk Management
For credit risk management, there was a broad range of model validation techniques used for machine learning
models. Ongoing performance monitoring, in-sample/out-of-sample testing, outcome monitoring against a
benchmark, explainability tools, and data quality validation were the top five techniques reported and were heavily
utilized across regions and institutions, as shown in Figure 6.1 below. Black box testing was the least commonly
performed technique in validation.

Figure 6.1: What model validation techniques are used to assess

machine learning model robustness for credit risk applications?

0 5 10 15 20 25

a. Ongoing performance monitoring

b. Outcome monitoring against a benchmark

c. Outcome monitoring against a non-ML model / A-B

testing

d. Black box testing

e. Explainability tools

f. Validation of engineered features

g. Data quality validation

h. In-sample / out-of-sample testing

i. Other

Euro Area Other Europe Latin America

U.S. Canada Asia Pacific (excl. Japan and China)
Japan China Middle East and Africa

6.B. AML
The validation techniques utilized for AML machine learning applications were very similar to those used in credit
risk management applications; however, ongoing performance monitoring was utilized significantly more than other
techniques for AML validation specifically.

10 IIF | EY
7. Controls against Unfairness / Bias
7.A. Credit Risk Management
Only one institution responding to this question was not utilizing certain mechanisms to avoid bias and
discriminatory outcomes for credit risk management. The top three responses were (1) an institution-level code of
ethics, (2) auditing, testing, and controls, and (3) excluding features such as gender, race, and other sensitive
attributes from the beginning to prevent them from being part of the feature analysis, selection, and engineering
process.

Figure 7.1: What mechanisms are in place for credit risk

applications to mitigate against ML models producing biased or
discriminatory outcomes?

0 2 4 6 8 10 12 14 16

a. Firm has a code of ethics defined at the institution level

b. Firm has a specific code of ethics for ML

c. Auditing, testing and controls

d. Firm has engaged external data ethics advisors or experts

to help inform your consideration of the ethics of particular
machine learning use cases
e. Firm has regular reporting to either board or group risk
committee (or equivalent) with ethical risks relating to
machine learning

f. Features such as gender/race/sensitive attributes are

not available due to country-level restrictions

g. Features such as gender/race/sensitive attributes are

included in analysis and development and subsequently
excluded from the model after assessing bias
h. Features such as gender/race/sensitive attributes are
excluded from the beginning and not part of the feature
analysis / selection / engineering process

i. Features such as gender/race/sensitive attributes are

excluded from model development but used to evaluate bias

j. Reject Inference

k. Any other approach

Euro Area Other Europe Latin America

U.S. Canada Asia Pacific (excl. Japan and China)
Japan China Middle East and Africa

11 IIF | EY
At the same time, there was a broad set of responses and notable regional trends. Euro Area firms utilized all three
top responses heavily, whereas European financial institutions outside the Euro Area primarily defined a code of
ethics at the institutional level. Institutions in the Middle East and Africa instead relied more on a specific code of
ethics for machine learning and regular reporting of ethical risks to the group risk committee and/or the board.

7.B. AML
For AML, the distribution of responses was similar to the distribution for credit risk management. One notable
difference was the order of the top three responses, where auditing, testing, and controls was the most common
response for AML.

8. Model Monitoring — Feedback Mechanisms and Controls

8.A. Credit Risk Management
For credit risk management, most institutions have existing feedback mechanisms and controls for correcting the
ML models to help ensure outcomes are as expected or were in the process of defining feedback mechanisms.
Regional banks were more likely to not have feedback mechanisms or controls already in place but were in the
process of defining such mechanisms and controls.

8.B. AML
For AML, similar to credit risk applications, most institutions either had existing feedback mechanisms and controls
in place for correcting the ML models to help ensure outcomes are as expected or were currently in the process of
defining feedback mechanisms. However, compared to the responses to credit risk management, a larger
proportion of respondents indicated that they were in the process of defining mechanisms vs. having them in
production currently.

12 IIF | EY
Conclusion
The majority of financial institutions are using machine learning techniques in production, and financial institutions
found numerous benefits as a result of adopting ML techniques in credit risk management and anti-money
laundering. These benefits for customers, employees, shareholders, and society were effectively recognized by the
survey respondents and manifested in the survey results.
At the same time, risk management is a key focus for financial institutions as they are progressing to expand
machine learning usage. From a governance perspective, existing model risk management or enterprise risk
frameworks are most often used. From a model validation perspective, key techniques used include ongoing
performance monitoring, monitoring against benchmarks, in-sample / out-of-sample testing, and data quality
validation. For model monitoring, most respondents either have feedback mechanisms and controls in place for
correcting the ML models to help ensure outcomes or are currently in the process of defining feedback mechanisms.
Top challenges highlighted by survey respondents included data quality, explainability, and IT-infrastructure.
Furthermore, key considerations raised by financial institutions with regulators include the “black box” nature of
some algorithms, bias and ethical issues related to the use of machine learning, and transparency.
The IIF and EY Survey Report on Machine Learning – Uses in Credit Risk and AML Applications Public Summary
is a continuation of a multiyear effort to study global machine learning risk management practices. In future surveys,
the benefits of using machine learning are expected to be further illuminated, along with challenges and risks and
the associated mitigants and control frameworks to overcome these challenges.

EY and IIF Contacts

IIF EY
Jessica Renier Peter Marshall
Managing Director, Digital Finance Partner/Principal, Financial Services Risk
[email protected] Management, Ernst & Young LLP
[email protected]
Conan French
Director, Digital Finance Jan Zhao
[email protected] Partner/Principal, Data and Analytics, Ernst &
Young LLP
[email protected]
Daniel Mendez Delgado
Associate Policy Advisor, Digital Finance Susan Raffel
[email protected] Partner/Principal, Financial Services Risk
Management, Ernst & Young LLP
[email protected]
Ryan Moore
Managing Director, Financial Services Risk
Management, Ernst & Young LLP
[email protected]
Aditya Desai
Manager, Financial Services Risk Management,
Ernst & Young LLP
[email protected]

13 IIF | EY
Glossary

14 IIF | EY
Artificial Intelligence: The theory and development of computer systems able to perform tasks that traditionally
have required human intelligence. 5 It is broadly applied when a machine mimics cognitive functions that humans
associate with other human minds, such as learning and problem-solving.

Bias: An unfair inclination for or prejudice against a person, group, object, or position.

Black Box Testing: Input-output testing without reference to the internal structure of the ML application. The
developer “experiments” with the model, feeding it different data inputs to better understand how the model makes
its predictions.

Credit Risk: The risk to current or projected financial condition and resilience arising from an obligor’s failure to
meet the terms of any contract with the bank or otherwise perform as agreed. Credit risk is found in all activities in
which settlement or repayment depends on counterparty, issuer, or borrower performance. Credit risk exists any
time bank funds are extended, committed, invested, or otherwise exposed through actual or implied contractual
agreements, whether reflected on or off the balance sheet.6

Data Quality Validation: Refers to when one or more techniques are used to help ensure potential issues with
data (such as class imbalances, missing or erroneous data) are understood and considered in the model
development and deployment process. Examples of these include data certification, source-to-source verification
or data issues tracking.

Ethics: A system of moral principles governing a person’s behavior or the conduct of an activity. In the case of
financial institutions, ethics bridges the gap between regulated and non-regulated spaces — that is, firms know
what they should do (what is right or wrong). Financial institutions have long-established ethical standards that are
enshrined in firms’ values and codes of conduct, incremental to those that are adopted in response to regulatory
requirements such as those relating to fair lending or best interest standards. It is important to note that what is
deemed “ethical” varies between individuals, societies, and jurisdictions, and can change over time.

Explainability Tools: Tools and techniques aimed at explaining the inner workings of the ML model.

Machine Learning (ML): One of the techniques used for AI and includes neural networks among others. In general,
ML is characterized by an algorithm autonomously “learning the rules” or “developing a model” from training data
and using it to predict outcomes for new data (i.e., not from the training set).

Example ML modeling approaches within the scope of this survey include:

• Ensemble methods (e.g., Gradient Boosting Machine, Random Forest, and Isolation Forest)

• Neural networks (trained through supervised, unsupervised, or semi-supervised learning) Kernel or

instance-based algorithms (e.g., Support Vector Machines and Support Vector regression)

• Complex dependence structure (e.g., Hidden Markov Models, Bayesian Networks, and Generative
Adversarial Networks)

• Online or reinforcement learning (e.g., Q-Learning, State-Action-Reward-State-Action, and Adaptive

Dynamic Programming)

Model Governance: Sets an effective framework with defined roles and responsibilities for clear communication of
model limitations and assumptions, as well as the authority to restrict model usage. A strong governance framework
provides explicit support and structure to risk management functions through policies defining relevant risk
management activities, procedures that implement those policies, allocation of resources, and mechanisms for
evaluating whether policies and procedures are being carried out as specified. Notably, the extent and sophistication
of a bank’s governance function is expected to align with the extent and sophistication of model usage.7

5
FSB’s report – ‘Artificial intelligence and machine learning in financial services - Market developments and financial stability
implications’, November 1, 2017.
6
Comptroller's Handbook: Large Bank Supervision | OCC, accessed December 2022.
7
SR 11-7 attachment: Supervisory Guidance on Model Risk Management (federalreserve.gov), accessed December 2022.

15 IIF | EY
Model Risk: The potential for adverse consequences from decisions based on incorrect or misused model outputs
and reports. Model risk can lead to financial loss, poor business and strategic decision-making, or damage to a
bank’s reputation.7

Model Validation: The set of processes and activities intended to verify that models are performing as expected,
in line with their design objectives and business uses. Effective validation helps ensure that models are sound. It
also identifies potential limitations and assumptions and assesses their possible impact.7

Anti-Money Laundering: Money laundering is the criminal practice of processing ill-gotten gains, or “dirty” money,
through a series of transactions; in this way, the funds are “cleaned” so that they appear to be proceeds from legal
activities. Money laundering generally does not involve currency at every stage of the laundering process. Although
money laundering is a diverse and often complex process, it basically involves three independent steps that can
occur simultaneously.8 Anti-money laundering consists of laws, rules, and regulations to prevent money laundering.

Outcome Monitoring against a Benchmark: Refers to when decisions or actions associated with the ML system
are monitored using one or multiple metrics. Performance is assessed against a certain benchmark value of those
metrics.

Outcome Monitoring against a Non-ML model / A-B testing: Decisions or actions associated with the ML system
that are monitored using one or multiple metrics. Performance is assessed by comparing it to the performance of a
separate, non-ML model. The same approach is used in A-B testing (also known as split testing).

Validation of Engineered Features: Engineered features used in the ML application are scrutinized, including
potential impacts on model performance.

8
FFIEC BSA/AML Examination Manual, accessed December 2022.

16 IIF | EY
 iif.com © Copyright 2022. The Institute of International Finance, Inc. All Rights Reserved.

This publication contains information in summary form and is therefore intended for general guidance only. It is
not intended to be a substitute for detailed research or the exercise of professional judgment. Member firms of the
global EY organization cannot accept responsibility for loss to any person relying on this article.

17 IIF | EY