Graph Learning

in Financial Networks
Collaboration with Tianyu Du, Fan-yun Sun, Jure Leskovec

Jiaxuan You
Stanford University
 Financial Networks
§ Financial Networks: Describe financial entities and their connections
International banking Bitcoin transactions
• Nodes: Countries • Nodes: BTC wallets
• Edges: Capital flows • Edges: Transactions

Image credit: The Political Economy of Global Finance: A Network Model Image credit: https://dailyblockchain.github.io/
Jiaxuan You, Stanford University 2
 Graph Learning in Financial Networks
§ Goal: A graph learning framework for financial networks
§ Applications: Fraud detection, Anti-money laundering, Anomaly detection
§ Solution: Graph representation learning!

Graph Node-level: Fraudsters, …

Neural
Network
Subgraph-level: Money
laundering subnetworks, …

Edge-level: fraudulent/anomalous
transactions, …

Input: Financial networks Output: Predictions

Jiaxuan You, Stanford University 3
 Why Graph Representation?
§ Transaction-based approach
§ “On 01/03, Client 𝐴 sends Company 𝐵 $500” $100, 01/06

§ Build models based on transaction attributes

§ Issues: ignore the context of a transaction $500, 01/03
$200, 01/02
$400, 01/05

§ Graph-based approach $200, 01/02

$100, 01/01
§ Represent transactions as a dynamic graph
§ Predictions are made based on the entire graph client bank company

§ Benefits:
§ Represents a transaction with a broader context
§ Requires fewer feature-engineering

Jiaxuan You, Stanford University 4

 Overview of This Talk

ROLAND: GNN Results & Application:

for Financial Insights from Anomaly
Networks ROLAND Detection
NN

NN NN

NN
NN

NN

client bank company

Graph Neural Networks Jiaxuan You, Stanford University 5

 J. You, T. Du, J. Leskovec. ROLAND: Graph Neural Networks for Dynamic Graphs, in submission

ROLAND: GNN for Financial Networks

§ ROLAND framework: Will there be a transaction? Yes
Self-supervised
What is the amount? $500
§ Transform financial networks as GNN (from raw data) When will it happen? 01/03
computational graphs Supervised
…
Is it a fraud? No
§ Learning from diverse signals (from external Is it money laundering? Yes
…
sources)

$100, 01/06 ?
NN

$400, 01/05 NN
$500, 01/03 NN
$200, 01/02
NN NN node
$200, 01/02
NN
embeddings
$100, 01/01

client company client company client bank company

bank bank

Financial networks Graph Neural Networks Learning Signals & Application

Jiaxuan You, Stanford University 6

 Message
passing
GNN Layer
layers

ROLAND Model: From Static to Dynamic GNN

MLP Layer Post-
process
MLP Layer layers
b) Extend static GNNs to dynamic GNNs
(()
This Hierarchical Node state #% = {#% " , … , #% }
a)isPre-A static GNN with
transaction
MLP Layer t+1
process Pred "% t+2"%'"
Pred
fraudulent
MLP Layer
modern
layers architectural design options t+3
($) ($)
#%&" #%
($)
Embedding #%&" Embedding
GNN Layer Linear
Pred " Aggregation update update
BatchNorm ($)
$%&" $%($)
GNN Layer #
($) # #
Attention

SOTA GNN Layer GNN Layer 2 Activation

Dropout GNN Layer 2 GNN Layer 2
(") (")
Static Skip
connections
#
(")
Dropout
Activation #%&"
(")
#%&"
#%
MLP Layer Post- Embedding Embedding …
GNN process GNN Layer 1 …
Attention
update update
layers BatchNorm
MLP Layer
Aggregation #
(")
$%&" $%(")
#
Linear
Graph ! GNN Layer 1 GNN Layer 1
Question: Can we adapt a
SOTA
Snapshotstatic
!%&" GNNSnapshot
to dynamic
!%

prediction tasks?
Jiaxuan You, Stanford University 7
 ROLAND Model: From Static to Dynamic GNN
§ Idea: Recurrently update node
Pred "! Pred "!$#
embeddings at each layer
§ Introduce a new module to a static GNN:
Embedding Embedding
update update
Input:
Pred " GNN Layer 2 GNN Layer 2
§ Previous embeddings from the
Embedding the same layer
GNN Layer 2 … Embedding Embedding … update
update update § Current embeddings from the
GNN Layer 1
previous layer
GNN Layer 1 GNN Layer 1
Output: Updated embeddings
Graph ! Snapshot !!"# Snapshot !!

§ Benefits:
Static GNN Dynamic GNN
§ Simple and effective
§ Benefit from the SOTA designs of a static GNN
Jiaxuan You, Stanford University 8
 ROLAND Training: Efficient Training
Live-update evaluation pipeline
§ Incremental training:
Time ! Time ! + #
§ Only keep these in GPU
!##! !##!$#
§ GNN Model 𝐺𝑁𝑁!
Train Train
§ Historical node states 𝐻!"#
Evaluate Evaluate § Incoming new graph snapshot 𝐺!
§ Efficient and work well in practice
"! "$! "!$# "$!$#
§ Meta training:
… State …
!##!"#
%!
!##!
§ Train a meta-GNN that can quickly adapt
to new data
Snapshot Snapshot
!!"# !! § Benefits: ROLAND does not need to be
Only this part in GPU frequently retrained
Jiaxuan You, Stanford University 9
 Overview of This Talk

ROLAND: GNN Results & Application:

for Financial Insights from Anomaly
Networks ROLAND Detection
NN

NN NN

NN
NN

NN

client bank company

Graph Neural Networks Jiaxuan You, Stanford University 10

 J. You, R. Ying, J. Leskovec. Design Space of Graph Neural Networks, NeurIPS 2020

ROLAND Implementation: GraphGym

§ GraphGym offers a standardized pipeline for graph learning
§ Design Space of Graph Neural Networks (NeurIPS 2020 spotlight)
§ Now officially part of PyG 2.0! (Stay tuned for my next talk :)

General GNN Clean interface for

design space comparing designs

§ We implement the financial prediction pipeline with GraphGym

§ We adopt many optimal design choices from GraphGym
Jiaxuan You, Stanford University 11
 An Example Financial Dataset

§ Transactions between companies captured by a bank clearing system

§ 336 million transactions, 6 million accounts, 6 years of daily data

Jiaxuan You, Stanford University 12

 Task 1: Classify Fraud Transactions
§ Task setting: Supervised learning
§ Unbalanced labels: 2% of all the transactions are fraudulent
§ Random dataset split: 80% training, 10% validation, 10% testing
§ Metric: AUC on test set (random guess has AUC 0.5, higher is better)
§ Best non-deep learning baseline
§ AUC: ~0.8
§ Simple GNN implementation using out-of-the-box PyG
§ AUC: 0.81
§ ROLAND framework works much better!
§ AUC: 0.94

Jiaxuan You, Stanford University 13

 Task 2: Forecast Future Transactions
§ Task setting: Self-supervised/Unsupervised learning
§ Rolling prediction: On each day, use all the historical information to predict the
transactions on the next day
§ Metric: Mean reciprocal ranking (MMR) of ground-truth future transactions

ROLAND significantly
outperforms SOTA
baselines
Jiaxuan You, Stanford University 14
 Analysis: ROLAND’s Performance over Time
Transaction
amount

# of
retraining Almost no
epochs retraining
needed!

Test MRR
Recall@1

(week)day of the year

ROLAND maintains a robust performance over transaction pattern changes

Jiaxuan You, Stanford University 15
 Analysis: Interpreting ROLAND’s Predictions
§ Question: What kind of transactions the
model paid attention to?
§ Method: Inspect edges' attention scores of Non-financial à
Household
the last GNN layer.
§ Select edges with low/high attention scores.
§ Get the feature value distribution of these
edges
Domestic -> Abroad
Higher attention
to large amount
transactions

Jiaxuan You, Stanford University 16

 Overview of This Talk

ROLAND: GNN Results & Application:

for Financial Insights from Anomaly
Networks ROLAND Detection
NN

NN NN

NN
NN

NN

client bank company

Graph Neural Networks Jiaxuan You, Stanford University 17

 Application: Anomaly Detection
§ Goal: Detect anomalous transactions without external supervision
§ Combine Graph Representation Learning with Outlier Detection
node embeddings

edge embeddings

Input: Financial networks

Jiaxuan You, Stanford University 18
 Self-supervised objectives
“Indicators and patterns of money laundering or terrorist financing”

Real world Indicators Objectives for GNN

(from baselgovernance.org)
investment funds sent to countries of concern Predict country
Large cash deposits into (company) accounts Predict amount
unusual subject of transaction Predict subject of transaction
(link prediction)
high volume of transactions within a short period predict acculumated amount

multiple individuals sending funds to the one beneficiary predict # transactions

Jiaxuan You, Stanford University 19

 Objectives for Anomaly Detection
§ Solution: ROLAND framework + diverse objectives
dim
t+1 Self-supervised Objectives
t+2
t+3 ● Objective 1: predict amount
● Objective 2: predict recipient
● Objective 3: predict country
|N| ● …

Outlier Detection Objectives

Jiaxuan You, Stanford University 20

Quantitative Evaluation for Anomaly Detection

● Task: Detect anomalous transactions without supervision

● Goal: Simulate real-world anomaly detection use cases
○ Corrupt the dataset to see if ROLAND can extract them
○ Report precision/recall of corrupted edges detected by ROLAND
○ Ways of corrupting the dataset:
■ Amount corruption (~1% transactions)
■ Country corruption (~1% transactions)
■ Recipient corruption (~1% transactions) → Rewire !

A B A B

C D C D
Jiaxuan You, Stanford University 21
 Quantitative Evaluation for Anomaly Detection
+14.7% Precision +14.2% Recall
Deep learning baseline:

Rule-based baseline: +172% Precision +171% Recall

Jiaxuan You, Stanford University
 Normal accounts
Potential anomalous
accounts

Jiaxuan You, Stanford University 23

 Overview of This Talk

ROLAND: GNN Results & Application:

for Financial Insights from Anomaly
Networks ROLAND Detection
NN

NN NN

NN
NN

NN

client bank company

Resources: ROLAND (in submission)

Graph Neural Networks Design Space for Graph Neural Networks
Learn more about GNN: PyG.org
Jiaxuan You, Stanford University 24