Lecture 11: Probabilistic Machine Learning for Neural

Networks
 Lecture #11
Outline

• Machine Learning Pipeline for Predictive Maintenance

• Probabilistic Machine Learning Techniques

• Gaussian Process Regression
• Bayesian Neural Network
• Monte Carol Dropout
• Neural Network Ensemble
• Deep Neural Network with Gaussian Process Regression
• Spectral-normalized Neural Gaussian Process

2
 Problem Definition

ML Pipeline for Predictive Maintenance

Physical Digital
domain domain
Sense

Bearing Vibration data

REIL Reliability Engineering &

Informatics Laboratory 3
 Problem Definition

ML Pipeline for Predictive Maintenance

Physical Digital ML model

domain domain
Sense Process Inference
De-noised Fault
Bearing Vibration data data probability

REIL Reliability Engineering &

Informatics Laboratory 4
 Problem Definition

ML Pipeline for Predictive Maintenance

Physical Digital ML model Physical

domain domain Decision domain
Sense Process Inference making
De-noised Fault
Bearing Vibration data data probability Maintenance

Act and improve

An example focus is on
bearing fault classification.

REIL Reliability Engineering &

Informatics Laboratory 5
 How to Quantify Predictive Uncertainty?

Looking at a 2D binary classification problem

Training data

+
𝑋𝑋2

Test data (OoD)

–

𝑋𝑋1

REIL Reliability Engineering &

Informatics Laboratory 6
 How to Quantify Predictive Uncertainty?

Spectral-normalized Neural Gaussian process (SNGP)

Distance-aware UQ

Class probability Predictive uncertainty

𝑋𝑋2

𝑋𝑋2

𝑋𝑋1 𝑋𝑋1
Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T. and Lakshminarayanan, B., 2020. Simple and principled uncertainty estimation
with deterministic deep learning via distance awareness. NIPS, 33, pp.7498-7512.

REIL Reliability Engineering &

Informatics Laboratory 7
 More on UQ of Machine Learning Models

Method 1: Gaussian process regression

Prior (zero mean) Conditionals of MVN

𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 Shape of
Kernel on input conditional PDF
features 𝐱𝐱 𝑝𝑝 𝑦𝑦∗ | 𝑦𝑦1 = 1
0.
𝑝𝑝(𝑦𝑦1, 𝑦𝑦*)

0.3
Posterior 0.2
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 4
Mean y1 0 2 4 -4 -2 0 2
y*
Variance Observation: y1 = 1

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 8
 More on UQ of Machine Learning Models

Method 1: Gaussian process regression A regular

neural network
Prior (zero mean) Conditionals of MVN
𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 Shape of
Kernel on input conditional PDF x y
features 𝐱𝐱 𝑝𝑝 𝑦𝑦∗ | 𝑦𝑦1 = 1
0.
𝑝𝑝(𝑦𝑦1, 𝑦𝑦*)

0.3
Posterior 0.2
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 4 Weights and biases
Mean y1 0 2 4 -4 -2 0 2
y*
Variance Observation: y1 = 1

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 9
 More on UQ of Machine Learning Models

Method 1: Gaussian process regression

Prior (zero mean) Conditionals of MVN

𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 Shape of
Kernel on input conditional PDF x y
features 𝐱𝐱 𝑝𝑝 𝑦𝑦∗ | 𝑦𝑦1 = 1
0.
𝑝𝑝(𝑦𝑦1, 𝑦𝑦*)

0.3 Posterior
0.2

PDF
Posterior 1
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 Weights and biases, Prior
4
Mean y1 0 2 4 -4 -2 0 2 e.g., 𝜃𝜃𝑘𝑘 ~𝒩𝒩 𝜇𝜇𝑘𝑘 , 𝜎𝜎𝑘𝑘2
y*
Variance Observation: y1 = 1 𝜃𝜃
1 Bayesian inference

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 10
 More on UQ of Machine Learning Models

Method 1: Gaussian process regression Method 2: Bayesian neural network

(MCMC or VI)
Conditionals of MVN Predictive
Prior (zero mean) distribution

PDF
𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 Shape of
Kernel on input conditional PDF x y
features 𝐱𝐱 𝑝𝑝 𝑦𝑦∗ | 𝑦𝑦1 = 1 𝑦𝑦
0. 2
𝑝𝑝(𝑦𝑦1, 𝑦𝑦*)

0.3 Posterior
0.2

PDF
Posterior 1
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 Weights and biases, Prior
4
Mean y1 0 2 4 -4 -2 0 2 e.g., 𝜃𝜃𝑘𝑘 ~𝒩𝒩 𝜇𝜇𝑘𝑘 , 𝜎𝜎𝑘𝑘2
y*
Variance Observation: y1 = 1 𝜃𝜃
1 Bayesian inference
2 Uncertainty propagation

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 11
 More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout

Forward
pass 1
x y
𝑦𝑦�1 = f(x; θ1)

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 12
 More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout

Forward
pass 1
x y
𝑦𝑦�1 = f(x; θ1)
…

𝑦𝑦�𝑀𝑀 = f(x; θM)

x y
Forward
pass M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 13
 More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout

Forward
pass 1
x y
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1)
…

𝑦𝑦�𝑀𝑀 = f(x; θM)

x y
Forward
pass M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 14
 More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout Method 4: Neural network ensemble

Forward Member 1
pass 1 𝜇𝜇 𝜇𝜇̂ = f (x; θ )
1 1 1,1
x y x
𝜎𝜎 𝜎𝜎�2 = f2(x; θ1,2)
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1) 𝑦𝑦�1 ~𝒩𝒩 𝜇𝜇̂ 1 , 𝜎𝜎�12
…

𝑦𝑦�𝑀𝑀 = f(x; θM)

x y
Forward
pass M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 15
 More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout Method 4: Neural network ensemble

Forward Member 1
pass 1 𝜇𝜇 𝜇𝜇̂ = f (x; θ )
1 1 1,1
x y x
𝜎𝜎 𝜎𝜎�2 = f2(x; θ1,2)
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1) 𝑦𝑦�1 ~𝒩𝒩 𝜇𝜇̂ 1 , 𝜎𝜎�12

…
…

2
𝑦𝑦�𝑀𝑀 ~𝒩𝒩 𝜇𝜇̂ 𝑀𝑀 , 𝜎𝜎�𝑀𝑀
𝑦𝑦�𝑀𝑀 = f(x; θM)
𝜇𝜇 𝜇𝜇̂ 𝑀𝑀 = f1(x; θM,1)
x y x
Forward 𝜎𝜎 𝜎𝜎�𝑀𝑀 = f2(x; θM,2)
pass M Member M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 16
 More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout Method 4: Neural network ensemble

Forward Member 1
pass 1 𝜇𝜇 𝜇𝜇̂ = f (x; θ )
1 1 1,1
x y x
𝜎𝜎 𝜎𝜎�2 = f2(x; θ1,2) 𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1) 𝑦𝑦�1 ~𝒩𝒩 𝜇𝜇̂ 1 , 𝜎𝜎�12

…
…

2
𝑦𝑦�𝑀𝑀 ~𝒩𝒩 𝜇𝜇̂ 𝑀𝑀 , 𝜎𝜎�𝑀𝑀 Gaussian
𝑦𝑦�𝑀𝑀 = f(x; θM) mixture
𝜇𝜇 𝜇𝜇̂ 𝑀𝑀 = f1(x; θM,1)
x y x
Forward 𝜎𝜎 𝜎𝜎�𝑀𝑀 = f2(x; θM,2)
pass M Member M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 17
 More on UQ of Machine Learning Models

Method 5: Deep neural network with GPR

(DNN-GPR)

Posterior
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2
x y
Mean
Variance

Prior (zero mean)

𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊
𝐖𝐖1 𝐖𝐖2
Kernel on hidden
features ℎnn 𝐱𝐱; 𝛉𝛉
Hidden layers
without spectral Output layer with GPR
normalization (SN)

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 18
 More on UQ of Machine Learning Models

Method 5: Deep neural network with GPR Method 6: Spectral-normalized Neural Gaussian
(DNN-GPR) Process (SNGP)

Posterior Posterior
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2
x y x y
Mean Mean
Variance Variance

Prior (zero mean) Prior (zero mean)

𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊
𝐖𝐖1 𝐖𝐖2 𝛾𝛾 � 𝐖𝐖1 𝛾𝛾 � 𝐖𝐖2
Kernel on hidden Kernel on hidden
features ℎnn 𝐱𝐱; 𝛉𝛉 𝐖𝐖1 2 𝐖𝐖2 2 features ℎnn 𝐱𝐱; 𝛉𝛉
Hidden layers
without spectral Output layer with GPR Hidden layers with Output layer with GPR
normalization (SN) SN

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &

Informatics Laboratory 19
 How to Quantify Predictive Uncertainty?

Spectral-normalized Neural Gaussian process (SNGP)

Distance-aware UQ

Class probability Predictive uncertainty

𝑋𝑋2

𝑋𝑋2

𝑋𝑋1 𝑋𝑋1
Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T. and Lakshminarayanan, B., 2020. Simple and principled uncertainty estimation
with deterministic deep learning via distance awareness. NIPS, 33, pp.7498-7512.

REIL Reliability Engineering &

Informatics Laboratory 20
𝑋𝑋2 Predictive uncertainty

Low
uncertainty

REIL Reliability Engineering &

Informatics 𝑋𝑋
Laboratory
1 21
𝑋𝑋2 Predictive uncertainty

Low uncertainty

REIL Reliability Engineering &

Informatics 𝑋𝑋
Laboratory
1 22
𝑋𝑋2 Predictive uncertainty

Low uncertainty

REIL Reliability Engineering &

Informatics 𝑋𝑋
Laboratory
1 23
𝑋𝑋2 Predictive uncertainty

High uncertainty

REIL Reliability Engineering &

Informatics 𝑋𝑋
Laboratory
1 24
𝑋𝑋2 Predictive uncertainty

High uncertainty

REIL Reliability Engineering &

Informatics 𝑋𝑋
Laboratory
1 25