0% found this document useful (0 votes)
8 views25 pages

Lecture 11 - Probabilistic Machine Learning For Neural Networks

The document outlines Lecture 11 on Probabilistic Machine Learning for Neural Networks, focusing on techniques for predictive maintenance. It discusses various methods such as Gaussian Process Regression, Bayesian Neural Networks, Monte Carlo Dropout, and Neural Network Ensembles for uncertainty quantification in machine learning models. The lecture emphasizes the application of these techniques in bearing fault classification and predictive maintenance decision-making.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
8 views25 pages

Lecture 11 - Probabilistic Machine Learning For Neural Networks

The document outlines Lecture 11 on Probabilistic Machine Learning for Neural Networks, focusing on techniques for predictive maintenance. It discusses various methods such as Gaussian Process Regression, Bayesian Neural Networks, Monte Carlo Dropout, and Neural Network Ensembles for uncertainty quantification in machine learning models. The lecture emphasizes the application of these techniques in bearing fault classification and predictive maintenance decision-making.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 25

Lecture 11: Probabilistic Machine Learning for Neural

Networks
Lecture #11
Outline

• Machine Learning Pipeline for Predictive Maintenance

• Probabilistic Machine Learning Techniques


• Gaussian Process Regression
• Bayesian Neural Network
• Monte Carol Dropout
• Neural Network Ensemble
• Deep Neural Network with Gaussian Process Regression
• Spectral-normalized Neural Gaussian Process

2
Problem Definition

ML Pipeline for Predictive Maintenance

Physical Digital
domain domain
Sense

Bearing Vibration data

REIL Reliability Engineering &


Informatics Laboratory 3
Problem Definition

ML Pipeline for Predictive Maintenance

Physical Digital ML model


domain domain
Sense Process Inference
De-noised Fault
Bearing Vibration data data probability

REIL Reliability Engineering &


Informatics Laboratory 4
Problem Definition

ML Pipeline for Predictive Maintenance

Physical Digital ML model Physical


domain domain Decision domain
Sense Process Inference making
De-noised Fault
Bearing Vibration data data probability Maintenance

Act and improve

An example focus is on
bearing fault classification.

REIL Reliability Engineering &


Informatics Laboratory 5
How to Quantify Predictive Uncertainty?

Looking at a 2D binary classification problem

Training data

+
𝑋𝑋2

Test data (OoD)


𝑋𝑋1

REIL Reliability Engineering &


Informatics Laboratory 6
How to Quantify Predictive Uncertainty?

Spectral-normalized Neural Gaussian process (SNGP)


Distance-aware UQ

Class probability Predictive uncertainty


𝑋𝑋2

𝑋𝑋2

𝑋𝑋1 𝑋𝑋1
Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T. and Lakshminarayanan, B., 2020. Simple and principled uncertainty estimation
with deterministic deep learning via distance awareness. NIPS, 33, pp.7498-7512.

REIL Reliability Engineering &


Informatics Laboratory 7
More on UQ of Machine Learning Models

Method 1: Gaussian process regression

Prior (zero mean) Conditionals of MVN


𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 Shape of
Kernel on input conditional PDF
features 𝐱𝐱 𝑝𝑝 𝑦𝑦∗ | 𝑦𝑦1 = 1
0.
𝑝𝑝(𝑦𝑦1, 𝑦𝑦*)

0.3
Posterior 0.2
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 4
Mean y1 0 2 4 -4 -2 0 2
y*
Variance Observation: y1 = 1

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 8
More on UQ of Machine Learning Models

Method 1: Gaussian process regression A regular


neural network
Prior (zero mean) Conditionals of MVN
𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 Shape of
Kernel on input conditional PDF x y
features 𝐱𝐱 𝑝𝑝 𝑦𝑦∗ | 𝑦𝑦1 = 1
0.
𝑝𝑝(𝑦𝑦1, 𝑦𝑦*)

0.3
Posterior 0.2
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 4 Weights and biases
Mean y1 0 2 4 -4 -2 0 2
y*
Variance Observation: y1 = 1

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 9
More on UQ of Machine Learning Models

Method 1: Gaussian process regression

Prior (zero mean) Conditionals of MVN


𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 Shape of
Kernel on input conditional PDF x y
features 𝐱𝐱 𝑝𝑝 𝑦𝑦∗ | 𝑦𝑦1 = 1
0.
𝑝𝑝(𝑦𝑦1, 𝑦𝑦*)

0.3 Posterior
0.2

PDF
Posterior 1
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 Weights and biases, Prior
4
Mean y1 0 2 4 -4 -2 0 2 e.g., 𝜃𝜃𝑘𝑘 ~𝒩𝒩 𝜇𝜇𝑘𝑘 , 𝜎𝜎𝑘𝑘2
y*
Variance Observation: y1 = 1 𝜃𝜃
1 Bayesian inference

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 10
More on UQ of Machine Learning Models

Method 1: Gaussian process regression Method 2: Bayesian neural network


(MCMC or VI)
Conditionals of MVN Predictive
Prior (zero mean) distribution

PDF
𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 Shape of
Kernel on input conditional PDF x y
features 𝐱𝐱 𝑝𝑝 𝑦𝑦∗ | 𝑦𝑦1 = 1 𝑦𝑦
0. 2
𝑝𝑝(𝑦𝑦1, 𝑦𝑦*)

0.3 Posterior
0.2

PDF
Posterior 1
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 Weights and biases, Prior
4
Mean y1 0 2 4 -4 -2 0 2 e.g., 𝜃𝜃𝑘𝑘 ~𝒩𝒩 𝜇𝜇𝑘𝑘 , 𝜎𝜎𝑘𝑘2
y*
Variance Observation: y1 = 1 𝜃𝜃
1 Bayesian inference
2 Uncertainty propagation

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 11
More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout

Forward
pass 1
x y
𝑦𝑦�1 = f(x; θ1)

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 12
More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout

Forward
pass 1
x y
𝑦𝑦�1 = f(x; θ1)

𝑦𝑦�𝑀𝑀 = f(x; θM)

x y
Forward
pass M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 13
More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout

Forward
pass 1
x y
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1)

𝑦𝑦�𝑀𝑀 = f(x; θM)

x y
Forward
pass M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 14
More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout Method 4: Neural network ensemble

Forward Member 1
pass 1 𝜇𝜇 𝜇𝜇̂ = f (x; θ )
1 1 1,1
x y x
𝜎𝜎 𝜎𝜎�2 = f2(x; θ1,2)
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1) 𝑦𝑦�1 ~𝒩𝒩 𝜇𝜇̂ 1 , 𝜎𝜎�12

𝑦𝑦�𝑀𝑀 = f(x; θM)

x y
Forward
pass M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 15
More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout Method 4: Neural network ensemble

Forward Member 1
pass 1 𝜇𝜇 𝜇𝜇̂ = f (x; θ )
1 1 1,1
x y x
𝜎𝜎 𝜎𝜎�2 = f2(x; θ1,2)
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1) 𝑦𝑦�1 ~𝒩𝒩 𝜇𝜇̂ 1 , 𝜎𝜎�12


2
𝑦𝑦�𝑀𝑀 ~𝒩𝒩 𝜇𝜇̂ 𝑀𝑀 , 𝜎𝜎�𝑀𝑀
𝑦𝑦�𝑀𝑀 = f(x; θM)
𝜇𝜇 𝜇𝜇̂ 𝑀𝑀 = f1(x; θM,1)
x y x
Forward 𝜎𝜎 𝜎𝜎�𝑀𝑀 = f2(x; θM,2)
pass M Member M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 16
More on UQ of Machine Learning Models

Method 3: Monte Carlo dropout Method 4: Neural network ensemble

Forward Member 1
pass 1 𝜇𝜇 𝜇𝜇̂ = f (x; θ )
1 1 1,1
x y x
𝜎𝜎 𝜎𝜎�2 = f2(x; θ1,2) 𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1) 𝑦𝑦�1 ~𝒩𝒩 𝜇𝜇̂ 1 , 𝜎𝜎�12


2
𝑦𝑦�𝑀𝑀 ~𝒩𝒩 𝜇𝜇̂ 𝑀𝑀 , 𝜎𝜎�𝑀𝑀 Gaussian
𝑦𝑦�𝑀𝑀 = f(x; θM) mixture
𝜇𝜇 𝜇𝜇̂ 𝑀𝑀 = f1(x; θM,1)
x y x
Forward 𝜎𝜎 𝜎𝜎�𝑀𝑀 = f2(x; θM,2)
pass M Member M

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 17
More on UQ of Machine Learning Models

Method 5: Deep neural network with GPR


(DNN-GPR)

Posterior
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2
x y
Mean
Variance

Prior (zero mean)


𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊
𝐖𝐖1 𝐖𝐖2
Kernel on hidden
features ℎnn 𝐱𝐱; 𝛉𝛉
Hidden layers
without spectral Output layer with GPR
normalization (SN)

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 18
More on UQ of Machine Learning Models

Method 5: Deep neural network with GPR Method 6: Spectral-normalized Neural Gaussian
(DNN-GPR) Process (SNGP)

Posterior Posterior
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2
x y x y
Mean Mean
Variance Variance

Prior (zero mean) Prior (zero mean)


𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊
𝐖𝐖1 𝐖𝐖2 𝛾𝛾 � 𝐖𝐖1 𝛾𝛾 � 𝐖𝐖2
Kernel on hidden Kernel on hidden
features ℎnn 𝐱𝐱; 𝛉𝛉 𝐖𝐖1 2 𝐖𝐖2 2 features ℎnn 𝐱𝐱; 𝛉𝛉
Hidden layers
without spectral Output layer with GPR Hidden layers with Output layer with GPR
normalization (SN) SN

Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..

REIL Reliability Engineering &


Informatics Laboratory 19
How to Quantify Predictive Uncertainty?

Spectral-normalized Neural Gaussian process (SNGP)


Distance-aware UQ

Class probability Predictive uncertainty


𝑋𝑋2

𝑋𝑋2

𝑋𝑋1 𝑋𝑋1
Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T. and Lakshminarayanan, B., 2020. Simple and principled uncertainty estimation
with deterministic deep learning via distance awareness. NIPS, 33, pp.7498-7512.

REIL Reliability Engineering &


Informatics Laboratory 20
𝑋𝑋2 Predictive uncertainty

Low
uncertainty

REIL Reliability Engineering &


Informatics 𝑋𝑋
Laboratory
1 21
𝑋𝑋2 Predictive uncertainty

Low uncertainty

REIL Reliability Engineering &


Informatics 𝑋𝑋
Laboratory
1 22
𝑋𝑋2 Predictive uncertainty

Low uncertainty

REIL Reliability Engineering &


Informatics 𝑋𝑋
Laboratory
1 23
𝑋𝑋2 Predictive uncertainty

High uncertainty

REIL Reliability Engineering &


Informatics 𝑋𝑋
Laboratory
1 24
𝑋𝑋2 Predictive uncertainty

High uncertainty

REIL Reliability Engineering &


Informatics 𝑋𝑋
Laboratory
1 25

You might also like