Lecture 11 - Probabilistic Machine Learning For Neural Networks
Lecture 11 - Probabilistic Machine Learning For Neural Networks
Networks
Lecture #11
Outline
2
Problem Definition
Physical Digital
domain domain
Sense
An example focus is on
bearing fault classification.
Training data
+
𝑋𝑋2
𝑋𝑋1
𝑋𝑋2
𝑋𝑋1 𝑋𝑋1
Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T. and Lakshminarayanan, B., 2020. Simple and principled uncertainty estimation
with deterministic deep learning via distance awareness. NIPS, 33, pp.7498-7512.
0.3
Posterior 0.2
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 4
Mean y1 0 2 4 -4 -2 0 2
y*
Variance Observation: y1 = 1
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
0.3
Posterior 0.2
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 4 Weights and biases
Mean y1 0 2 4 -4 -2 0 2
y*
Variance Observation: y1 = 1
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
0.3 Posterior
0.2
PDF
Posterior 1
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 Weights and biases, Prior
4
Mean y1 0 2 4 -4 -2 0 2 e.g., 𝜃𝜃𝑘𝑘 ~𝒩𝒩 𝜇𝜇𝑘𝑘 , 𝜎𝜎𝑘𝑘2
y*
Variance Observation: y1 = 1 𝜃𝜃
1 Bayesian inference
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
PDF
𝐲𝐲, 𝑦𝑦∗ T ~𝒩𝒩 𝟎𝟎, 𝐊𝐊 Shape of
Kernel on input conditional PDF x y
features 𝐱𝐱 𝑝𝑝 𝑦𝑦∗ | 𝑦𝑦1 = 1 𝑦𝑦
0. 2
𝑝𝑝(𝑦𝑦1, 𝑦𝑦*)
0.3 Posterior
0.2
PDF
Posterior 1
0
1
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 -4 -2 Weights and biases, Prior
4
Mean y1 0 2 4 -4 -2 0 2 e.g., 𝜃𝜃𝑘𝑘 ~𝒩𝒩 𝜇𝜇𝑘𝑘 , 𝜎𝜎𝑘𝑘2
y*
Variance Observation: y1 = 1 𝜃𝜃
1 Bayesian inference
2 Uncertainty propagation
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
Forward
pass 1
x y
𝑦𝑦�1 = f(x; θ1)
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
Forward
pass 1
x y
𝑦𝑦�1 = f(x; θ1)
…
x y
Forward
pass M
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
Forward
pass 1
x y
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1)
…
x y
Forward
pass M
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
Forward Member 1
pass 1 𝜇𝜇 𝜇𝜇̂ = f (x; θ )
1 1 1,1
x y x
𝜎𝜎 𝜎𝜎�2 = f2(x; θ1,2)
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1) 𝑦𝑦�1 ~𝒩𝒩 𝜇𝜇̂ 1 , 𝜎𝜎�12
…
x y
Forward
pass M
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
Forward Member 1
pass 1 𝜇𝜇 𝜇𝜇̂ = f (x; θ )
1 1 1,1
x y x
𝜎𝜎 𝜎𝜎�2 = f2(x; θ1,2)
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1) 𝑦𝑦�1 ~𝒩𝒩 𝜇𝜇̂ 1 , 𝜎𝜎�12
…
…
2
𝑦𝑦�𝑀𝑀 ~𝒩𝒩 𝜇𝜇̂ 𝑀𝑀 , 𝜎𝜎�𝑀𝑀
𝑦𝑦�𝑀𝑀 = f(x; θM)
𝜇𝜇 𝜇𝜇̂ 𝑀𝑀 = f1(x; θM,1)
x y x
Forward 𝜎𝜎 𝜎𝜎�𝑀𝑀 = f2(x; θM,2)
pass M Member M
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
Forward Member 1
pass 1 𝜇𝜇 𝜇𝜇̂ = f (x; θ )
1 1 1,1
x y x
𝜎𝜎 𝜎𝜎�2 = f2(x; θ1,2) 𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦~𝒩𝒩
� 𝜇𝜇,̂ 𝜎𝜎� 2
𝑦𝑦�1 = f(x; θ1) 𝑦𝑦�1 ~𝒩𝒩 𝜇𝜇̂ 1 , 𝜎𝜎�12
…
…
2
𝑦𝑦�𝑀𝑀 ~𝒩𝒩 𝜇𝜇̂ 𝑀𝑀 , 𝜎𝜎�𝑀𝑀 Gaussian
𝑦𝑦�𝑀𝑀 = f(x; θM) mixture
𝜇𝜇 𝜇𝜇̂ 𝑀𝑀 = f1(x; θM,1)
x y x
Forward 𝜎𝜎 𝜎𝜎�𝑀𝑀 = f2(x; θM,2)
pass M Member M
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
Posterior
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2
x y
Mean
Variance
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
Method 5: Deep neural network with GPR Method 6: Spectral-normalized Neural Gaussian
(DNN-GPR) Process (SNGP)
Posterior Posterior
𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2 𝑦𝑦�∗ ~𝒩𝒩 𝜇𝜇,̂ 𝜎𝜎� 2
x y x y
Mean Mean
Variance Variance
Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X. and Hu, C., 2023. Uncertainty quantification in
machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205,
p.110796..
𝑋𝑋2
𝑋𝑋1 𝑋𝑋1
Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T. and Lakshminarayanan, B., 2020. Simple and principled uncertainty estimation
with deterministic deep learning via distance awareness. NIPS, 33, pp.7498-7512.
Low
uncertainty
Low uncertainty
Low uncertainty
High uncertainty
High uncertainty