ECE 57000 – Artificial Intelligence

Variational Autoencoders

Murat Kocaoglu
 sha1_base64="/BwqdIsDrPG+401H09YaT2MC0Gc=">AAAJ3HicjVZfjxs1EN+Wf5fw7wqPvBiSk9oqF5K2Ah44qacDqUIghVOvrZRNT16vk1jx2pbt7eXO+I03xCufg0+DxBN8FMabzWV3k0hssspk5jc/e8bjsRPFmbGDwd937r719jvvvnfQar//wYcffXx475MXRuaa0AsiudSvEmwoZ4JeWGY5faU0xVnC6ctkcRbsL99QbZgUz+21opMMzwSbMoItqC4Pf7gfZ9jOk6lb+l/W4o1/EBuWodjSpXXfL9UUZ74KTC9jO6cW3694PLg87Az6g+JB28KwFDpR+Ywu7x3cxKkkeUaFJRwbMx4OlJ04rC0jnPp2nBuqMFngGR2DKHBGzcQVQXt0BJoUTaWGV1hUaKseDmfGXGcJIMMkTdMWlPtsdp7dmtqxoFdEZhkWqQthP8N8JjULGBdjjRNG3Ebl23WytYERv9tQdxjndvrNxDGhcksFWYU5zTmyEoX1QynTlFh+DQImQMAIInOYBbGwyjWm9A1TpszZcpU0GAoCkJpCiNecOsUxE76IsNS78te756UwNjAe1MqkBlNaKmlYMPhxqZy4UUVbg3MK+asAfyz+1yBEask51tcV2NmtrjHxlE6Z2DFORb9h+W4PGCouz1QDfLpRNsYEAetFnaLUbdzPS1C9aphIqUJQLopqFX9efILo6zDC/PjxxHWGqPToPEIxp1Mbwxa2qPM41mw2t/2Gm6j4CWlL7sow+3lqJWnZ4qZepCZPCF4no6LPcm6Zlld+56ZqN7Vhg5oGAwCtlNw00EsDEm0EqBTxznXjkWHd49GZj3eicG5lhpV3pyD8dDq6RdVhpogd6zB997DIUUXTwIakldD4Vg4+D5tIMIV3TZklcumOj499CW6gCdOhw63A3Q26K7dKIgDXcLlB7gaupiCrw++aaXX4vYwWM77i22C2IiHVbMpGNnfmlOzN6VZIwVrjr2UK7S3fIHGWaGgcodA0W/YUttAghdkGmDlW1PQSYJlpmYt0B4ZgDu37aI9rSqF3Feep6RUzNb3gUMr9DA7K3pTZ3ro1MjELZIEJ+Fz7qKUkE/bLftFn0Alyq/XpoVTjqx5iQlCNINATOE2fkAzIOO8JmVI49QQcE9acODh4jkLHWpO2Asf/p9xm285Cbf7VzM80VnNGls3iWMBedd1F2LOosbbnfjycwJYOfQCS5c59Z9ht+heXiyQroKu7SBg1dBN34V2wOPgKcG0eu4mUC4sTA+dncUUQjEC8oIFk6Bz2SRtuK8Pm3WRbePGoP/yqP/j5Sefpt+W95SD6LPoiuh8No6+jp9GzaBRdRCT6M/or+if6t/W69Wvrt9bvK+jdO6XPp1Htaf3xH2wCoKQ=</latexit>
<latexit
sha1_base64="FmEZ/IzLG+URUhpnSAhw+Awm/xc=">AAAJtXicjVbNjts2EFbSv7X7k6Q59sLUXiANtK6dBEkPXSCL7SGHFnAX2SSAZSwoirIJUyRBUll7CT1AnybX9FH6NhnK8lqSbaCyBY9nvvnIGQ6HjBVnxg6H/925+8WXX339zVGn++133/9w7/6DH98amWtCL4nkUr+PsaGcCXppmeX0vdIUZzGn7+LFube/+0C1YVK8sStFpxmeCZYygi2oru73ogzbeZy6myIyLEPqKrJzavHjrf4XQA0Hw/JBu8KoEnpB9YyvHhzdRIkkeUaFJRwbMxkNlZ06rC0jnBbdKDdUYbLAMzoBUeCMmqkroynQMWgSlEoNr7Co1NY9HM6MWWUxIP0cTdvmlYdsdp7dmrqRoNdEZhkWifNRv8Z8JjXzGBdhjWNG3FZVdJtkGwMjxX5D02GS2/S3qWNC5ZYKsg4zzTmyEvmFQQnTlFi+AgETIGAEkTnMglhYvgZT8oEpU+VsuU4aDAUBSE0hxBWnTnHMRFFGWOld9Vu4N5UwMTAeFMG0AVNaKmmYNxSTSjl145q2AecU8lcD/ln+b0CI1JJzrFc12PmtrjXxhKZM7Bmnpt+y/HEADBWXZ6oFPtsqW2OCgPWiSVHptu4XFahZNUwkVCEoF0W1ih6VHy8WTRhhxeTZ1PVGqPLoPUURp6mNYG9a1HsWaTab20HLTdT8hLQVd22YwzyNkrRscdMsUpPHBG+SUdNnObdMy+ti76bqtrV+g5oWAwCtlNy00EsDEm0FqBQpnOtHY8P6J+PzItqLwrmVGVaFOwPhr7PxLaoJM2XsWPvpuydljmqaFtYnrYJGt7L3edJGgsm/G8oslkt3cnJSVOAWmjDtO9wa3N+i+3KnJDxwA5db5H7gegqyPvy+mdaHP8hoMeNrvi1mJxJSz6ZsZXNvTsnBnO6E5K0N/kam0MHy9RJnsYbG4QtNs2WosIUGKcwuwMyxoiaMgWWmZS6SPRiCObTv4wOuCYXeVR6UJixnakLvUMmDDM7JMGU23LRGJmaezDMBn+sed5Rkwv46KPsMOkVuvT4hSjS+DhETgmoEgZ7CafqcZEDGeShkQuHUE3BMWHPq4OA59h1rQ9rxHP+fcpdtNwuN+dczP9NYzRlZtotjAXvV9Rd+z6LW2l4Uk9EUtrTvA5Asd1H0Rv22f3m3iLMSGlm6tM6P6ruJuyyctzj4CnBtH7uxlAuLYwPnZ3lFEIxAvKCBZOgc9kkXbiuj9t1kV3j7dDB6MRj+/bz36vfq3nIU/BT8HDwORsHL4FXwOhgHlwEJ/gk+Bp+CfzsvO9NO0knX0Lt3Kp+HQePpyM/b0o+a</latexit>
<latexit
neural network d that accepts z.
z ⇠ p✓ (z)

(x|z) ⇠ Expfam(x|d✓ (z))

We model the data-generating process as follows:
Deep Latent Variable Models
Some latent variable Z generates observed variable X.

Data (x) is generated from some exponential family parameterized by a deep

Typically, z is assumed to be distributed with some known prior, such as Gaussian.
Exponential Family of Distributions
Any probability density that can be written as follows:

: Sufficient statistic
: Natural/canonical parameters
: Partition function
: Log-partition function

Exponential family is larger than you might think:

- Exponential distribution
- Gaussian
- Bernoulli
Importance of Exponential Family
Pitman–Koopman–Darmois theorem:
A distribution belongs to the exponential family if and only if n IID samples
from it can be characterized by a finite dimensional sufficient statistic that
does not grow with the number of samples n.

Recall definition of sufficient statistic:

A function T(X) of data is called a sufficient statistic for underlying parameter

if the conditional probability distribution of the data given T is not a
function of .

In other words, likelihood’s dependence on parameter is only through T(x).

 sha1_base64="7M6vyjW1oApbBOKmoBEb09XMzDY=">AAAJvXicjVbNjts2EFbSv7X7k0177IWtvcAmlV0rCdoeumgW20MOLeAuskkAy1lQFG0TpkiCpLLeEHqIPk2v7WP0bTqU5bUk20BlCx7PfPORMxwOmSjOjB2N/r13/4MPP/r4k6NO99PPPv/iwfHDL18ZmWtCr4jkUr9JsKGcCXplmeX0jdIUZwmnr5Plhbe/fke1YVK8tLeKTjM8F2zGCLaguj7+Tp2uHqEzpN6uTqOBevTWRYNVEaLY0pV1qFjFTKDYjcIoLq6Pe6PhqHzQrhBVQi+onvH1w6P3cSpJnlFhCcfGTKKRslOHtWWE06Ib54YqTJZ4TicgCpxRM3VlVAU6AU2KZlLDKywqtXUPhzNjbrMEkBm2C9O2eeUhm11kd6ZuLOgNkVmGRepiu6AvMJ9LzTzGxVjjhBG3VRXdJtnGwEix39B0mOR29tPUMaFySwVZhznLObIS+QVCKdOUWH4LAiZAwAgiC5gFsbCMDab0HVOmytlqnTQYCgKQmkKIt5w6xTETRRlhpXfVb+FeVsLEwHhQDNMGTGmppGHeUEwq5dSNa9oGnFPIXw34W/m/ASFSS86xvq3BLu50rYmndMbEnnFq+i3LrwfAUHF5plrg862yNSYIWC+bFJVu635ZgZpVw0RKFYJyUVSr+Jvy48WiCSOsmDydul6EKo/eExRzOrMx7FGLek9jzeYLO2y5iZqfkLbirg1zmKdRkpYt3zeL1OQJwZtk1PRZzi3T8qbYu6m6ba3foKbFAEArJTct9MqARFsBKkUK5/rx2LD+YHxRxHtROLcyw6pw5yD8fj6+QzVhpowdaz9997jMUU3TwvqkVdD4TvY+j9tIMPl3Q5klcuUGg0FRgVtowrTvcGtwf4vuy52S8MANXG6R+4HrKcj68PtmWh/+IKPFjK/5tpidSEg9m7KVzb05JQdzuhOStzb4G5lCB8vXS5wlGhqHLzTNVqHCFhqkMLsAs8CKmjABlrmWuUj3YAjm0L5PDrimFHpXeWCasJypCb1DJQ8zanE4YzbctEYm5p7MMwGf6550lGTCfj8s+wyctm69PiFKNb4JEROCagSBnsFp+oxkQMZ5KGRK4dQTcExYc+bg4DnxHWtD2vEc/59yl203C4351zM/11gtGFm1i2MJe9X1l37PotbaXhaTaApb2vcBSJa7LHpRv+3vjUmSldD1jcOP6ruJuyqctzj4CnBtH7uJlEuLEwPnZ3lFEIxAvKCBZOgc9kkXbitR+26yK7x6Mox+GI7+eNZ7/nN1bzkKvg6+DU6DKPgxeB68CMbBVUCCP4O/gr+Dfzq/dGiHd8Qaev9e5fNV0Hg6N/8BTriP5w==</latexit>
<latexit
Sufficient statistic: T(X)=X
p(x) = px (1  p)1−x , x 2 {0, 1}
Why is Bernoulli in Exponential Family?
 sha1_base64="/BwqdIsDrPG+401H09YaT2MC0Gc=">AAAJ3HicjVZfjxs1EN+Wf5fw7wqPvBiSk9oqF5K2Ah44qacDqUIghVOvrZRNT16vk1jx2pbt7eXO+I03xCufg0+DxBN8FMabzWV3k0hssspk5jc/e8bjsRPFmbGDwd937r719jvvvnfQar//wYcffXx475MXRuaa0AsiudSvEmwoZ4JeWGY5faU0xVnC6ctkcRbsL99QbZgUz+21opMMzwSbMoItqC4Pf7gfZ9jOk6lb+l/W4o1/EBuWodjSpXXfL9UUZ74KTC9jO6cW3694PLg87Az6g+JB28KwFDpR+Ywu7x3cxKkkeUaFJRwbMx4OlJ04rC0jnPp2nBuqMFngGR2DKHBGzcQVQXt0BJoUTaWGV1hUaKseDmfGXGcJIMMkTdMWlPtsdp7dmtqxoFdEZhkWqQthP8N8JjULGBdjjRNG3Ebl23WytYERv9tQdxjndvrNxDGhcksFWYU5zTmyEoX1QynTlFh+DQImQMAIInOYBbGwyjWm9A1TpszZcpU0GAoCkJpCiNecOsUxE76IsNS78te756UwNjAe1MqkBlNaKmlYMPhxqZy4UUVbg3MK+asAfyz+1yBEask51tcV2NmtrjHxlE6Z2DFORb9h+W4PGCouz1QDfLpRNsYEAetFnaLUbdzPS1C9aphIqUJQLopqFX9efILo6zDC/PjxxHWGqPToPEIxp1Mbwxa2qPM41mw2t/2Gm6j4CWlL7sow+3lqJWnZ4qZepCZPCF4no6LPcm6Zlld+56ZqN7Vhg5oGAwCtlNw00EsDEm0EqBTxznXjkWHd49GZj3eicG5lhpV3pyD8dDq6RdVhpogd6zB997DIUUXTwIakldD4Vg4+D5tIMIV3TZklcumOj499CW6gCdOhw63A3Q26K7dKIgDXcLlB7gaupiCrw++aaXX4vYwWM77i22C2IiHVbMpGNnfmlOzN6VZIwVrjr2UK7S3fIHGWaGgcodA0W/YUttAghdkGmDlW1PQSYJlpmYt0B4ZgDu37aI9rSqF3Feep6RUzNb3gUMr9DA7K3pTZ3ro1MjELZIEJ+Fz7qKUkE/bLftFn0Alyq/XpoVTjqx5iQlCNINATOE2fkAzIOO8JmVI49QQcE9acODh4jkLHWpO2Asf/p9xm285Cbf7VzM80VnNGls3iWMBedd1F2LOosbbnfjycwJYOfQCS5c59Z9ht+heXiyQroKu7SBg1dBN34V2wOPgKcG0eu4mUC4sTA+dncUUQjEC8oIFk6Bz2SRtuK8Pm3WRbePGoP/yqP/j5Sefpt+W95SD6LPoiuh8No6+jp9GzaBRdRCT6M/or+if6t/W69Wvrt9bvK+jdO6XPp1Htaf3xH2wCoKQ=</latexit>
<latexit
Posterior calculations
sha1_base64="FmEZ/IzLG+URUhpnSAhw+Awm/xc=">AAAJtXicjVbNjts2EFbSv7X7k6Q59sLUXiANtK6dBEkPXSCL7SGHFnAX2SSAZSwoirIJUyRBUll7CT1AnybX9FH6NhnK8lqSbaCyBY9nvvnIGQ6HjBVnxg6H/925+8WXX339zVGn++133/9w7/6DH98amWtCL4nkUr+PsaGcCXppmeX0vdIUZzGn7+LFube/+0C1YVK8sStFpxmeCZYygi2oru73ogzbeZy6myIyLEPqKrJzavHjrf4XQA0Hw/JBu8KoEnpB9YyvHhzdRIkkeUaFJRwbMxkNlZ06rC0jnBbdKDdUYbLAMzoBUeCMmqkroynQMWgSlEoNr7Co1NY9HM6MWWUxIP0cTdvmlYdsdp7dmrqRoNdEZhkWifNRv8Z8JjXzGBdhjWNG3FZVdJtkGwMjxX5D02GS2/S3qWNC5ZYKsg4zzTmyEvmFQQnTlFi+AgETIGAEkTnMglhYvgZT8oEpU+VsuU4aDAUBSE0hxBWnTnHMRFFGWOld9Vu4N5UwMTAeFMG0AVNaKmmYNxSTSjl145q2AecU8lcD/ln+b0CI1JJzrFc12PmtrjXxhKZM7Bmnpt+y/HEADBWXZ6oFPtsqW2OCgPWiSVHptu4XFahZNUwkVCEoF0W1ih6VHy8WTRhhxeTZ1PVGqPLoPUURp6mNYG9a1HsWaTab20HLTdT8hLQVd22YwzyNkrRscdMsUpPHBG+SUdNnObdMy+ti76bqtrV+g5oWAwCtlNy00EsDEm0FqBQpnOtHY8P6J+PzItqLwrmVGVaFOwPhr7PxLaoJM2XsWPvpuydljmqaFtYnrYJGt7L3edJGgsm/G8oslkt3cnJSVOAWmjDtO9wa3N+i+3KnJDxwA5db5H7gegqyPvy+mdaHP8hoMeNrvi1mJxJSz6ZsZXNvTsnBnO6E5K0N/kam0MHy9RJnsYbG4QtNs2WosIUGKcwuwMyxoiaMgWWmZS6SPRiCObTv4wOuCYXeVR6UJixnakLvUMmDDM7JMGU23LRGJmaezDMBn+sed5Rkwv46KPsMOkVuvT4hSjS+DhETgmoEgZ7CafqcZEDGeShkQuHUE3BMWHPq4OA59h1rQ9rxHP+fcpdtNwuN+dczP9NYzRlZtotjAXvV9Rd+z6LW2l4Uk9EUtrTvA5Asd1H0Rv22f3m3iLMSGlm6tM6P6ruJuyyctzj4CnBtH7uxlAuLYwPnZ3lFEIxAvKCBZOgc9kkXbiuj9t1kV3j7dDB6MRj+/bz36vfq3nIU/BT8HDwORsHL4FXwOhgHlwEJ/gk+Bp+CfzsvO9NO0knX0Lt3Kp+HQePpyM/b0o+a</latexit>
<latexit
neural network d that accepts z.
z ⇠ p✓ (z)

perform approximate posterior inference.

(x|z) ⇠ Expfam(x|d✓ (z))
We model the data-generating process as follows:
Some latent variable Z generates observed variable X.
Back to Deep Latent Variable Models

inference. Idea is to use amortized inference, which trains another model to

Data (x) is generated from some exponential family parameterized by a deep

are intractable so we will resort to approximate

Typically, z is assumed to be distributed with some known prior, such as Gaussian.
 sha1_base64="rsHC/jkI5KldrBFbEjz2M9EGEkk=">AAAJ5nicjVbNbhs3EN4k/bHUP6c99sJWMpAEsiolQdpDDcRwDzm0gGrYSQCtYHC5lESISxIkN5bN8hV6K3rtc/RNeuu1fYoOVytrdyUBXWmh0cw3HznD4ZCJ4szYweCve/cfvPf+Bx8etNofffzJp58dPvz8tZG5JvSSSC712wQbypmgl5ZZTt8qTXGWcPomWZwF+5t3VBsmxYW9UXSS4ZlgU0awBdXV4YW6iu2cWvwozrCdJ1N363trcenR45MdgMdbuqX/pWK+OuwM+oPiQdvCsBQ6UfmMrh4e3MapJHlGhSUcGzMeDpSdOKwtI5z6dpwbqjBZ4BkdgyhwRs3EFeF7dASaFE2lhldYVGirHg5nxtxkCSDDHE3TFpT7bHae3ZnasaDXRGYZFqkLCXiF+UxqFjAuxhonjLiNyrfrZGsDI363oe4wzu30u4ljQuWWCrIKc5pzZCUKK4lSpimx/AYETICAEUTmMAtiYb1rTOk7pkyZs+UqaTAUBCA1hRBvOHWKYyZ8EWGpd+WvdxelMDYwHlTNpAZTWippWDD4camcuFFFW4NzCvmrAH8s/tcgRGrJOdY3FdjZna4x8ZROmdgxTkW/YflhDxgqLs9UA3y6UTbGBAHrRZ2i1G3cz0tQvWqYSKlCUC6KahV/VXyC6Oswwvz42cR1hqj06DxFMadTG8NmtqjzLNZsNrf9hpuo+AlpS+7KMPt5aiVp2eK2XqQmTwheJ6Oiz3JumZbXfuemaje1YYOaBgMArZTcNNBLAxJtBKgU8c5145Fh3ePRmY93onBuZYaVd6cg/HQ6ukPVYaaIHeswffekyFFF08CGpJXQ+E4OPk+aSDCFd02ZJXLpjo+PfQluoAnTocOtwN0Nuiu3SiIA13C5Qe4GrqYgq8Pvmml1+L2MFjO+4ttgtiIh1WzKRjZ35pTszelWSMFa469lCu0t3yBxlmhoHKHQNFv2FLbQIIXZBpg5VtT0EmCZaZmLdAeGYA7t+2iPa0qhdxUnq+kVMzW94FDK/QyOzN6U2d66NTIxC2SBCfhc+6ilJBP2m37RZ9AJcqv16aFU4+seYkJQjSDQEzhNn5MMyDjvCZlSOPUEHBPWnDg4eI5Cx1qTtgLH/6fcZtvOQm3+1czPNFZzRpbN4ljAXnXdRdizqLG25348nMCWDn0AkuXOfWfYbfoXd4skK6CxpUvrwqihm7hL74LFwVeAa/PYTaRcWJwYOD+LK4JgBOIFDSRD57BP2nBbGTbvJtvC66f94Yv+4OfnnZffl/eWg+jL6OvoUTSMvo1eRq+iUXQZkejP6O/on+jf1rz1a+u31u8r6P17pc8XUe1p/fEfYfCkpA==</latexit>
<latexit
Usually Gaussian
VAE Basics

to the data distribution

p✓ (z, x) = p✓ (z)p✓ (x|z)

Ex.:
sha1_base64="EU+5HhPK7XYC4smc1ayq4lg7mC8=">AAAKEnicjVZfbyM1EN8c/5pwQA8EL7wYkkrtKQ1JDwEPVLrSe7gHkEJ1vTspm4u8Xiex4rUt23tNa63Eh+DT8IZ45QvwBfgcjDebZneTSGwaZTrzm589s+MZR4ozY/v9fxoP3nn3vfc/OGi2Pnz40cefHD769KWRqSb0mkgu9esIG8qZoNeWWU5fK01xEnH6Klpcevurt1QbJsULe6voOMEzwaaMYAuqyeFv4ailJi60c2pxdhxGksfmNoEft8xQmLAYlXV32Qk6R6HSMp64+HyQvXHPAGbp0rqfqM7Q8XISF26GzRJ8HO8mB6KTk1Y4bk0O2/1eP3/QtjAohHZQPMPJo4O7MJYkTaiwhGNjRoO+smOHtWWE06wVpoYqTBZ4RkcgCpxQM3Z5qjJ0BJoYTaWGr7Ao15Y9HE6M3yIgE2znpm7zyn02O0/uTa1Q0BsikwSLOE/Ac8xnUjOPcSHWOGLEbVRZq0q2NjCS7TZUHUapnf4wdkyo1FJBVmFOU46sRP6to5hpSiy/BQETIGAEkTnsgliojQpT/JYpU+RsuUoaLAUBSE0hxFtOneKYiSyPsNC74jdzLwphZGA9qLBxBQZ1o6Rh3pCNCuXYDUvaCpxTyF8J+HP+fwVCpJacY31bgl3e62obj+mUiR3rlPQblmd7wFBxaaJq4IuNsrYmCFgvqhSFbuN+VYCqVcNETBWCclFUq/Cr/OPFrAojLBs9Gbv2ABUe7TMUcjq1IRx8i9pPQs1mc9uruYmSn5C24C4ts5+nUpKWLe6qRWrSiOB1Mkr6JOWWaXmT7TxUrbrWH1BTYwCglZKbGnppQKK1AJUimXOdcGhY53R4mYU7UTi1MsEqcxcg/HIxvEdVYSaPHWu/ffc4z1FJU8P6pBXQ8F72Po/rSDD575oSWuPSnZ6eZgW4hiZM+w63Anc26I7cKgkPXMPlBrkbuNqCLC+/a6fl5fcyWsz4im+D2YqElLMpa9ncmVOyN6dbIXlrhb+SKbS3fL3EWaShcfhC02zZVdhCgxRmG2DmWFHTjYBlpmUq4h0Ygjm076M9rjGF3pVPYdPNd2q63qGQewmMzO6U2e66NTIx82SeCfhc66ipJBP2m17eZ2Aqu9X76aJY45suYkJQjSDQc5im35IEyDjvChlTmHoCxoQ15w4Gz5HvWGvSpuf4/5TbbNtZqOy/nPmZxmrOyLJeHAs4q66z8GcW1d7tVTYajOFI+z4AyXJXWXvQqft7YxQlOXR1M/Gr+m7irjPnLQ7+BLjWx24k5cLiyMD8zK8IghGIFzSQDJ3COfG3lUH9brItvDzrDb7r9X89az/9sbi3HARfBl8Hx8Eg+D54GjwPhsF1QIJ/Gw8bnze+aP7e/KP5Z/OvFfRBo/D5LKg8zb//A4PFscI=</latexit>
<latexit
p✓ (x | z) =
D
Y

d=1
Product of exponential family (Gaussian, Bernoulli)
https://danijar.com/building-variational-auto-encoders-in-tensorflow/

Ber(xd | σ(d✓ (z)))

Parameterize encoder/decoder to capture probability distribution between latent
code and input. Encoder maps data to latent code and decoder maps latent code
 sha1_base64="tkLaxogu7irewlcES1t1p9deLxw=">AAAKa3icjVZtb9s2EJa7t0R7S9dv2z5oswMkhePZ6bAN2II1yD70wzZ4QdMWsIyAomibMEVyJNXYIfhD+yP2H3aU5ViSbaByDF/unnvIO90dmUhGten337UeffDhRx9/cnAYfvrZ5198efT4q1da5AqTGyyYUG8SpAmjnNwYahh5IxVBWcLI62R+5e2v3xKlqeAvzVKScYamnE4oRgZUt0fv4lH4762N5Yy6kzgRLNXLDH7svYvijKZRVbdwp9EFqJGZYcTs3wBhZGJOol2Ok1tLuiDmdd6F+7UGL1Y+BaCQRCEjFEcZsSlFU3dSUmg6zdD7sJxGsaLTmTkN43F4e9Tu9/rFE20Lg1JoB+UzvH18cB+nAucZ4QYzpPVo0JdmbJEyFDPiwjjXRCI8R1MyAtFvVI9t8Q5cdAwaCFoo+HITFdqqh0WZ9psFpM+fbtq8cp/NzLIHUxhzcodFliGe2tjMyAvEpkJRj7ExUiih2G5ULqyTrQ0Uu92GusMoN5NfxpZymRvC8SrMSc4iIyJfTlFKFcGGLUFAGAgojvAMdoENFF2NKX1LpS5ztlglDZaCAIQiEOKSESsZotwVEZZ6W/46+7IURhrWg9Id12BSCSk09QY3KpVjO6xoa3BGIH8V4J/F/zUIFkowhtSyArt60DU2npIJ5TvWqeg3LH/sAUPF5ZlsgC83ysaaICA1r1OUuo37dQmqVw3lKZERlAu0nIy/Kz5edHUYpm70bGzbg6j0aJ+vGj6GiWKi9rNVt/Uabrzix4UpuSvL7OeplaSh8/t6keo8wWidjIo+y5mhSty5nU0VNrW+QXWDAYBGCKYb6IUGiTQClBI7azvxUNPO2fDKxTtRKDciQ9LZSxD+uhw+oOowXcSOlN++fVrkqKJpYH3SSmj8IHufp00kmPx3TQlDcmHPzs5cCW6gMVV+wq3AnQ26I7ZKwgPXcLFB7gautiCqy+/aaXX5vYwGUbbi22C2IsHVbIpGNnfmFO/N6VZI3lrjr2Uq2lu+XmI0UTA4fKEpuuhKZGBAcr0N0DMkie4mwDJVIufpDgwcvjC+j/e4pgRmV3G8626xU931DqXcy4hB3Qk13fVopHzqyTwT8Nnw+FAKys0PvWLOwIFvV++nG6UK3XUjyjlREQR6AafpjzgDMsa6XKQETj0Ox4TRFxYOnmM/sdakh57j/Sm32bazUNt/NfNTheA6gBfN4phDr9rO3Pds1Hi31240GENLr68216496DT9vTFJsgIaG7Iw1q/qp4m9cdZbLPxxcG0eu4kQc4MSDedncUXgFEO8oIFkqBz6xN9WBs27ybbw6rw3+KnX/+e8/fy38t5yEHwTfB+cBIPg5+B58CIYBjcBbv3eIi3eEof/hU/Cr8NvV9BHrdLnSVB7wuP/ARqV0tU=</latexit>
<latexit
sha1_base64="0aFp8NtWshujG7fBdtcQeORs2M0=">AAAKL3icjVbNbhs3EF4l/bHUP6c99sJWMhAHsirZRdtDDcRwDzm0gGrESQCtYHC5lESISzIkN5ZN7KP0Ifo0RS9Fr32LDlcra3e1ArKSoNHMNx85o+EMI8WZscPh361Hjz/48KOPD9qdTz797PMvDp98+crIVBN6TSSX+k2EDeVM0GvLLKdvlKY4iTh9HS0vvf31O6oNk+KlvVN0muC5YDNGsAXVzeEf4aTz9saFasGyp2EkeWzuEvhy9xkKExajsm6VHaNz9LYRR5tIwOEYhVgpLVdIAcIuqMXvtVAnnHY6N4fd4WCYP2hXGBVCNyie8c2Tg/swliRNqLCEY2Mmo6GyU4e1ZYTTrBOmhipMlnhOJyAKnFAzdXkeM3QEmhjNpIaPsCjXlj0cTozfISATbBembvPKfTa7SB5MnVDQWyKTBIs4z8kLzOdSM49xIdY4YsRtVVmnSrYxMJI1G6oOk9TOfpo6JlRqqSDrMGcpR1YiXxIoZpoSy+9AwAQIGEFkAbsgFgqnwhS/Y8oUOVutkwZLQQBSUwjxjlOnOGYiyyMs9K74ztzLQpgYWA/Kb1qBQZEoaZg3ZJNCOXXjkrYC5xTyVwL+mv+uQIjUknOs70qwywddbeMxnTHRsE5Jv2X5ZQ8YKi5NVA18sVXW1gQB62WVotBt3a8KULVqmIipQlAuimoVfpO/vJhVYYRlk7Op645Q4dE9RSGnMxtCV7CoexZqNl/YQc1NlPyEtAV3aZn9PJWStGx5Xy1Sk0YEb5JR0icpt0zL26zxUHXqWn9ATY0BgFZKbmrolQGJ1gJUimTO9cKxYb2T8WUWNqJwamWCVeYuQPjtYvyAqsJMHjvWfvvuWZ6jkqaG9UkroOGD7H2e1ZFg8p8NJXTGlTs5OckKcA1NmPYdbg3ubdE9uVMSHriByy2yGbjegiwv37TT8vJ7GS1mfM23xexEQsrZlLVsNuaU7M3pTkjeWuGvZArtLV8vcRZpaBy+0DRb9RW20CCF2QWYBVbU9CNgmWuZirgBQzCH9n20xzWm0LvyEW36+U5N3zsU8iCBKdqfMdvftEYm5p7MMwGf6xy1lWTCfjfI+wwMbbf+f/oo1vi2j5gQVCMI9Bym6fckATLO+0LGFKaegDFhzbmDwXPkO9aGtO053p9yl203C5X9lzM/1xhuE2RVL44lnFXXW/ozi2r/7VU2GU3hSPs+AMlyV1l31Kv7e2MUJTk0tHRlnV/VdxN3nTlvcfAW4Fofu5GUS4sjA/MzvyIIRiBe0EAydArnxN9WRvW7ya7w6nQw+mEw/P20+/zn4t5yEHwdfBs8DUbBj8Hz4EUwDq4D0nrcOm6dts7af7b/av/T/ncNfdQqfL4KKk/7v/8B4H6+Gg==</latexit>
<latexit
VAE Encoder

Recognition network fits the posterior distribution

q (z | x) = q(z | e (x)) ⇡ p✓ (z | x)

q (z | x) = N (z | fe,µ (x; φ), diag(fe, (x; φ)))

https://danijar.com/building-variational-auto-encoders-in-tensorflow/

If we assume posterior is Gaussian with diagonal covariance, we have

VAE Architecture

https://krasserm.github.io/2018/07/27/dfc-vae/
 sha1_base64="mfurLKujCx0BYwtSiK+PCJ25rcA=">AAALb3icvVbdbts2FFbc/STOtjbdxS4GDNzsFEngeHY6bLtYgAbZRS86wAuatoBlBBRF24QpUiWpxgnBBx2wl9gb7FCWY0m2gfVmcgyfnPOd7/yIPGSUcqZNr/fXTuPRJ59+9vnuXnP/iy+/evzk4OkbLTNF6DWRXKp3EdaUM0GvDTOcvksVxUnE6dtoduntbz9QpZkUr81dSkcJngg2ZgQbUN0c7PwdRnTChMWcTcSJa746CiPJY32XwI8NzZQa7DqookynzKEwYXFFPXfH6Nk5Cpkw6P3NAlUhu9/iFHI5QeFYYWLTm2XIoyqqmsG9O3b2Y2KAtoPiOjBsPgs5fV8k8PF5//8pny9y3RrzuBlSET+8zJsnrV63lz9oXegXQisonsHNwe59GEuSJVQYwrHWw34vNSOLlWGEU9cMM01TTGZ4QocgCpxQPbL5OnToEDQxGksFX2hmri17WJxonyogE2ymum7zym02M00eTM1Q0FsikwRDrb4RLzGfSMU8xoZY4YgRu1K5ZpVsaWDEbTZUHYaZGf86skykmaGCLMocZxwZifyWQjFTlBh+BwImQMAIIlPIghjYeBWm+ANLddGz+aJpEAoKkIpCiXec2pRjJlxeYaG3xa+zrwthqCEebN9RBZYqmUrNvMENC+XIDkraCpxT6F8J+Cr/vwIhUknOsborwS4fdLXEYzpmYkOckn7F8vsWMKy4LElr4IuVshYTBKxmVYpCt3K/KkDVVcNETFMEyyWlKg2/zz9edFUYYW74fGRbfVR4tM5gC9KxCWGqGtR6Hio2mZpuzU2U/IQ0BXcpzHaeypI0bHZfXaQ6iwheNqOkTzJumJK3buOmata1foPqGgMAjZRc19BzDRKtFZimxFnbDgeatU8Hly7ciMKZkQlOnb0A4Y+LwQOqCtN57Vj59O1J3qOSpob1TSug4YPsfU7qSDD575ISRuTcnp6eugJcQxOm/IRbgNsrdFuuLQkPXMLlCrkZuEhBlsNvyrQcfiujwYwv+FaYtUpIuZuy1s2NPSVbe7pWkrdW+CudQluXr5c4ixQMDr/QFJt3UmxgQAq9DtBTnFLdiYBlomQm4g0YgjmM78MtrjGF2ZVfcXQnz1R3vEMhdxM4OjtjZjrL0cjExJN5JuCzzcO9VMJ94MduPmfg2LWL9wMnssK3HcSEoApBoedwmv5EEiDjvCNkTOHUE3BMGH1u4eA59BNrSbrnOf475Trbehcq+Zc7P1EYLhpkXl8cM9irtj3zexbV3u2VG/ZHsKX9HIBm2SvX6rfr/t4YRUkODQ2dG+uj+mlir531Fgt/Alzrx24k5czgSMP5mV8RBCNQL2igGSqDfeJvK/363WRdeHPW7f/c7f151nrxW3Fv2Q2+DX4IjoJ+8EvwIngZDILrgDQuG6yhGrr5z/43+9/towW0sVP4fB1Unv3jfwEHLTky</latexit>
<latexit
Z
Training VAE

Z
L(✓,  | x) = q (z | x) log

 log q (z | x)
p✓ (x, z)
p✓ (x, z)

q (z | x)
q (z | x)
dz

dz = log p✓ (x)
We use ELBO to maximize data likelihood indirectly by maximizing ELBO.

Recall that ELBO is a lower bound to data likelihood due to Jensen’s inequality:

Therefore, for fixed encoder \phi, increasing ELBO should increase data likelihood.
 sha1_base64="KJozOGjFSnv1JHobBDpuP/mvUH8=">AAALRnicjVbdbts2FFayv0TdT7vdbTfc7ABp63h2O2y7WIEW7YB264asaNoClhFQFG0TpkiOpBonBJ9qL7FX2EvsbtjtDmU5lmQbmBzDJ4ff+c6PeMiTKs6MHQz+2tt/59333v/g4DC+8eFHH39y89anr4wsNKFnRHKp36TYUM4EPbPMcvpGaYrzlNPX6fxxWH/9lmrDpHhpLxUd53gq2IQRbEF1fmvvj2QUPz9OUskzc5nDj0vsjFrse6ihVDPmUZKzrKFe+NvoAYqTHNtZmrof/blrWFFlGJcCLA3LUQkjmLtf/fHSZOIGwVElP/O3PZBxOrEjlHA5Rep8Fc5x0+1mKFceImk4z4sQTgj8uB3z3SbSsGmOd4ITmUmLtuYFi5pNZ3aM4hP0JARLF9b9/NxDgCGNY/T7NtarHaVMFNaYc8qRauFXjm7HyTg+v9kZ9AflgzaFYSV0ouo5Pb91cJVkkhQ5FZZwbMxoOFB27LC2jHDq46QwVGEyx1M6AlHgnJqxK7eXR0egydBEavgKi0pt3cLh3IRAARlepGmvBeWuNTvLr5fiRNALIvMci6x87U8xn0rNAsYlUJmUEbdW+bhJtlpgxG9faBqMCjv5fuyYUIWlgizTnBQcWYlCp6CMaUosvwQBEyBgBJEZREEs9FODKXvLlKlqtlgWDVxBAlJTSPGSU6c4ZsKXGVZ6V/1697ISRgb8QVeOGzClpZKGhQU/qpRjd1rTNuCcQv1qwOfl/w0IkVpyjvVlDfb4WtcKPKMTJrb4qenXLE92gGHHFblqgR+tlS2fIGA9b1JUurX5iwrU3DVMZFQh2C6KapV8WX6C6Jswwvzo/th1hqiy6NxbNmwCh6VFnfvLbuu3zETNTkhbcdfc7OZpbEnL5lfNTWqKlOBVMWr6vOCWaXnhtzZV3NaGBjUtBgBaKblpoRcGJNpKUCninesmp4Z1T04f+2QrChdW5lh59wiEXx6dXqOaMFPmjnUI390pa1TTtLChaBU0uZaDzZ02EpbCd0UJB+TCnZyc+ArcQhOmwwm3BHfX6K7c2BIBuILLNXI7cBmCrLvfFmnd/U5Gixlf8q0xG5mQejVlq5pba0p21nQjpbDa4G9UCu3cvkHiLNVwcISNptmip7CFA1KYTYCZYUVNLwWWqZaFyLZgYDqA4/toh2lG4ewqJxfTKyM1vWBQyf0cBoXehNne6mhkYhrIAhPwufjoUEkm7Nf98pyBecEt308PZRpf9BATgmoEiT6A2/QbkgMZ5z0hMwq3noBrwpoHDi6eo3BirUgPA8f/p9xk26xCI/565acawzBBFu3NMYdedd156FnUercv/Gg4hpZezV4vfGfYbdsv57e8hC5nmOA1nCbuzLuw4uBPgGn72k2lnFucGrg/yxFBMAL5ggaKoQvokzCtDNuzyabw6l5/+G1/8Nu9zsMfqrnlIPoi+io6jobRd9HD6Gl0Gp1FZP/z/Yf7z/Z/iv+M/47/if9dQvf3KpvPosZzI/oPVDsoMw==</latexit>
<latexit
sha1_base64="VRMTNBVM0Lw6OXxDwLjqqgDFjLc=">AAAKJnicjVbNbuM2EJZ3+xO7f9n22AtbO8Buqrh2dtEWaANskB720AJusNldwDICiqJtwhRJkNTGCaG36EP0aXorit76HD11KMuxJNvAyhY0mvnmI2c0HDJWnBk7GPzTevDwvfc/+PCg3fno408+/ezw0eevjMw0oVdEcqnfxNhQzgS9ssxy+kZpitOY09fx4sLbX7+l2jApXtpbRScpngk2ZQRbUF0f/h6NO1EseWJuU3i4uxydoem1oyGK0ix/XLUt8x9R9T1Sc5Y/Qd+s8YbNUvxOLpFMpK0bqDKMS5F3oknn+rA76A+KC20Lw1LoBuU1un50cBclkmQpFZZwbMx4OFB24rC2jHAKpJmhCpMFntExiAKn1Exckb4cHYEmQVOp4RYWFdqqh8Op8dMEZIrt3DRtXrnPZufpvakTCXpDZJpikbjIzukLzGdSM49xEdY4ZsRtVHmnTrY2MJLvNtQdxpmd/jBxTKjMUkFWYU4zjqxEvhJQwjQllt+CgAkQMILIHGZBLNRLjSl5y5Qpc7ZcJQ2GggCkphDiLadOccz8x4MIS70rn7l7WQpjA+NB1U1qMKWlkoZ5Qz4ulRM3qmhrcE4hfxXgL8V7DUKklpxjfVuBXdzrGhNP6JSJHeNU9BuWn/eAoeKyVDXA5xtlY0wQsF7UKUrdxv2yBNWrhomEKgTloqhW0VfFz4t5HUZYPn46cd0hKj26pyjidGojaAYWdZ9Gms3mtt9wExU/IW3JXRlmP0+tJC1b3NWL1GQxwetkVPRpxi3T8ibfuag6Ta1foKbBAEArJTcN9NKARBsBKkVy53rRyLDeyegij3aicGZlilXuzkH49Xx0j6rDTBE71n767rjIUUXTwPqkldDoXvY+x00kmPy9poT2uHQnJyd5CW6gCdO+w63AvQ26J7dKwgPXcLlB7gaupiCrw++aaXX4vYwWM77i22C2IiHVbMpGNnfmlOzN6VZI3lrjr2UK7S1fL3EWa2gcvtA0W4YKW2iQwmwDzBwrasIYWGZaZiLZgSGYQ/s+2uOaUOhdxc5swmKmJvQOpdxPqcXhlNlw3RqZmHkyzwR8rnPUVpIJ+22/6DOwk7vV9wlRovFNiJgQVCMI9Ax202ckBTLOQyETCruegG3CmjMHG8+R71hr0rbneHfKbbbtLNTmX838TGM4IZBlszgWsFZdb+HXLGp828t8PJzAkvZ9AJLlLvPusNf098Y4TgtoZOnSOj+q7ybuKnfe4uAvwLW57cZSLiyODeyfxRFBMALxggaSoTNYJ/60MmyeTbaFV6f94Xf9wW+n3ec/leeWg+DL4OvgcTAMvg+eBy+CUXAVkOC/Fmo9aR23/2j/2f6r/fcK+qBV+nwR1K72v/8DTui7EQ==</latexit>
<latexit
⇥

sha1_base64="53+ijlDXzxayZZVGdL0Wnn0EF/o=">AAAKE3icjVZfjxs1EN8r/y6BwhUkeODFkJxUqiQkVwQ8cFJPV6Q+FCmcem2lbHTyep3Eitde2d5eUssPfAg+DW+IVz4An4CvwXh3c9ndJBKbizI385ufPbMzY0cpZ9oMh/8c3Xvn3ffe/+C41f7wo/sff3Ly4NOXWmaK0GsiuVSvI6wpZ4JeG2Y4fZ0qipOI01fR8tLbX72hSjMpXph1SqcJngs2YwQbUN2c/BZO2n0UJtgsosj+7G5sGEke63UCP3blUKhZUtgJ5vapAw2nMzNBzx9WkaFZUINdD9WU6YIBPmExqrN+g0LF5gszbYfTm5POcDDMH7QrjEqhE5TP+ObB8dswliRLqDCEY60no2FqphYrwwinrh1mmqaYLPGcTkAUOKF6avNcOXQKmhjNpIKvMCjXVj0sTrTfJyB90Lpp88pDNrNI7kztUNBbIpMEizjPzjPM51Ixj7EhVjhixG5Vrl0n2xgYcfsNdYdJZmY/Ti0TaWaoIEWYs4wjI5F/7ShmihLD1yBgAgSMILKAXRADxVFjit+wVJc5WxVJg6UgAKkohLjm1KYcM+HyCEu9LX+dfVEKEw3rQYlNa7BUyVRq5g1uUiqndlzR1uCcQv4qwOf5/zUIkUpyjtW6Aru80zU2HtMZE3vWqei3LE8PgKHisiRtgC+2ysaaIGC1rFOUuq37VQmqVw0TMU0RlEtKVRp+lX+86Oowwtzk8dR2Rqj06JwVTRpC5xvUeVw026DhJip+QpqSu7LMYZ5aSRq2fFsvUp1FBG+SUdEnGTdMyVu3t6naTa1vUN1gAKCRkusGeqVBoo0A05Q4a7vhWLNuf3zpwr0onBmZ4NTZCxB+uRjfoeownceOld++fZTnqKJpYH3SSmh4J3ufR00kmPx3QwnzcWX7/b4rwQ00YcpPuALc3aK7cqckPHADl1vkfmCxBVldft9Oq8sfZDSY8YJvi9mJhFSzKRvZ3JtTcjCnOyF5a42/lil0sHy9xFmkYHD4QlNs1UuxgQEp9C5AL3BKdS8ClrmSmYj3YODEhPF9esA1pjC78mNY9/Kd6p53KOVBAudpb8ZMbzMamZh7Ms8EfLZ92kolE+bbQT5n0DmyxfvpoVjh2x5iQlCFINBzOE2/IwmQcd4TMqZw6gk4Jow+t3DwnPqJtSFteY7/T7nLtpuF2v6rmZ8rDBcEsmoWxxJ61XaXvmdR491eucloCi29uY9cuc6o2/QvLjNJDg0NXRnrV/XTxF476y0W/gS4No/dSMqlwZGG8zO/IghGIF7QQDJUBn3ShtvKqHk32RVeng1G3w+Gv551nvxU3luOgy+Dr4OHwSj4IXgSPAvGwXVAgn+P7h99fvRF6/fWH60/W38V0HtHpc9nQe1p/f0fSYOzeQ==</latexit>
<latexit
KL term can be computed in closed form.
Reparameterization Trick

not depend on inference network parameters.

z = fe,µ (x; ) + fe,σ (x; ) ! ✏

⇤
Unit Gaussian

We can now take the gradient of −Ex⇠D [L(✓,  | x)] since expectations do
L(✓,  | x) = E✏⇠N (0,I) log p✓ (x | z = µ (x) +   (x)  ✏) DKL (q (z | x) k p(z))
To compute ELBO and its gradient, we use the so-called reparameterization trick.

Suppose encoder/inference network estimates parameters of Gaussian posterior.

 sha1_base64="KJozOGjFSnv1JHobBDpuP/mvUH8=">AAALRnicjVbdbts2FFayv0TdT7vdbTfc7ABp63h2O2y7WIEW7YB264asaNoClhFQFG0TpkiOpBonBJ9qL7FX2EvsbtjtDmU5lmQbmBzDJ4ff+c6PeMiTKs6MHQz+2tt/59333v/g4DC+8eFHH39y89anr4wsNKFnRHKp36TYUM4EPbPMcvpGaYrzlNPX6fxxWH/9lmrDpHhpLxUd53gq2IQRbEF1fmvvj2QUPz9OUskzc5nDj0vsjFrse6ihVDPmUZKzrKFe+NvoAYqTHNtZmrof/blrWFFlGJcCLA3LUQkjmLtf/fHSZOIGwVElP/O3PZBxOrEjlHA5Rep8Fc5x0+1mKFceImk4z4sQTgj8uB3z3SbSsGmOd4ITmUmLtuYFi5pNZ3aM4hP0JARLF9b9/NxDgCGNY/T7NtarHaVMFNaYc8qRauFXjm7HyTg+v9kZ9AflgzaFYSV0ouo5Pb91cJVkkhQ5FZZwbMxoOFB27LC2jHDq46QwVGEyx1M6AlHgnJqxK7eXR0egydBEavgKi0pt3cLh3IRAARlepGmvBeWuNTvLr5fiRNALIvMci6x87U8xn0rNAsYlUJmUEbdW+bhJtlpgxG9faBqMCjv5fuyYUIWlgizTnBQcWYlCp6CMaUosvwQBEyBgBJEZREEs9FODKXvLlKlqtlgWDVxBAlJTSPGSU6c4ZsKXGVZ6V/1697ISRgb8QVeOGzClpZKGhQU/qpRjd1rTNuCcQv1qwOfl/w0IkVpyjvVlDfb4WtcKPKMTJrb4qenXLE92gGHHFblqgR+tlS2fIGA9b1JUurX5iwrU3DVMZFQh2C6KapV8WX6C6Jswwvzo/th1hqiy6NxbNmwCh6VFnfvLbuu3zETNTkhbcdfc7OZpbEnL5lfNTWqKlOBVMWr6vOCWaXnhtzZV3NaGBjUtBgBaKblpoRcGJNpKUCninesmp4Z1T04f+2QrChdW5lh59wiEXx6dXqOaMFPmjnUI390pa1TTtLChaBU0uZaDzZ02EpbCd0UJB+TCnZyc+ArcQhOmwwm3BHfX6K7c2BIBuILLNXI7cBmCrLvfFmnd/U5Gixlf8q0xG5mQejVlq5pba0p21nQjpbDa4G9UCu3cvkHiLNVwcISNptmip7CFA1KYTYCZYUVNLwWWqZaFyLZgYDqA4/toh2lG4ewqJxfTKyM1vWBQyf0cBoXehNne6mhkYhrIAhPwufjoUEkm7Nf98pyBecEt308PZRpf9BATgmoEiT6A2/QbkgMZ5z0hMwq3noBrwpoHDi6eo3BirUgPA8f/p9xk26xCI/565acawzBBFu3NMYdedd156FnUercv/Gg4hpZezV4vfGfYbdsv57e8hC5nmOA1nCbuzLuw4uBPgGn72k2lnFucGrg/yxFBMAL5ggaKoQvokzCtDNuzyabw6l5/+G1/8Nu9zsMfqrnlIPoi+io6jobRd9HD6Gl0Gp1FZP/z/Yf7z/Z/iv+M/47/if9dQvf3KpvPosZzI/oPVDsoMw==</latexit>
<latexit
sha1_base64="FmYnt1ifnhvXU8y5mGXCvYygOA0=">AAAKk3icjVZtb9s2EJa7t0R7azbs075wswMkne3Z7rAN2AI0Sz4UaAt4QdMWsByDomibMEVqJNU4JfQT9nX7bfs3O8pyLMn2MCWGz3fPPcc7HY8ME8606fX+aTx47/0PPvzo4ND/+JNPP/v84dEXr7RMFaHXRHKp3oRYU84EvTbMcPomURTHIaevw8WFs79+S5VmUrw0dwkdx3gm2JQRbEA1OWp4wci/nNjA0KWxz55n2ckfKEiwwpxTjpJTdIYCncYTuzjrZzf2WYYCTqdmBF9ytpJPUDBVmNh+ZgPNZjGeLACl2GxuTtF3hXFtuRmA6iSI08kCdVDv9GaQ2QEKSCQN6meguucarEnGyD9DnYr+P1a0idOp/XAx8/D9NbEfjP3Jw2av28sftC30C6HpFc9wcnTwLogkSWMqDOFY61G/l5ixxcowwmnmB6mmCSYLPKMjEAWOqR7b/F1l6Bg0EZpKBR9hUK4te1gca30Xh4CMsZnrus0p99nMPL43+YGgt0TGMRYRvN05fYr5TCrmMDaA9xsyYjeqzK+SrQ2MZLsNVYdRaqY/jy0TSWqoIKs0pylHRiLXdihiihLD70DABAgYQWQOqyAGmrPCFL1liS5qtlwVDUJBAlJRSPGOU5twzESWZ1jobfGd2ZeFMNIQD1p8XIElSiZSM2fIRoVybIclbQXOKdSvBHye/65AiFSSc6zuSrCLe11t4RGdMrEjTkm/YbncA4aOS+OkBj7fKGsxQcBqUaUodBv3qwJU7RomIpogaJeEqiT4Jv9zYlaFEZaNHo9tE7bVyqM5WG3JACaPQc3Hq93WrbmJkp+QpuAuhdnPU2lJwxbvqk2q05DgdTFK+jjlhil5m+3cVH5d6zaorjEA0EjJdQ291CDRWoJJQjJrW8FQs1ZneJEFO1E4NTLGSWbPQXhxPrxHVWE6zx0rt3z7KK9RSVPDuqIV0OBedj6P6kgwuc+aMg7l0nY6nawA19CEKTfhVuDWBt2SWy3hgGu43CB3A1dLkOXwu1ZaDr+X0WDGV3wbzFYmpFxNWavmzpqSvTXdSslZK/yVSqG97eskzkIFg8M1mmLLdoINDEihtwF6jhOq2yGwzJRMRbQDQzCH8X28xzWiMLvya4Bu5yvVbedQyN2YGtyeMtNej0YmZo7MMQGf9Y8PE8mE+b6bzxm4J9jV+2mjSOHbNmJCUIUg0TM4TX8gMZBx3hYyonDqCTgmjD6zcPAcu4m1Jj10HP+fcpttuwqV9ZcrP1M4mTOyrDfHAvaqbS3cnkW1d3uVjfpj2NJuDkCx7FXW7Lfq/s4YhnEOXd2rXFQ3Tex1Zp3Fwr8A1/qxG0q5MDjUcH7mVwTBCOQLGiiGSmGfuNtKv3432RZeDbr9H7u93wfNJ78W95YD72vvW+/E63s/eU+8p97Qu/ZIY9b4s/FX42//K/8X/zf/cgV90Ch8vvQqj//iX9zM1yM=</latexit>
<latexit
sha1_base64="kWxl/8ChrkbSkOhceAY/pTQ+wtY=">AAAJ0HicjVZfb9s2EFe7f7H3p+n22BdtdoC2UDy7Ldo9NEOD7KEDtsELmraAZQQURduEKZIgqcYOQQx97efYp9nr9rJvs6Msx5JsA1Ni+Hz3ux95p+MdE8moNv3+v7duf/TxJ59+dtBqf/7Fl1/dObz79WstcoXJBRZMqLcJ0oRRTi4MNYy8lYqgLGHkTTI/8/Y374jSVPBXZinJOENTTicUIwOqy8Mf5f04ESzVywy+7LV7EJ6EcYbMDCNmf3M1a99F1Z8/uweXh51+r1884bYwKIVOUD7Dy7sH13EqcJ4RbjBDWo8GfWnGFilDMSOuHeeaSITnaEpGIHKUET22RaAuPAJNGk6Egg83YaGteliUab85QPoQdNPmlftsZpbdmNoxJ1dYZBniqY3NjLxEbCoU9RgbI4USiu1G5dp1srWBYrfbUHcY5Wbyw9hSLnNDOF6FOclZaETo31mYUkWwYUsQEAYCikM8g11gA2+2xpS+o1KXOVuskgZLQQBCEQhxyYiVDFHuighLvS2/nX1VCiMN60F9jGswqYQUmnqDG5XKsR1WtDU4I5C/CvCX4ncNgoUSjCG1rMDObnSNjadkQvmOdSr6DctPe8BQcXkmG+DTjbKxJghIzesUpW7jfl6C6lVDeUpkCOUiiZLxt8WfF10dhqkbPR7bziAsPTqPwpiRiYnh2Jqw8zhWdDozvYYbr/hxYUruyjL7eWolaej8ul6kOk8wWiejos9yZqgSV27noWo3tf6A6gYDAI0QTDfQCw0SaQQoJXbWduOhpt3j4ZmLd6JQbkSGpLOnIPx6OrxB1WG6iB0pv337sMhRRdPA+qSV0PhG9j4Pm0gw+c+aEpriwh4fH7sS3EBjqnyHW4G7G3RXbJWEB67hYoPcDVxtQVSX37XT6vJ7GQ2ibMW3wWxFgqvZFI1s7swp3pvTrZC8tcZfy1S4t3y9xGiioHH4QlN0EUlkoEFyvQ3QMySJjhJgmSqR83QHBuYftO+jPa4pgd5VzFAdFTvVkXco5V5GDIom1ETr1kj51JN5JuCz7aOWFJSb73tFn4GZa1fvJwpTha6ikHJOVAiBnsA0fYIzIGMs4iIlMPU4jAmjTywMniPfsdakLc/x/ym32bazUNt/NfNTheSM4kWzOOZwVm137s9s2Hi35240GMORXt8uzl1n0G36e2OSZAU0NmRhrF/VdxN74ay3WPjn4Nocu4kQc4MSDfOzuCJwiiFe0EAyVA7npA23lUHzbrItvH7UGzzt9X9/0nnxvLy3HAT3gu+C+8EgeBa8CF4Gw+AiwMGfwV/B38E/rfPWovVH6/0KevtW6fNNUHtaH/4D1k+anA==</latexit>
<latexit
DKL (q k p) =
X

k=1
K 
log
✓
⇥

1
σk
◆
p(z) = N (0, I)

+
σk2
2·1
+ (µk  0) 2
KL term can be computed in closed form.

sha1_base64="6mn/P4wUBvp5iPUpW4ILKnkbZZQ=">AAAJ83icjVbNbhs3EN6kf5b657THXraVDMTBWpWSoO2hBmK4hxzaQjXiJIBWMLhcSiLEJVmSG0tm+CS5Bbn2OfoMfYhe22uHq5W1u5KAri1oNPPNR87scIaJZFSbfv+vO3ffe/+DDz86aLU//uTTzz4/vPfFcy1yhcklFkyolwnShFFOLg01jLyUiqAsYeRFMj/39heviNJU8GdmKck4Q1NOJxQjA6qrQ/T7/TgRLNXLDL7sjTsOT8M4Q2aGEbO/uob1dfVnnOUuig1ZGJtSNK1jY02nGXLHx1eHnX6vXzzhtjAohU5QPsOrewc3cSpwnhFuMENajwZ9acYWKUMxI64d55pIhOdoSkYgcpQRPbZFLlx4BJo0nAgFH27CQlv1sCjTfouA9FHqps0r99nMLLs1tWNOrrHIMsRTG5sZeYrYVCjqMTZGCiUU243KtetkawPFbreh7jDKzeSHsaVc5oZwvApzkrPQiNC/1jClimDDliAgDAQUh3gGu8AGXn6NKX1FpS5ztlglDZaCAIQiEOKSESsZotwVEZZ6W347+6wURhrWgxIa12BSCSk09QY3KpVjO6xoa3BGIH8V4M/F7xoECyUYQ2pZgZ3f6hobT8mE8h3rVPQblp/2gKHi8kw2wGcbZWNNEJCa1ylK3cb9ogTVq4bylMgQykUSJeOviz8vujoMUzd6NLadQVh6dB6GMSMTE8PJNmHnUazodGZ6DTde8ePClNyVZfbz1ErS0PlNvUh1nmC0TkZFn+XMUCWu3c5D1W5q/QHVDQYAGiGYbqAXGiTSCFBK7KztxkNNuyfDcxfvRKHciAxJZ89A+OVseIuqw3QRO1J++/ZBkaOKpoH1SSuh8a3sfR40kWDynzUltMaFPTk5cSW4gcZU+Q63Anc36K7YKgkPXMPFBrkbuNqCqC6/a6fV5fcyGkTZim+D2YoEV7MpGtncmVO8N6dbIXlrjb+WqXBv+XqJ0URB4/CFpugikshAg+R6G6BnSBIdJcAyVSLn6Q4MjEho30d7XFMCvasYszoqdqoj71DKvYwYFE2oidatkfKpJ/NMwGfbRy0pKDff9oo+A2PZrt5PFKYKXUch5ZyoEAI9hWn6GGdAxljERUpg6nEYE0afWhg8R75jrUlbnuP/U26zbWehtv9q5qcKyRnFi2ZxzOGs2u7cn9mw8W4v3GgwhiO9voBcuM6g2/T3xiTJCujq+uFX9d3EXjrrLRb+Obg2x24ixNygRMP8LK4InGKIFzSQDJXDOWnDbWXQvJtsC88f9gbf9fq/Pe48+bG8txwEXwXfBPeDQfB98CR4GgyDywAHfwZ/B/8E/7by1pvW29a7FfTundLny6D2tP74D0l9qq0=</latexit>
<latexit
2
1

Reparameterization Trick

=
2
K

k=1
1 X⇥
⇤

2 2

See, e.g., https://mr-easy.github.io/2020-04-16-kl-divergence-between-2-gaussian-distributions/ for a derivation.

2
q(z) = N (z|µ, diag(σ))

log σk  σk  µk + 1
⇤
L(✓,  | x) = E✏⇠N (0,I) log p✓ (x | z = µ (x) +   (x)  ✏) DKL (q (z | x) k p(z))
VAEs vs. Autoencoders (AEs)
VAE can be used to generate new data, unlike AEs.
- Sample z from unit Gaussian, use decoder network to push forward and
sample new datapoint x.

VAE embeds each image into a Gaussian in latent space, vs. AEs encode each
image into a point in latent space.
VAE Sample
Implementation
VAE Learned Embeddings

https://colab.research.google.com/github/probml/pyprobml/blob/master/notebooks/book1/20/vae_mnist_conv_lightning.ipynb
 VAE vs. AE Embeddings
VAE AE

https://colab.research.google.com/github/probml/pyprobml/blob/ https://colab.research.google.com/github/probml/pyprobml/blob/
master/notebooks/book1/20/vae_mnist_conv_lightning.ipynb master/notebooks/book1/20/ae_mnist_conv.ipynb
 sha1_base64="7ZOGCsKV3Bjsjl+b0DoJKixl47w=">AAAJsHicjVZfj+M0EM8d/7blz+3BIxIytCsdp25p9k7AAyvdakG6B5DK6vZupaaqHMdtTR3b2M5ttyZvfBpe4cvwbRin6TZJW4m0Uaczv/nZMx6PHSvOjB0M/n3w8J1333v/g6NW+8OPPv7k0fHjT18bmWlCr4nkUt/E2FDOBL22zHJ6ozTFaczpm3hx6e1v3lJtmBSv7J2i4xTPBJsygi2oJsdfrCYhOkdRiu08jt1P+cT9/mT1x3ISfp2PVuPJcWfQHxQP2hXCUugE5TOcPD5aRYkkWUqFJRwbMwoHyo4d1pYRTvN2lBmqMFngGR2BKHBKzdgVgeToBDQJmkoNr7Co0FY9HE6NuUtjQPoJm6bNKw/Z7Dy9N7UjQW+JTFMsEhfZOX2J+Uxq5jEuwhrHjLitKm/XyTYGRvL9hrrDKLPT78eOCZVZKsg6zGnGkZXIrwlKmKbE8jsQMAECRhCZwyyIhZWrMSVvmTJlzpbrpMFQEIDUFEK849QpjpnIiwhLvSt/c/eqFEYGxoP1H9dgSkslDfOGfFQqx25Y0dbgnEL+KsCfi/81CJFaco71XQV2ea9rTDyhUyb2jFPRb1l+PACGistS1QBfbJWNMUHAelGnKHVb96sSVK8aJhKqEJSLolpFXxYfL+Z1GGH56NnYdUJUenTOUMTp1EawLS3qPIs0m81tv+EmKn5C2pK7MsxhnlpJWrZY1YvUZDHBm2RU9GnGLdPyNt+7qdpNrd+gpsEAQCslNw300oBEGwEqRXLnutHQsO7p8DKP9qJwZmWKVe4uQPjlYniPqsNMETvWfvruaZGjiqaB9UkrodG97H2eNpFg8u+GMo3l0p2enuYluIEmTPsOtwZ3t+iu3CkJD9zA5Ra5H7iegqwOv2+m1eEPMlrM+Jpvi9mJhFSzKRvZ3JtTcjCnOyF5a42/lil0sHy9xFmsoXH4QtNs2VPYQoMUZhdg5lhR04uBZaZlJpI9GII5tO+TA64Jhd5VnJGmV8zU9LxDKfdTanFvymxv0xqZmHkyzwR8rn3SUpIJ+02/6DNwwrr1+vRQovFtDzEhqEYQ6Dmcps9JCmSc94RMKJx6Ao4Ja84dHDwnvmNtSFue4/9T7rLtZqE2/2rmZxqrOSPLZnEsYK+67sLvWdRY26t8FI5hS/s+AMlyV3kn7Db91xeNtIBGli6t86P6buKuc+ctDr4CXJvHbizlwuLYwPlZXBEEIxAvaCAZOoN90obbSti8m+wKr8/64bf9wa/POy9+KO8tR8HnwVfBkyAMvgteBC+DYXAdkODP4K/g7+Cf1lnrpjVp4TX04YPS57Og9rR++w8bAIxo</latexit>
<latexit
sha1_base64="gooIoy5qt9n0PqYy4ob41hTCtXY=">AAAJsXicjVZfjxo3EN8k/XPQf5fmsQ91CyelKVC4i9o+9KScrg95aCV6yiWpWES9XgMWXtuyvTnA2sd+mr62H6bfpuNlOXYXkLqwYpj5zc+e8XjsSHFmbL//74OHj957/4MPTxrNjz7+5NPPTh9//trIVBN6SySX+m2EDeVM0FvLLKdvlaY4iTh9Ey2uvf3NO6oNk+KVXSk6TvBMsCkj2IJqcvrlGl2ikINDjNF6MvgWPR10i//frCfnk9NWv9fPH7QvDAqhFRTPcPL4ZB3GkqQJFZZwbMxo0Fd27LC2jHCaNcPUUIXJAs/oCESBE2rGLo8kQ2egidFUaniFRbm27OFwYswqiQCZYDs3dZtXHrPZeXJvaoaC3hGZJFjELrRz+hLzmdTMY1yINY4YcTtV1qySbQ2MZIcNVYdRaqc/jh0TKrVUkE2Y05QjK5FfFBQzTYnlKxAwAQJGEJnDLIiFpaswxe+YMkXOlpukwVAQgNQUQlxx6hTHTGR5hIXeFb+Ze1UIIwPjQQGMKzClpZKGeUM2KpRjNyxpK3BOIX8l4C/5/wqESC05x3pVgl3f62oTj+mUiQPjlPQ7lp+PgKHi0kTVwFc7ZW1MELBeVCkK3c79pgBVq4aJmCoE5aKoVuFX+ceLWRVGWDa6GLvWABUerXPYcHRqQ9iXFrUuQs1mc9uruYmSn5C24C4Nc5ynUpKWLdbVIjVpRPA2GSV9knLLtLzLDm6qZl3rN6ipMQDQSslNDb00INFagEqRzLl2ODSs3R1eZ+FBFE6tTLDK3BUIv14N71FVmMljx9pP3z3Lc1TS1LA+aQU0vJe9z7M6Ekz+3VImkVy6brebFeAamjDtO9wG3N6h23KvJDxwC5c75GHgZgqyPPyhmZaHP8poMeMbvh1mLxJSzqasZfNgTsnRnO6F5K0V/kqm0NHy9RJnkYbG4QtNs2VHYQsNUph9gJljRU0nApaZlqmID2AI5tC+z464xhR6V35Imk4+U9PxDoXcS6jFnSmznW1rZGLmyTwT8LnmWUNJJux3vbzPwCHrNuvTQbHGdx3EhKAaQaCXcJo+JwmQcd4RMqZw6gk4Jqy5dHDwnPmOtSVteI7/T7nPtp+FyvzLmZ9prOaMLOvFsYC96toLv2dRbW1vstFgDFva9wFIlrvJWoN23d8boyjJoaGlS+v8qL6buNvMeYuDrwDX+rEbSbmwODJwfuZXBMEIxAsaSIZOYZ804bYyqN9N9oXX573B973+b89bL34q7i0nwRfB18HTYBD8ELwIXgbD4DYgwZ/BX8HfwT+Ni8bvjT8a0Qb68EHh8ySoPI3FfxmDizs=</latexit>
<latexit
z1 = Eq(z|x1 ) [z]

sha1_base64="K3ULMB+YHmgWkmfJnaT4JeVRiE0=">AAAJoXicjVZtb9s2EFa7t9h7abp97BdudoChUFy7LbZ9WIAG2YAO2AAvaNoClhBQFG0TpkiCpBo7rH5Jv3Y/av9mR1mOJdkGJlvw+e65h7zj8chEcWbscPjvvfuffPrZ518cdbpffvX1Nw+OH3772shcE3pFJJf6bYIN5UzQK8ssp2+VpjhLOH2TLC68/c07qg2T4pVdKRpneCbYlBFsQXV9/CDKsJ0nifu9mCzf38bXx73hYFg+aFcYVUIvqJ7x9cOj2yiVJM+osIRjYyajobKxw9oywmnRjXJDFSYLPKMTEAXOqIldOfMCnYAmRVOp4RUWldq6h8OZMassAaSfp2nbvPKQzc6zO1M3EvSGyCzDInWRndOXmM+kZh7jIqxxwojbqopuk2xjYKTYb2g6THI7/SV2TKjcUkHWYU5zjqxEfhFQyjQllq9AwAQIGEFkDrMgFpaqwZS+Y8pUOVuukwZDQQBSUwhxxalTHDNRlBFWelf9Fu5VJUwMjAcLHjdgSkslDfOGYlIpYzeuaRtwTiF/NeCf5f8GhEgtOcd6VYNd3OlaE0/plIk949T0W5bfDoCh4vJMtcDnW2VrTBCwXjQpKt3W/bICNauGiZQqBOWiqFbR9+XHi0UTRlgxeRa73ghVHr2nKOJ0aiPYhxb1nkWazeZ20HITNT8hbcVdG+YwT6MkLVvcNovU5AnBm2TU9FnOLdPypti7qbptrd+gpsUAQCslNy300oBEWwEqRQrn+tHYsP7p+KKI9qJwbmWGVeHOQfjrfHyHasJMGTvWfvrucZmjmqaF9UmroNGd7H0et5Fg8u+GMkvk0p2enhYVuIUmTPsOtwb3t+i+3CkJD9zA5Ra5H7iegqwPv2+m9eEPMlrM+Jpvi9mJhNSzKVvZ3JtTcjCnOyF5a4O/kSl0sHy9xFmioXH4QtNsGSpsoUEKswswc6yoCRNgmWmZi3QPhmAO7fvkgGtKoXeVh6IJy5ma0DtU8iCjFodTZsNNa2Ri5sk8E/C57klHSSbsk0HZZ9AZcuv1CVGq8U2ImBBUIwj0DE7T5yQDMs5DIVMKp56AY8KaMwcHz4nvWBvSjuf4/5S7bLtZaMy/nvmZxmrOyLJdHAvYq66/8HsWtdb2spiMYtjSvg9Astxl0Rv12/7r+0VWQiNLl9b5UX03cVeF8xYHXwGu7WM3kXJhcWLg/CyvCIIRiBc0kAydwz7pwm1l1L6b7Aqvnw5GPw2Gfz/vvfi1urccBY+CH4Ifg1Hwc/AieBmMg6uABHnwIfgY/NPpdf7ojDuXa+j9e5XPd0Hj6Uz+Az9lhyg=</latexit>
<latexit
E[x|z] sha1_base64="VdqpSBgjln39l30A1wbTjIBM50Q=">AAAJsHicjVZfjxo3EN+k/w76J5f2sVLlFk5KI44Cido+9KScrpXy0Er0lEtOYhHyeg24eG3X9uYAd9/6afrafpl+m46X5dhdQOrCimHmNz97xuOxI8WZsb3evw8evvPue+9/cNJofvjRx588On386WsjU03oDZFc6tsIG8qZoDeWWU5vlaY4iTh9Ey2uvP3NW6oNk+KVXSk6TvBMsCkj2IJqcvrFejJAFyhMsJ1Hkfspm7jfn6z/WE4GX2ej9Xhy2up1e/mD9oV+IbSC4hlOHp+sw1iSNKHCEo6NGfV7yo4d1pYRTrNmmBqqMFngGR2BKHBCzdjlgWToDDQxmkoNr7Ao15Y9HE6MWSURIP2ETd3mlcdsdp7cm5qhoHdEJgkWsQvtnL7EfCY18xgXYo0jRtxOlTWrZFsDI9lhQ9VhlNrp92PHhEotFWQT5jTlyErk1wTFTFNi+QoETICAEUTmMAtiYeUqTPFbpkyRs+UmaTAUBCA1hRBXnDrFMRNZHmGhd8Vv5l4VwsjAeLD+4wpMaamkYd6QjQrl2A1L2gqcU8hfCfhz/r8CIVJLzrFelWBX97raxGM6ZeLAOCX9juXHI2CouDRRNfDlTlkbEwSsF1WKQrdzvy5A1aphIqYKQbkoqlX4Zf7xYlaFEZaNno1dq48Kj9YAhZxObQjb0qLWs1Cz2dx2a26i5CekLbhLwxznqZSkZYt1tUhNGhG8TUZJn6TcMi3vsoObqlnX+g1qagwAtFJyU0MvDUi0FqBSJHOuHQ4Na58Pr7LwIAqnViZYZe4ShF8uh/eoKszksWPtp++e5jkqaWpYn7QCGt7L3udpHQkm/24pk0gu3fn5eVaAa2jCtO9wG3B7h27LvZLwwC1c7pCHgZspyPLwh2ZaHv4oo8WMb/h2mL1ISDmbspbNgzklR3O6F5K3VvgrmUJHy9dLnEUaGocvNM2WHYUtNEhh9gFmjhU1nQhYZlqmIj6AIZhD+z474hpT6F35GWk6+UxNxzsUcjehFnemzHa2rZGJmSfzTMDnmmcNJZmw33TzPgMnrNusTwfFGt91EBOCagSBXsBp+pwkQMZ5R8iYwqkn4Jiw5sLBwXPmO9aWtOE5/j/lPtt+FirzL2d+prGaM7KsF8cC9qprL/yeRbW1vc5G/TFsad8HIFnuOmv123X/zUUjyaGhpUvr/Ki+m7ibzHmLg68A1/qxG0m5sDgycH7mVwTBCMQLGkiGTmGfNOG20q/fTfaF14Nu/9tu79fnrRc/FPeWk+Dz4KvgSdAPvgteBC+DYXATkODP4K/g7+CfxqBx25g08Ab68EHh81lQeRq//Qcteoxq</latexit>
<latexit
z = λz1 + (1 − λ)z2
Push them through to obtain interpolation images as
Let x1 and x2 be input images. Let their encodings be
Decode points in the linear interpolation between them.
VAE Latent Space Interpolation

z2 = Eq(z|x2 ) [z]

Take a bunch of linearly spaced points obtained by convex combination:

Take two input images (from test set). Encode each one to obtain latent codes.
VAE vs AE Latent Space Interpolation