BIRLA INSTITUTE OF TECHNOLOGY & SCIENCE, PILANI

WORK INTEGRATED LEARNING PROGRAMMES

Digital
Part A: Content Design

Course Title Advanced Deep Leaning

Course No(s) ZG513

Credit Units 4

Credit Model 1 - 0.5 - 1.5

1 unit for class room hours, 0.5 unit for Tutorial, 1.5 units for
Student preparation.
1 unit = 32 hours
Content Authors Dr. Sugata Ghosal

Version 1.0

Date Nov 22nd, 2023

Course Description

This course will cover two areas of deep learning in which labeled data is not required: Deep Generative
Models and Self-supervised Learning. Recent advances in generative models have made it possible to
realistically model high-dimensional raw data such as natural images, audio waveforms and text corpora.
Strides in self-supervised learning have started to close the gap between supervised representation learning and
unsupervised representation learning in terms of fine-tuning to unseen tasks. This course will cover the
theoretical foundations of these topics as well as their newly enabled applications.

Topics:

Introduction to Representation Learning, PCA and variants, likelihood based models, flow models,
autoregressive models, latent variables, Deep autoencoders, Boltzmann Machines, Generative Adversarial
learning, Variants of GAN and applications, DeepDream, neural style transfer, self-supervised learning, semi-
supervised learning, language model learning, applications in time series modelling
Course Objectives

No Topics

CO1 Learn about foundational principles of unsupervised deep learning

CO2 Learn about semi-supervised and self-supervised deep learning

CO3 Learn about representation learning

CO4 Learn about generative modelling and applications

Text Book(s)

T1 Deep Learning (Adaptive Computation and Machine Learning series) Hardcover – 18

November 2016, Aaron Courville, Ian Goodfellow, Yoshua Bengio

Reference Book(s) & other resources

R1 Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts,

Tools, and Techniques to Build Intelligent Systems, Aurélien Géron

R2 Deep Learning with Python, François Chollet

Module 1. Introduction

1.1 Unsupervised, semi-supervised, self-supervised learning

1.2 Representation learning
1.3 Generative Modeling

Module 2. PCA variants

2.1 Randomized PCA
2.2 Incremental PCA
2.3 Kernel PCA
2.3 Probabilistic PCA
2.5 Sparse PCA
2.6 Canonical Correlation Analysis (CCA)
2.6 Locally Linear Embedding (LLE)
2.6 Independent Component Analysis
2.7 Factor Analysis
 2.8 Manifold learning

Module 3. Autoencoders
3.1 Type of activation and loss functions
3.2 Undercomplete Vs. Overcomplete autoencoders
3.3 Relationship with PCA
3.4 Regularization
3.4.1 Denoising autoencoder
3.4.2 Sparse autoencoder
3.4.3 Contractive Autoencoders
5.5 Effect of Depth
5.6 Application of Autoencoders

Module 4. Autoregressive Models

4.1 Motivation
4.2 Simple generative models: histograms
4.3 Parameterized distributions and maximum likelihood
4.4 Autoregressive Models
4.4.1 Recurrent Neural Nets
4.4.2 Masking-based Models
Masked AEs for Distribution Estimation (MADE)
Masked Convolutions
WaveNet
PixelCNN and Variants
Applications in super-resolution, colorization
Speed Up Strategies

Module 5. Normalizing Flow Models

5.1 Difference with AR Models
5.2 Foundations of 1-D Flow
5.3 2-D Flow
5.4 N-dimensional flows
i. AR and inverse AR flows
ii. NICE/ RealNVP
iii. Glow, Flow++
5.5 Dequantization
5.6 Applications in super-resolution, text/audio synthesis, point cloud generation

Module 6. Variational Inteferencing

6.1 Latent Variable Models
6.2 Training Latent Variable Models
6.2.1 Exact Likelihood
6.2.2 Sampling; Prior, Importance
6.2.3 Importance Weighted AE
Variational/Evidence Lower Bound
Optimizing VLB/ELBO
 VAE Variants: VQ-VAE, AR_VAE, Beta VAE
Variational Dequantization

Module 7. Generative Adversarial Network

7.1 Variants
7.1.1 Cycle GAN
7.1.2. DCGan
7.1.3. Style GAN
7.2. Applications of GAN

Module 8. Semi-supervised and Self-Supervised Learning

8.1 In-painting, colorization
8.3 proxy tasks in computer vision
8.3.1 relative patch prediction
8.3.2. jigjaw puzzles, rotations,
8.3.3. contrastive learning

Module 9. Language Modeling

9.1 Motivation and Intro,
9.2 Introduction to Language Models
9.3 A digression into Transformer, Word2Vec, BERT, ….., GPT

Module 10. Time Series Modeling and Generation

10.1 Advanced VAE and GAN techniques for modelling of time-series data
10.2 Generation of time-series data (ARIMA, S-ARIMA etc.)

Learning Outcomes:

No Learning Outcomes

LO1 A strong understanding of the foundations of deep unsupervised learning

LO2 Able to solve problems using appropriate unsupervised learning techniques

LO3 A strong understanding of the foundations of generative modeling

Part B: Learning Plan

Academic Term 2023-24 Semester 1

Course Title Advanced Deep Learning

Course No ZG 513

Lead Instructor Dr. Sugata Ghosal

Session No. Topic Title Study/HW Resource
Reference
1 Introduction
Unsupervised, semi-supervised, self-supervised learning ; T1 – Ch1
Representation learning; Refresher of supervised deep
learning

2 PCA and Variants

2.1 Randomized PCA
2.2 Incremental PCA
2.3 Kernel PCA
2.3 Probabilistic PCA
2.5 Sparse PCA
2.6 Canonical Correlation Analysis (CCA)
2.6 Locally Linear Embedding (LLE)
2.6 Independent Component Analysis
2.7 Factor Analysis

2.8 Manifold learning

3 Autoencoders
3.1 Type of activation and loss functions
3.2 Undercomplete Vs. Overcomplete
autoencoders
3.3 Relationship with PCA

4 Autoencoders (contd)
3.4 Regularization
3.4.1 Denoising autoencoder
3.4.2 Sparse autoencoder
3.4.3 Contractive Autoencoders
5.5 Effect of Depth

5.6 Application of Autoencoders

5 Autoregressive Models
 4.1 Motivation
4.2 Simple generative models: histograms
4.3 Parameterized distributions and maximum
likelihood
4.4 Autoregressive Models
4.4.1 Recurrent Neural Nets
4.4.2 Masking-based Models
Masked AEs for Distribution Estimation
(MADE)
Masked Convolutions
WaveNet
PixelCNN and Variants
Applications in super-resolution,
colorization

6 Normalizing Flow Models

5.7 Difference with AR Models
5.8 Foundations of 1-D Flow
5.9 2-D Flow
5.10 N-dimensional flows
iv. AR and inverse AR flows
v. NICE/ RealNVP
vi. Glow, Flow++
5.11 Dequantization
Applications in super-resolution, text/audio synthesis,
point cloud generation

7 Variational Inteferencing (1)

6.1 Latent Variable Models
6.2 Training Latent Variable Models
6.2.1 Exact Likelihood
6.2.2 Sampling; Prior, Importance
6.2.3 Importance Weighted AE
Variational/Evidence Lower Bound
Optimizing VLB/ELBO
VAE Variants: VQ-VAE, AR_VAE, Beta VAE
Variational Dequantization

8 Review of Session 1 to 7 Books, Web references

and Slides

9 Generative Adversarial Networks

Principles, minimax optimization, DCGAN

10 Other variants of Generative Adversarial Networks

 Cycle GAN, Style GAN, Applications of GAN

11 Semi-supervised and Self-Supervised Learning

In-painting, colorization, split-brain autoencoder, proxy

tasks in computer vision: relative patch prediction, jigjaw
puzzles, rotations, contrastive learning:

12 Semi-supervised and Self-Supervised Learning (Contd)

word2vec, contrastive predictive coding, instance
discrimination, current instance discrimination models

13 Language Modeling

Motivation and Intro, Introduction to Language Models,

History of Neural Language Models, A digression into
Transformer

14 Language Modeling (Contd)

Word2Vec, BERT, ….., GPT

15 Time Series Modeling and Generation

Advanced VAE and GAN techniques for modelling and

generation of time-series data (ARIMA, S-ARIMA etc.)

16 Review of session 9 to 15 Books, Web references

and Slides
Detailed Plan for Lab work

Lab Lab Objective Lab Sheet Session

No. Access URL Reference
1 Autoencoders
2 Deep Autoencoders
3 Convolutional Autoencoders
4 Variational Autoencoders
5 Generative Adversarial Networks

Evaluation Scheme:
Legend: EC = Evaluation Component; AN = After Noon Session; FN = Fore Noon Session
No Name Type Duratio Weight Day, Date, Session, Time
n
EC-1 Quiz – Best 2 out of 3 Online ~1 hour 10%
Assignment-I Take ~2-3 10%
Home weeks
Assignment-II Take ~2-3 10%
Home weeks
EC-2 Mid-Semester Test Closed 30%
Book
EC-3 Comprehensive Exam Open Book 40%

8310758441

Note:
Syllabus for Mid-Semester Test (Open Book): Topics in Session Nos. 1 to 8
Syllabus for Comprehensive Exam (Open Book): All topics (Session Nos. 1 to 16)
Important links and information:
Elearn portal: https://elearn.bits-pilani.ac.in or Canvas
Students are expected to visit the Elearn portal on a regular basis and stay up to date with the latest
announcements and deadlines.
Contact sessions: Students should attend the online lectures as per the schedule provided on the
Elearn portal.
Evaluation Guidelines:
1. EC-1 consists of either two Assignments or three Quizzes. Students will attempt them
through the course pages on the Elearn portal. Announcements will be made on the portal,
in a timely manner.
2. For Closed Book tests: No books or reference material of any kind will be permitted.
3. For Open Book exams: Use of books and any printed / written reference material (filed or
bound) is permitted. However, loose sheets of paper will not be allowed. Use of calculators
is permitted in all exams. Laptops/Mobiles of any kind are not allowed. Exchange of any
material is not allowed.
4. If a student is unable to appear for the Regular Test/Exam due to genuine exigencies, the
student should follow the procedure to apply for the Make-Up Test/Exam which will be
made available on the Elearn portal. The Make-Up Test/Exam will be conducted only at
 selected exam centres on the dates to be announced later.

It shall be the responsibility of the individual student to be regular in maintaining the self study
schedule as given in the course handout, attend the online lectures, and take all the prescribed
evaluation components such as Assignment/Quiz, Mid-Semester Test and Comprehensive Exam
according to the evaluation scheme provided in the handout.