BIRLA INSTITUTE OF TECHNOLOGY & SCIENCE, PILANI

WORK INTEGRATED LEARNING PROGRAMMES

Digital
Part A: Content Design
Course Title Deep Learning

Course No(s)

Credit Units 4

Credit Model 1 - 0.5 - 1.5.

1 unit for class room hours, 0.5 unit for Tutorial, 1.5 units for
Student preparation. 1 unit = 32 hours

Content Authors Ms. Seetha Parameswaran

Version 2.0

Date August 07th, 2019

Course Objectives
No Course Objective

CO1 Introduce students to the basic concepts and techniques of Deep Learning.

CO2 Students will be able apply deep learning models to applications.

CO3 Students will be able to evaluate deep learning algorithms.

Text Book(s)
T1 Deep Learning by Ian Goodfellow , Yoshua Bengio, Aaron Courville. MIT Press
2016.

Reference Book(s) & other resources

R1 Introduction to Deep Learning by Eugene Charniak. The MIT Press 2019

R2 Deep Learning with Python by Francois Chollet. 1st Edition. Manning Publications
 Co 2018.

R3 Dive into Deep Learning by Aston Zhang, Zack C. Lipton, Mu Li, Alex J. Smola. 2019

Content Structure
1. Introduction
1.1. Objective of the course
1.2. Review of Machine Learning and Neural Network

2. Deep Feedforward Network

2.1. Multilayer Perceptron
2.2. Gradient based learning
2.3. Architecture design
2.4. Back propagation

3. Regularization for Deep models

3.1. L2 and L1 Regularization
3.2. Constrained Optimization and Under-Constrained problems
3.3. Early Stopping
3.4. Parameter Tying and Parameter Sharing
3.5. Sparse representations
3.6. Dropout

4. Optimization of Deep models

4.1. Challenges in Neural Network Optimization
4.2. Basic algorithms in optimization
4.3. Parameter Initialization Strategies
4.4. Algorithms with Adaptive Learning Rates
4.5. Approximate Second-Order Methods
4.6. Optimization Strategies and Meta-Algorithms

5. Convolutional Networks
5.1. The Convolution Operation
5.2. Pooling
5.3. Convolution and Pooling as an Infinitely Strong Prior
5.4. Structured Outputs

6. Recurrent and Recursive Nets

6.1. Computational Graphs
6.2. Recurrent Neural Networks
6.3. Bidirectional RNNs
6.4. Encoder-Decoder Sequence-to-Sequence Architectures
6.5. Deep Recurrent Networks
6.6. Recursive Neural Networks
6.7. The Long Short-Term Memory and Other Gated RNNs

7. Autoencoders
7.1. Regularized Autoencoders
7.2. Representational Power, Layer Size and Depth
 7.3. Stochastic Encoders and Decoders
7.4. Denoising Autoencoders
7.5. Applications of Autoencoders

8. Generative Adversarial Networks

8.1. Overview
8.2. Applications of GAN

9. Applications
9.1. Computer Vision
9.2. Speech Recognition
9.3. Natural Language Processing
9.4. Recommender Systems

Learning Outcomes:

No Learning Outcomes

LO1 Able to understand the basics of Deep Learning.

LO2 Able to understand and apply techniques related to Deep Learning to applications.

LO3 Able to identify appropriate tools to implement the solutions to problems related
to Deep Learning and implement solutions.

Part B: Learning Plan

Academic Term

Course Title Deep Learning

Course No

Lead Instructor

Session Study / HW
No. Topic Title Resource
Reference

1 Objective of the course

Review of Machine Learning and Learning algorithms– Task,
T1 – Ch5
Performance Measure, Experience R2 – Ch2
Review of Neural Network – Activation function, Loss function,
Optimizer, Stochastic Gradient Descent, Back propagation
 Algorithm

2 Deep Feedforward Networks

Review of Multilayer Perceptron, Gradient based learning, Cost
function, output units, hidden units T1 – Ch6
Demo of one application using Keras and tensorflow R2 – Ch3
Architecture design, Universal Approximation Properties and
Depth, Computational Graph

3 Regularization for Deep models

L2 and L1 Regularization, Constrained Optimization and Under- T1 – Ch7
Constrained

4 Regularization for Deep models (contd)

Early Stopping, Parameter Tying and Parameter Sharing, Sparse T1 – Ch7
representations, Dropout

5 Optimization of Deep models

Challenges in Neural Network Optimization, Basic algorithms in
T1 – Ch8
optimization, Stochastic Gradient Descent, Momentum,
Parameter Initialization Strategies

6 Optimization of Deep models (contd)

Algorithms with Adaptive Learning Rates, AdaGrad, RMSProp,
Adam, Approximate Second-Order Methods, Conjugate gradient, T1 – Ch8
BFGS, Optimization Strategies and Meta-Algorithms , Batch
normalization

7 Convolutional Networks
The Convolution Operation, Pooling, Convolution and Pooling as T1 – Ch9
an Infinitely Strong Prior, Structured Outputs, Demo of CNN on R2 – Ch5
computer vision application

8 Review of Session 1 to 7 Books, Slides,

Web references

9 Convolutional Networks (contd) Web references

T1 – Ch10
 Variants of CNN – ImageNet, VGG16, Inception, ResNet, AlexNet R2 – Ch6
Recurrent and Recursive Nets
Unfolding Computational Graphs, Recurrent Neural Networks

10 Recurrent and Recursive Nets (contd)

Bidirectional RNNs, Deep Recurrent Networks, Encoder-Decoder T1 – Ch10
R2 – Ch6
Sequence-to-Sequence, Recursive Neural Networks

11 Recurrent and Recursive Nets (contd)

The Long Short-Term Memory, Optimization for Long-Term T1 – Ch10
R2 – Ch6
Dependencies, Demo of RNN for text and sequences

12 Autoencoders
PCA, ICA, Regularized Autoencoders, Sparse encoders, T1 – Ch14
R2 – Ch8
Representational Power, Layer Size and Depth

13 Autoencoders (contd)
Stochastic Encoders and Decoders, Denoising Autoencoders, T1 – Ch14
R2 – Ch8
Applications of Autoencoders, Demo of Autoencoder

14 Generative Adversarial Networks

An overview, applications of GAN T1 – Ch20

15 Applications of Deep Learning

Case studies related to Speech Recognition, Natural Language T1 – Ch12
Processing, Recommender Systems

16 Books, Slides,
Review of session 9 to 15
Web references

Detailed Plan for Lab work

Lab Lab Sheet Access Session

Lab Objective
No. URL Reference

1 Introduction to Tensorflow and Keras 2

 Deep Neural Network with
2 4
Back-propagation and optimization

3 CNN 7

4 RNN 10

5 LSTM 11

6 Autoencoders 13

Evaluation Scheme:
Legend: EC = Evaluation Component; AN = After Noon Session; FN = Fore Noon Session

No Name Type Duration Weight Day, Date, Session, Time

EC-1 Quizzes Online 10%

EC-2 Assignments Take Home 15%

EC-3 Mid-Semester Test Closed Book 1.5 Hrs 30%

EC-4 Comprehensive Open Book 2.5 Hrs 45%

Exam

Note:
Syllabus for Mid-Semester Test (Closed Book): Topics in Session Nos. 1 to 8
Syllabus for Comprehensive Exam (Open Book): All topics (Session Nos. 1 to 16)
Important links and information:

Elearn portal: https://elearn.bits-pilani.ac.in or Canvas

Students are expected to visit the Elearn portal on a regular basis and stay up to
date with the latest announcements and deadlines.

Contact sessions: Students should attend the online lectures as per the schedule
provided on the Elearn portal.

Evaluation Guidelines:
1. EC-1 consists of two Quizzes. Students will attempt them through the course
pages on the Elearn portal. Announcements will be made on the portal, in a
timely manner.
2. EC-2 consists of either one or two Assignments. Students will attempt them
through the course pages on the Elearn portal. Announcements will be made
on the portal, in a timely manner.
3. For Closed Book tests: No books or reference material of any kind will be
permitted.
4. For Open Book exams: Use of books and any printed / written reference
material (filed or bound) is permitted. However, loose sheets of paper will
not be allowed. Use of calculators is permitted in all exams. Laptops/Mobiles
of any kind are not allowed. Exchange of any material is not allowed.
5. If a student is unable to appear for the Regular Test/Exam due to genuine
exigencies, the student should follow the procedure to apply for the
Make-Up Test/Exam which will be made available on the Elearn portal. The
Make-Up Test/Exam will be conducted only at selected exam centres on the
dates to be announced later.

It shall be the responsibility of the individual student to be regular in maintaining

the self-study schedule as given in the course hand-out, attend the online lectures,
and take all the prescribed evaluation components such as Assignment/Quiz,
Mid-Semester Test and Comprehensive Exam according to the evaluation scheme
provided in the hand-out.