Deep Learning - Theory and Practice

CCE 2020 • January – May 2020

Announcements Logistics Syllabus Pre-requisites Textbooks Slides

Announcements

Final Class

Final class held on May 14, 2020.

Final Exam

June 7, 2020, from 6 PM. Open book and notes. Online exam — question paper sent by email, answers to be returned by email before the deadline.

Google Group

Discussions and announcements for the course are posted to the dl_cce2020 Google group.

dl_cce2020

Logistics

Instructor

Sriram Ganapathy

deeplearning doT cce2020 aT gmail doT com

Office: Electrical Engineering C 334 (2nd Floor)

Teaching Assistant

Prachi Singh

Office: Electrical Engineering C 328 (2nd Floor)

Course Content

Basics of pattern recognition, neural networks. Introduction to deep learning, convolutional networks, applications in audio, image and text. All coding done in Python.

Pre-requisites

Random Process / Probability and Statistics

Recommended book: "Probability and Statistics" by Stark and Woods.

Video Lectures
Linear Algebra / Matrix Theory

Recommended book: "Linear Algebra" by G. Strang.

Author's Videos & Text
Basic Programming with Python

numpy based array and matrix operations.

Calculus

Textbooks

B1

Pattern Recognition and Machine Learning

C.M. Bishop, 2nd Edition, Springer, 2007.

B2

Neural Networks

C.M. Bishop, 2nd Edition, Springer, 1995.

B3

Deep Learning

I. Goodfellow, Y. Bengio, A. Courville, MIT Press, 2016.

HTML Version

Slides

16-01-2020
Introduction to Deep Learning course. Examples. Roadmap of the course.
Slides Code
23-01-2020
Decision theory for machine learning. Maximum posterior probability rule. Minimum mean square estimation for regression.
Slides
06-02-2020
Matrix calculus. Differentiating vectors and matrices. Principal component analysis. Preprocessing data - standardizing and whitening.
Slides
13-02-2020
Linear regression. Regularization. Least squares model for classification. Logistic regression, Lecture 1.
Slides
20-02-2020
Linear regression. Regularization. Least squares model for classification. Logistic regression, Lecture 2.
Slides
Assignment #2. Due on March 10.
HW2 Preprocessing Code
27-02-2020
Linear regression. Regularization. Least squares model for classification. Logistic regression, Lecture 3.
Slides
05-03-2020
Midterm Exam.
12-03-2020
Neural networks with one or more hidden layers. Error function. Model update using gradient descent. Backpropagation algorithm. Stochastic gradient descent algorithm.
Slides
Assignment #3. Due on April 17.
HW3 Preprocessing Code
26-03-2020
Problem solving session.
Video
02-04-2020
Discussion on depth versus width. Practical considerations in deep learning. Avoiding overfitting - regularization, dropout. Convolutional neural networks.
Slides Video
09-04-2020
Dropout in detail. Training and testing with dropouts. Convolutional neural networks. Convolution, max-pooling operations.
Slides Video
16-04-2020
CNNs in detail. DNN versus CNN in terms of number of parameters. Backpropagation in CNNs. Convolution and max-pooling backpropagation.
Slides Video
23-04-2020
t-SNE (t-distributed stochastic neighborhood embedding) for data visualization. Understanding deep networks using t-SNE. Using image classification examples. Identifying representation learning using reconstruction. Deep networks in speech processing.
Slides Video
30-04-2020
Recurrent neural networks. Forward and backward propagation. Various architectures for sequence-to-sequence and sequence-to-vector mapping.
Slides Video
07-05-2020
Recap of recurrent networks. Architectures for vector-to-sequence mapping and encoder-decoder sequences. Introduction to LSTM-RNNs. Attention mechanism in encoder-decoder models.
Slides Video
14-05-2020
Assignment #4. Due on May 31.
HW4
14-05-2020
Recap of attention mechanism in encoder-decoder models. Unsupervised models for deep learning. Autoencoders and generative adversarial networks. Image generation using GANs.
Slides Video