Deep Learning - Theory and Practice
DL19 • Spring 2019
Announcements
Midterm Exam
March 7, 2019, during class hours.
No Class on Election Day
April 18, 2019.
Final Exam
May 2, 2019, 6:00 – 7:45 PM. Venue: Lecture Hall 10, Lecture Hall Complex, Centre for Continuing Education (CCE), near Prakruti Cafe, Indian Institute of Science, Bangalore-560012.
Logistics
Instructor
Sriram Ganapathy
deeplearning doT cce2019 aT gmail doT com
Office: Electrical Engineering C 334 (2nd Floor)
Teaching Assistants
Shreyas R, Purvi A
Electrical Engineering C 328 (2nd Floor)
Course Content
- •Basics of pattern recognition and neural networks.
- •Introduction to deep learning, convolutional networks.
- •Applications in audio, image and text.
- •All coding done in Python.
Pre-requisites
Random Process / Probability and Statistics
Recommended book: "Probability and Statistics" by Stark and Woods.
Video LecturesLinear Algebra / Matrix Theory
Recommended book: "Linear Algebra" by G. Strang.
Author's Webpage (Videos and Text)Basic Programming with Python
numpy based array and matrix operations.
Calculus
Textbooks
B1
Pattern Recognition and Machine Learning
C.M. Bishop, 2nd Edition, Springer, 2007.
B2
Neural Networks
C.M. Bishop, 2nd Edition, Springer, 1995.
B3
Slides
Date
Topic
Slides
17-01-2019
Basics of Machine Learning - decision and inference problems, joint probability and posterior probabilities. Likelihood and priors. Loss matrix. Rule of maximum posterior probability. Loss function for regression.
31-01-2019
Matrix derivatives. Maximum likelihood estimation and Gaussian example. Linear models for classification. Perpendicular distance of a point from a surface. Logistic regression.
Refs: PRML, Bishop (Ch. 3, Appendix C)
07-02-2019
Logistic regression two class motivation. Posterior probability, sigmoid function, properties. Maximum likelihood for two class logistic regression. Cross entropy error for K classes. Logistic regression for K classes, softmax function. Non-convex optimization (local and global minima). Gradient descent - motivation and algorithm.
Refs: PRML, Bishop (Ch. 4, Sec. 4.2)
14-02-2019
Summary of previous lectures. Gradient descent for K class logistic regression. Implementing logistic regression for MNIST dataset. Perceptron model and motivation. Introduction to single hidden layer neural networks. Training and validation data sets. Logistic regression code discussion.
Refs: PRML, Bishop (Sec. 4.2) and NN, Bishop (Sec. 3.5, 7.5)
21-02-2019
Perceptron and 1 hidden layer neural networks. Non-linear separability with hidden layer network. Type of hidden layer and output layer activations - sigmoid, tanh, relu, softmax functions. Error functions in MLPs.
Refs: NN, Bishop (Sec. 3.5, Sec. 4.8)
28-02-2019
Backpropagation in multi-layer deep neural networks. Universal approximation properties of single hidden layer networks. Need for depth. The trade-off between depth and width of networks. Representation learning in DNNs. Hierarchical data abstractions. Example in images.
Refs: "Deep Learning", I. Goodfellow (Ch. 6) · arxiv.org/abs/1311.2901
07-03-2019
Midterm Exam posted. Due by 19:45 hrs (90 minutes).
14-03-2019
Convolutional neural networks. Kernels and convolutional operations. Maxpooling and subsampling. Backpropagation in CNN layers. CNN example for MNIST.
Refs: "Deep Learning", I. Goodfellow (Ch. 9)
21-03-2019
Back propagation in CNNs. Choice of kernels and convolutional operations. CNN example for MNIST. Midterm exam discussion.
Refs: "Deep Learning", I. Goodfellow (Ch. 9)
28-03-2019
Recurrent neural networks, back propagation in recurrent neural networks. Different recurrent architectures - teacher forcing networks, encoder/decoder networks, bidirectional networks.
Refs: "Deep Learning", I. Goodfellow (Ch. 10)
04-04-2019
Issue of vanishing and exploding gradients. Long short term memory networks (LSTM). Attention mechanism in neural networks.
Refs: "Deep Learning", I. Goodfellow (Ch. 10)
11-04-2019
Advanced topics, network in network, convolutional LSTM networks, unsupervised learning, autoencoders, adversarial learning.
Refs: "Deep Learning", I. Goodfellow (Ch. 16)
25-04-2019
Applications of deep learning for natural language processing, image processing and speech processing.