Deep Learning - Theory and Practice

Introduction to Deep Learning course. Examples. Roadmap of the course.

Basics of Machine Learning - decision and inference problems, joint probability and posterior probabilities. Likelihood and priors. Loss matrix. Rule of maximum posterior probability. Loss function for regression.

Matrix derivatives. Maximum likelihood estimation and Gaussian example. Linear models for classification. Perpendicular distance of a point from a surface. Logistic regression.

Logistic regression two class motivation. Posterior probability, sigmoid function, properties. Maximum likelihood for two class logistic regression. Cross entropy error for two class.

Logistic regression for K classes, softmax function. Non-convex optimization (local and global minima). Gradient descent - motivation and algorithm.

Training and validation data sets. Logistic regression code discussion. Perceptron and 1 hidden layer neural networks. Non-linear separability with hidden layer network.

Multi-layer perceptrons, type of hidden layer and output layer activations - sigmoid, tanh, relu, softmax functions. Error functions in MLPs. Backpropagation learning in MLP. MLP for logistic regression in Keras.

Backpropagation in multi-layer deep neural networks. Universal approximation properties of single hidden layer networks. Need for depth. The trade-off between depth and width of networks. Representation learning in DNNs. Hierarchical data abstractions. Example in images.

Convolutional neural networks. Kernels and convolutional operations. Maxpooling and subsampling. Backpropagation in CNN layers. CNN example for MNIST.

Recurrent neural networks, backpropagation in recurrent neural networks. Different recurrent architectures - teacher forcing networks, encoder/decoder networks, bidirectional networks.

Vanishing gradient problem in RNNs. Long short term memory networks. Unsupervised representation learning - Restricted Boltzmann machines, Autoencoders. Discussion of mid-term exam.