Machine Learning for Signal Processing
E9 205 • Fall 2019
Announcements
MLSP Final Exam
Nov. 29, 1:30 PM – 4:30 PM, B308 Classroom.
MLSP Project Final Evaluation
Dec. 10, 9:30 AM – 1:00 PM, C313 (opposite the classroom).
Project Presentation Guidelines
Prepare a 10 minute presentation containing mostly your work done for the project including novelty. Presentation should clearly describe your contributions. A maximum of 10 slides is permitted for a single-person project, or 15 slides for a two-person project. Slides should be sent by email by Dec. 10th morning, 8 AM.
Project Report
A two page, two column report on the work done for the project should be submitted by Dec. 9th, latest by 4 PM. Students are encouraged to use the Overleaf template of ICML.
Logistics
Class Times
Mon & Wed
3:30 PM – 5:00 PM
Where: EE B308
Teaching Assistant
Prachi Singh
prachisingh aT iisc doT ac doT in
Lab: C 328 (2nd Floor)
Syllabus
- •Introduction to real world signals - text, speech, image, video.
- •Feature extraction and front-end signal processing - information rich representations, robustness to noise and artifacts.
- •Basics of pattern recognition, Generative modeling - Gaussian and mixture Gaussian models.
- •Discriminative modeling - support vector machines, neural networks and back propagation.
- •Introduction to deep learning - convolutional and recurrent networks, attention in neural networks, pre-training and practical considerations in deep learning, understanding deep networks.
- •Deep generative models - Autoencoders, Boltzmann machines, Adversarial Networks, Variational Learning.
- •Applications in NLP, computer vision and speech recognition.
Grading Details
15%
Assignments
20%
Midterm Exam
30%
Project
35%
Final Exam
Pre-requisites
Must - Random Process/Probability and Statistics
Must - Linear Algebra/Matrix Theory
Preferred - Basic Digital Signal Processing/Signals and Systems
Textbooks
B1
Pattern Recognition and Machine Learning
C.M. Bishop, 2nd Edition, Springer, 2011.
B2
Neural Networks
C.M. Bishop, Oxford Press, 1995.
B3
References
Deep Learning: Methods and Applications
Li Deng, Microsoft Technical Report.
Automatic Speech Recognition - Deep Learning Approach
D. Yu, L. Deng, Springer, 2014.
Various Published Papers and Online Material
Python Programming Basics
PDFSlides
Date
Topic
Slides
05-08-2019
Introduction to real world signals - text, speech, image, video. Learning as a pattern recognition problem. Examples. Roadmap of the course.
12-08-2019
Basics of Natural Language Processing - token, document and corpus. TF-IDF features. Language modeling. Smoothing and back-off. Introduction to audio signal processing. Discrete Fourier Transform, Short Term Fourier Transform.
Refs: Information Extraction Book (Ch. 6) - TF-IDF · Stanford Reading Material - Language Modeling
19-08-2019
Short-term Fourier Transform considerations. Mel-frequency cepstral coefficient (MFCC) features. Image processing - filtering, convolutions. Matrix derivatives. Dimensionality Reduction - Principal Component Analysis.
Refs: Columbia Univ. STFT Tutorial · PRML - Bishop (Appendix C)
21-08-2019
Unsupervised dimensionality reduction using Principal Component Analysis. Maximum variance formulation. Solution using eigenvectors of data covariance matrix. Minimum error formulation. Whitening and standardization.
Refs: PRML - Bishop (Ch. 12.1, 4.1.4)
26-08-2019
PCA for high dimensional data. Supervised dimensionality reduction using linear discriminant analysis (LDA). Fisher discriminant. Solution for 2 class LDA. Multi-class LDA. Comparison between PCA and LDA.
Refs: PRML - Bishop (Ch. 4.1.4, 1.5)
28-08-2019
Introduction to basics of decision theory. Inference and decision problems. Prior, likelihood and posterior. Maximum-a-posteriori decision rule for two class example. Decision theory for regression. MMSE estimation. Multi-variate Gaussian Modeling. Interpretation of Covariance. Diagonal and Full Covariance. Maximum Likelihood estimation of mean and covariance.
Refs: PRML - Bishop (Ch. 1.6)
04-09-2019
Short-comings of single Gaussian modeling. Introduction to mixture Gaussian modeling. Properties and parameters. Expectation Maximization algorithm - auxiliary function, proof of convergence.
Refs: Tutorial on GMMs · Proof of EM algorithm
09-09-2019
Expectation Maximization Algorithm for GMMs. Initialization using K-means. Other Considerations in GMMs. GMM example for unsupervised clustering.
Refs: EM algorithm for GMMs
11-09-2019
Assignment #1. Due on 23-09-2019. Analytical part submitted in class. Coding part submitted via mlsp19.iisc aT gmail doT com.
11-09-2019
GMM Initialization. Other Considerations in GMMs. Factor Analysis. EM Algorithm for Factor Analysis. Applications.
16-09-2019
Non-negative Matrix Factorization. Model formulation. Learning the parameters. Applications in audio source separation.
Refs: Lee Paper on NMFs · Audio Applications For NMF
23-09-2019
Linear Regression revisited. Maximum likelihood formulation and equivalence to least squares error. Regularized least squares.
Refs: PRML - Bishop (Ch. 3)
25-09-2019
Mid-Term #1.
27-09-2019
Decomposition of total loss into bias, variance and noise. Bias variance tradeoff in Regularized linear regression. Linear models for classification.
Refs: PRML - Bishop (Ch. 3, 4)
30-09-2019
Probabilistic Linear Models for Classification - Logistic Regression. Motivation and formulation for 2-class case and K-class case. Comparison with linear models for classification. Regularized least squares revisited - Primal and dual form. Optimization in the dual space. Introduction to kernel function and Gram matrix.
Refs: PRML - Bishop (Ch. 4, 6)
02-10-2019
Assignment #3. Due on 14-10-2019. Analytical part submitted in class. Coding part submitted via mlsp19.iisc aT gmail doT com.
02-10-2019
Properties of kernel functions. Rules for constructing kernels. The RBF kernel. Maximum margin classifiers - problem formulation for linearly separable case. Optimization fundamentals - primal and dual problems, strong duality, KKT conditions.
Refs: PRML - Bishop (Ch. 6, 7) · Introduction to Convex Optimization - Boyd (Ch. 5)
09-10-2019
Maximum margin classifiers (non-overlapping condition), primal and dual. Definition of support vectors. Complementary slackness and KKT conditions for SVM.
Refs: PRML - Bishop (Ch. 7)
14-10-2019
Maximum margin classifiers (overlapping condition), slack variables, KKT conditions. Applications of SVMs. Support Vector Regression.
Refs: PRML - Bishop (Ch. 7)
16-10-2019
Introduction to artificial neural networks - extension of kernel machines. Perceptron model. Multi-layer perceptron. Activation Functions. Input-output Mapping.
Refs: NNPR - Bishop (Ch. 3, 4)
18-10-2019
Forward pass in MLP. Backpropagation algorithm - recursion. Choice of hidden layer activation function.
Refs: NNPR - Bishop (Ch. 4)
21-10-2019
Cross entropy for two class. Expected cross entropy loss and posterior probability estimation. General condition on error function for outputs to be posterior probability. Weight learning - gradient descent method. Properties of gradient descent using quadratic approximation. Learning rate parameter.
Refs: NNPR - Bishop (Ch. 6, 7)
23-10-2019
Computational complexity in Gradient Descent. Definition of Jacobian and Hessian matrices. Choice of error function. Mean square and conditional expectation. Accelerating Gradient Descent Method, Momentum, Adagrad and Adam Optimizers.
Refs: NNPR - Bishop (Ch. 6)
25-10-2019
Assignment #4. Analytical part submitted in class and codes submitted via mlsp19.iisc aT gmail doT com (due 04-11-2019).
28-10-2019
Bias-variance tradeoff in neural networks. Improving generalization in deep learning. Regularization - weight decay, dropout strategy, training with noise. Early stopping. Committee of Neural networks.
Refs: NNPR - Bishop (Ch. 9)
30-10-2019
Introduction to deep learning. Depth versus Width. Intuition behind deep representation learning. Folding analogy of deep learning. Convolutional operations in deep neural networks.
Refs: On the number of linear regions of deep neural networks
04-11-2019
Convolutional Neural Networks. Computation of convolutions. Number of parameters. Advantages over deep neural networks. Pooling and subsampling. Backpropagation in convolution. Introduction to recurrence operations in modeling sequence data.
Refs: Deep Learning Book - Goodfellow et al. (Ch. 9)
06-11-2019
Recurrent Neural Networks. Backpropagation in time. Problem of vanishing gradients. Long short term memory networks. Various RNN architectures and applications.
Refs: Deep Learning Book - Goodfellow et al. (Ch. 10)
06-11-2019
Assignment #5. Analysis of coding part submitted as report in class. Codes submitted separately via mlsp19.iisc aT gmail doT com (due 20-11-2019).
08-11-2019
Understanding and Visualizing Neural Network Activations. Stochastic Neighborhood Embedding and t-distributed Stochastic Neighborhood Embedding (tSNE). Visualizing activations in image networks and audio networks.
Refs: tSNE paper
11-11-2019
Deep generative modeling - Restricted Boltzmann Machines (RBMs). Conditional independence property. RBM parameter learning. Positive and negative phase of learning. Intuitions behind contrastive divergence algorithm.
Refs: Deep Learning Book - Goodfellow et al. (Ch. 18, 19, 20)
13-11-2019
Autoencoders (AE). Variational autoencoders. Variational lower bound derivation. KL divergence derivation. Data generation with VAEs.
Refs: Kingma's paper · VAE Tutorial
18-11-2019
Generative Adversarial Networks. Intuition and Model Description. Theoretical bounds on the goals of GANs. GANs for data generation. Deep learning models for text. Word2vec model.
Refs: GAN paper
20-11-2019
Deep learning for speech. Speech recognition models. End-to-end speech modeling. Deep learning for computer vision. Image classification and segmentation. Summary of the Course.