Statistical Learning Theory (2025)
Table of Contents
Welcome to EE6327 (Statistical Learning Theory).
This is the official webpage of the SLT course for 2025. This is a 3-credit course and all classes will be conducted in person. Online resources and references will be shared here. Much of our online interaction (homework submissions, announcements, off-classroom discussions, etc) will be on Google classroom. Invites will be sent to registered students by the first week of the course. If you have not received an invite by the second class, please send me an email.
As the title suggests, this course serves as an introduction to the mathematical foundations of statistical learning, and the focus will be on supervised learning. We will mathematically formulate the problem of supervised learning, and derive fundamental bounds on what can and cannot be achieved.
The course will be mathematical in nature, and some of the assignment questions will involve programming.
Prerequisites
- Strong foundation in probability and random processes
- Comfort with programming in python. We will mainly use the numpy library.
1. Assessment (tentative):
Each student will be expected to
- attend classes and participate actively
- solve homework assignments
- solve 4-5 exams
| Exam 1 | 20% | |
| Exam 2 | 20% |
If you are planning to audit the course, you must secure a regular pass grade to be eligible for an AU grade.
2. Instructor:
| Name | Dr. Shashank Vatedka |
| shashankvatedka@ee.iith.ac.in | |
| Office | EE616, EECS building |
3. Class timings:
- Slot R
- Class venue: A-LH1
4. Primary references:
We will primarily use the following material for the course:
- Understanding Machine Learning: From Theory to Algorithms by Shai-Shalev Schwartz and Shai-Ben David (free pdf from author website)
- Foundations of Machine Learning by Mehryar Mohri, Afshin Rostamizadeh and Ameet Talwalkar
- Learning Theory from First Principles by Francis Bach (pdf)
References for concentration inequalities (which we will use for the first part of the course)
- High-dimensional Probability by Roman Vershynin
- Concentration Inequalities by Stephane Boucheron, Gabor Lugosi and Pascal Massart
- “Introduction to statistical learning theory” by Olivier Bousquet, Stephane Boucheron and Gabor Lugosi
To recap basics in probability and random processes:
- Probability, Random Variables and Stochastic Processes, Athanasios Papoulis and Unnikrishna Pillai
- Probability with Engineering Applications by Bruce Hajek
- Random Processes for Engineers by Bruce Hajek
5. Tentative list of topics
5.1. Concentration inequalities
- Introduction, Review of basics on probability and random variables
- Sample space, probability measure - need for sample space for uncountable spaces
- Random variables, pmf, cdf and pdf, examples of common distributions
- Expectation, variance, higher moments and MGF
- Volume of n-ball and concentration, Stirling’s approximation, volume of hamming ball and l1 ball
- Sequences of random variables: iid and dependent sources
- Basic inequalities: Markov and Chebyshev inequalities
- Exponential concentration: Chernoff bound, log mgf and the best bounds
- Concentration for sums of iid random variables
- Examples: Gaussian, Poisson and Bernoulli
- Subgaussian random variables
- Bounded random variables: Hoeffding’s inequality
- Subgamma random variables
- Bennett’s inequality
- Bernstein’s inequality
- Example: Dimensionality reduction and the Johnson-Lindenstrauss lemma
- Bounding the variance: Efron-Stein inequality
- A concentration inequality for functions with bounded differences
- Empirical Risk Minimization
5.2. Foundations of supervised learning
- Fundamental principles of statistical learning
- Mathematical model and assumptions
- Binary classification
- Empirical risk minimization
- PAC learning
- Learning via uniform convergence
- Bias-complexity tradeoff
- VC dimension and the fundamental theorem of PAC learning
- Nonuniform learning
- Computational complexity of learning
5.3. Kernel methods and other topics
- Linear classification and SVM
- Kernel methods: PSD kernels and representer theorem
- PCA and Kernel PCA
- Boosting (if time permits)
6. Class notes and recordings
Class notes will be uploaded regularly
7. Academic honesty and plagiarism
Students are encouraged to discuss with each other regarding class material and assignments. However, verbatim copying in any of the assignments or exams (from your friends or books or an online sources) is not allowed. This includes programming assignments. It is good to collaborate when solving assignments, but the solutions and programs must be written on your own. Copying in assignments or exams will result in a fail grade.
See this page (maintained by the CSE department), this page, and this one to understand more about plagiarism.