Introduction to Statistical Learning Theory (2024)
Table of Contents
Welcome to EE5604 (Introduction to Statistical Learning Theory).
This is the official webpage of the Intro to SLT course for 2024. This is a 1-credit course and all classes will be conducted in person. Online resources and references will be shared here. Much of our online interaction (homework submissions, announcements, off-classroom discussions, etc) will be on Google classroom. Invites will be sent to registered students by the first week of the 3-4 segment. If you have not received an invite by the second class, please send me an email.
As the title suggests, this course serves as an introduction to the mathematical foundations of statistical learning, and the focus will be on supervised learning. We will mathematically formulate the problem of supervised learning, and derive fundamental bounds on what can and cannot be achieved.
The course will be mathematical in nature, and some of the assignment questions will involve programming. This course is a firm prerequisite for EE5605 Kernel Methods.
Prerequisites
- Strong foundation in probability and random processes
- EE5603/AI2200 Concentration Inequalities
- Comfort with programming in python. We will mainly use the numpy library.
1. Assessment (tentative):
Each student will be expected to
- attend classes and participate actively
- solve homework assignments
- solve 2 exams
| Mid-term exam | 40% | Week of 25th Sept (tentative) |
| Final exam | 60% | 17th October 2024 |
If you are planning to audit the course, you must secure a regular pass grade to be eligible for an AU grade.
2. Instructor:
| Name | Dr. Shashank Vatedka |
| shashankvatedka@ee.iith.ac.in | |
| Office | EE616, New EECS building |
3. Class timings:
- Slot P: Mondays 14:30-16:00, Thursdays 16:00-17:30
- Class venue: A-221
4. Primary references:
We will primarily use the following material for the course:
- Understanding Machine Learning: From Theory to Algorithms by Shai-Shalev Schwartz and Shai-Ben David (free pdf from author website)
- Foundations of Machine Learning by Mehryar Mohri, Afshin Rostamizadeh and Ameet Talwalkar
- Learning Theory from First Principles by Francis Bach (pdf)
References for concentration inequalities (which we will use to some extent in the course)
- Concentration Inequalities by Stephane Boucheron, Gabor Lugosi and Pascal Massart
- High-dimensional Probability by Roman Vershynin
- “Introduction to statistical learning theory” by Olivier Bousquet, Stephane Boucheron and Gabor Lugosi
To recap basics in probability and random processes:
- Probability, Random Variables and Stochastic Processes, Athanasios Papoulis and Unnikrishna Pillai
- Probability with Engineering Applications by Bruce Hajek
- Random Processes for Engineers by Bruce Hajek
5. Tentative list of topics
- Fundamental principles of statistical learning
- Mathematical model and assumptions
- Binary classification
- Empirical risk minimization
- PAC learning
- Learning via uniform convergence
- Bias-complexity tradeoff
- VC dimension and the fundamental theorem of PAC learning
- Nonuniform learning
- Computational complexity of learning
6. Class notes and recordings
Class notes will be uploaded regularly to this folder.
Some class recordings will be uploaded to this YouTube playlist.
7. Academic honesty and plagiarism
Students are encouraged to discuss with each other regarding class material and assignments. However, verbatim copying in any of the assignments or exams (from your friends or books or an online sources) is not allowed. This includes programming assignments. It is good to collaborate when solving assignments, but the solutions and programs must be written on your own. Copying in assignments or exams will result in a fail grade.
See this page (maintained by the CSE department), this page, and this one to understand more about plagiarism.