EE5390 Course Information

Each student will be expected to

• attend classes regularly
• solve homework assignments, roughly one per week (collaboration is fine as long as it is explicitly declared, but you have to write the solutions in your own words), and participate in class (online and/or offline)
• solve in-class quizzes: roughly 2-3
• sit for a mid-term (tentative) and a final exam

• Slot E: Tuesdays 11:00-12:00, Thursdays 12:00-13:00 and Fridays 09:00-10:00
• Class venue: A-LH1
• Quiz schedule: TBA

Primarily, the material will be drawn from

• Elements of Information Theory, (EIT) Thomas M Cover and Joy A Thomas, second edition, Wiley inc. (Amazon link). Limited number of copies available in the university library. While the first edition of the book has all the material we need for the course, the second edition has a much expanded problem set.
• Lecture notes, as and when needed.

Other references:

• "A mathematical theory of communication", Claude Shannon, Bell systems Tech Journal, 1948. The classic paper that started this field.
• Information theory, inference, and learning algorithms, David MacKay (soft copy available for free on the author's website). A very nice book, but gives a very different flavour that what will be covered in the course. Suggested for additional reading.
• A student's guide to coding and information theory, Stefan Moser (amazon link).
• Information theory: coding theorems for discrete memoryless systems, Imre Csiszar and Janos Korner. An advanced textbook. Recommended reading after covering this course as well as EE6317.

Light reading:

• The information: A history, a theory, a flood James Gleick
• A mind at play: How Claude Shannon invented the information age Jimmy Soni and Rob Goodman. A biography of Claude Shannon.

Recap of basic concepts in information theory; Source coding theorem; Fixed versus variable length compression; Typical sets and the asymptotic equipartition property; Proof of the source coding theorem; Variable length compression: nonsingular, uniquely decodable and prefix-free codes; Kraft inequality; Optimal codes and bounds on expected length; Huffman codes and optimality; Shannon-Fano-Elias codes and optimality; Universal codes; Arithmetic codes; Lempel-Ziv codes and optimality.

Update: New videos for various topics added in the appropriate sections. These are not the actual lectures, but videos covering some essential concepts while omitting many details.