Erik Learned-Miller
elm@cs.umass.edu
545-2993

Prerequisites

Stats 515 or equivalent. Basic college-level probability. Basic programming ability in C++, Matlab, or Java.

• Required text: Elements of Information Theory, Second Edition, by Thomas Cover and Joy Thomas.

• NOTE THAT THIS IS THE SECOND EDITION, NOT THE FIRST EDITION!!! If you have a copy of the first edition of this book, you might be able to use it, as the second edition is not too different. However, you should at least have access to the second edition to check for differences, page number references, problem set numbers, and other differences. I will not answer questions about the differences between the books, as I do not have time to figure this out. You should have at least one friend with a copy of the second edition if you want to use the first edition, so that you can find appropriate page number references and so on.

Test 2 Review

Minimum entropy joint alignment

Mutual Information Alignment

David Mackay's Information Theory book (electronic edition is free and on the web)

Guest Lecturer John Fisher's slides

Guest Lecturer Hossein Pishro Nik's slides

Assignment 3 Solutions (not quite complete)

Problem sets are due at any time on the day indicated on the course web page. If you turn them in after class, please slide them under the door of my office, or put them in my faculty mailbox. The latest you can turn in a problem set is midnight on the day due. The fact that you cannot get into the building after hours does not count as an excuse. If you're worried about that, turn it in earlier. I may take off 50% for problem sets turned in late.

If you want written feedback on your homework (such as how many points per problem you got), please hand it in on paper, rather than as an email.

This course will introduce the basic concepts of Information Theory: entropy, relative entropy, mutual information, channel capacity, and rate distortion. Applications, rather than proofs, will be emphasized. In addition, the statistical problem of computing information theoretic quantities from data will be emphasized. Non-parametric and semi-parametric statistical models will also be covered.