Massachusetts Institute of Technology

Department of Electrical Engineering and Computer Science

6.437 Inference and Information
Spring 2022

Information Sheet

Lecturers
Polina Golland ([email protected])
Lizhong Zheng ([email protected])
Teaching Assistants
Theodore (Ted) Grunberg ([email protected])
Michael Diao ([email protected])
Molin Zhang ([email protected])
Administrative Assistant
Sheila Sharbetian ([email protected])

Lectures: Tuesday and Thursday, 9:30am–11am

Room 34-101
Recitations: Friday, 1–2pm or 2–3pm
Room 32-155
Office Hours: Days, times and locations posted on course site.

Welcome to 6.437 !
This course offers an immersive graduate-level introduction to the principles of statis-
tical inference, with an emphasis on information theoretic perspectives. As such, it is
a core graduate subject for students in the relevant subfields both within and beyond
Course 6 (EECS). The material in this course constitutes a common foundation for
work in, for example, machine learning and data science, artificial intelligence, signal
processing, computer vision and computational imaging, communication, bioinfor-
matics and neuroscience, finance and data analytics, and networks, robotics, and
control.
It is worth stressing that 6.437 is an introductory graduate subject: it is not
an advanced graduate subject for students who have already mastered statistics,
estimation and decision theory, and information theory, and now want to understand
such material at an even more sophisticated level. That said, by its structure, 6.437
will ultimately reveal connections among these fields, and provide background for
more advanced treatments.
Ultimately, the course is about teaching you contemporary approaches to—and
perspectives on—problems of statistical inference and learning. The development of
the material that forms the basis for this subject has historically been strongly driven
by applications, and continues to be. However, our focus in the course will not be on
these applications—which form the basis for entire courses of their own—but rather
on the common problem-solving frameworks that they share. Nevertheless, we will
cite various relevant applications as we develop the material and sometimes extract
simplified illustrative examples from these contexts.
As with other core subjects, 6.437 has lectures and recitations, which are designed
to complement each other. Recitations begin the first week of classes. There are
two possible recitation times from which you can choose. Select to attend whichever
suits your schedule best; if strong imbalances result, we will make adjustments. In
addition, the course staff will hold several scheduled office hours throughout the week,
the details of which are provided on the class website. The lectures, recitations and
office hours will be in-person! These are great opportunities for us to get to know
you, and to tailor our teaching toward your specific questions. We encourage you to
participate as much as you can.

Prerequisites
The official prerequisite is 6.008, 6.041/6.431, 18.05, 18.600, or 6.436. The effective
prerequisite is fluency with basic quantitative probabilistic reasoning and analysis,
together with the kind of mathematical maturity that often comes from taking at least
one higher level undergraduate subject that has a significant mathematical component.
As such, a student having had 6.436 would be sufficiently well prepared, while a student
having only had 6.008, 6.041/6.431, 18.05, or 18.600 and no subsequent subjects of a
strong mathematical flavor would likely need additional preparation. As an example,
having had one of these subjects together with an introductory subject in analysis
(e.g., 18.100), would be sufficient, but not necessary, preparation.
When in doubt, students whose undergraduate degrees are not from MIT should
consult the staff to determine if they have had subjects that are effectively equivalent
to the official prerequisites.

Reading
There are extensive and detailed course notes for the class, which you will access
by clicking on the Files tab in the navigation bar of the class website and going to
the “Course Notes” section. The notes will be posted one section at a time over the
course of the semester, with each installment posted in advance of its corresponding
lecture(s). These notes are under perpetually active development, and as such are
necessarily rough in places and contain bugs, which we will count on you to help us
catch. Also, occasionally a section of the notes will be marked with a “∗” after its
number. Such sections constitute bonus material and thus optional reading; read them
as your interests dictate and your time permits.

2
 More generally, if you are interested in further reading, either to strengthen your
background, reinforce some of the concepts from lecture, or to probe some topics in
more detail, you might want to take a look at the additional references on the course
website, which have been selected with some care. To find them, click the Files tab in
the navigation bar of the class website and go to the “Further Reading” section. In
particular, you’ll find several papers containing a variety of useful insights, which are
worth the effort to work through.
We emphasize that the course notes do not replace the lectures and recitations.
The notes will be necessarily incomplete, and many of the perspectives, interpretations,
and insights will be developed in class. Conversely, lectures and recitations are not a
substitute for the notes. Indeed, we will defer many details of our development to the
reading materials. Thus, you should view class and the written materials as strongly
complementary.

Problem Sets and Exercises

There will be 10 problem sets. Problem sets are due by 5pm on the Tuesdays (except
for the final problem set, which is never due). You will submit your problem set
by scanning it and uploading to gradescope via the class web site. Problem set
solutions will be available on the class website shortly after the due dates and times,
and submissions received after the solutions are posted will be recorded as completed,
but not graded.
While you should do all the assigned problems, only a randomly chosen subset
will actually be graded. You will find some problems in the problem sets marked as
“practice”. These are not required, but you might find it helpful to work through them
if you are looking for more practice working with the concepts introduced in class.
Don’t be misled by the relatively few points assigned to homework grades in the
final grade calculation! While the grade you get on your homework is only a minor
component of your final grade, working through (and, yes, often struggling with at
length!) the homework is a crucial part of the learning process and will invariably
have a major impact on your understanding of the material. Some of the problem sets
involve a Python component, to help you explore different aspects of the material.
Many students find collaborating with one or two classmates on the problem
sets helpful, and we very much support and encourage this. Just remember that
your writeup (and all code) needs to be your own, and you need to identify your
collaborators in your writeup. We provide several different ways for finding partners,
discussed later in this document.
To help you get started with the problem sets, there will be an accompanying
exercises handout posted on the website at the same time. These exercises are simple
worked examples that illustrate some of the basic calculations that the problem sets
will, in turn, build on. You’re free to use these exercises any way you find helpful to
you in preparing for the homework.

3
Project
In addition to the problem sets, the class will include a two-part Python-based class
project. Part I will be a guided exercise, while the continuation Part II will be a more
open ended challenge. The project is intended to be educational, interesting, and fun!
The project will go out on Tuesday, April 12, and Part I will be due the following
Tuesday. There will be no problem set issued during Part I of the project. Part II
will be due on Friday, May 6. You may want to make a note of the project dates in
your schedule now, to help you with planning your time and coordinating with other
classes during the semester.
The project details will be announced closer to the time it goes out, but plan on
an interesting and engaging experience with the material.

Exams
There will be two quizzes in the subject. The midterm quiz will be held on Wednesday,
March 16, 7:30-10:30pm (50-340). The final quiz will be held in the final exam week.
The exact date of the final quiz and the locations of both quizzes will be announced
later in the semester. The quizzes will be designed to require 2 hours of effort, but
we’ll use the three hour format to minimize the effects of time pressure. The quizzes
will both be closed book. You will be allowed to have two 8.5 × 11-inch sheet of notes
(both sides) for the midterm quiz, and four 8.5 × 11-inch sheets of notes for the final
quiz.

Class Grade
The final grade in the class is based upon our best assessment of your understanding
of the material during the semester. Roughly, the weights used in grade assignment
will be:
Midterm Quiz 40%
Final Quiz 40%
Project 5%
Homework 15%
with an additional property that if you do better on the final quiz than the midterm
quiz, and you have done all the problem sets and the project, then the
midterm quiz will not count, i.e., the midterm quiz can only help you if you are doing
all the homework. Also, the project must be completed to receive a grade in the
subject.
As always, other factors such as contributions to the lecture discussion and other
interactions can make a significant difference in the final grade.

4
Meeting and Working with Your Classmates
One of the most important aspects of the experience of taking a subject at MIT like
6.437 is the community of people you are taking the subject with. This community
forms and bonds naturally through side conversations and interactions among you in
and outside of the classroom, as you navigate the semester together. And out of that
shared experience emerge collaborations and friendships that endure long after the
subject is over.
We encourage you to join Pset Partners, which is an online system developed at
MIT that matches students to small groups for working on problem sets, based on
the preferences specified by each student. The first matching round will take place on
February 4.

Online Platforms and Resources

We will make use of multiple online platforms and resources in 6.437.

Canvas, Athena, Certificates

We will post information and handouts on the 6.437 class website, which resides on
the Cavas platform:

https://canvas.mit.edu/courses/13719

You should first make sure that you have an active Athena account (by visiting http:
//ist.mit.edu/support/accounts/ if necessary) as well as a personal certificate (by
visiting https://ca.mit.edu/ca/ if necessary). If you have problems or if you are
not a regular MIT student, please contact one of the TAs for assistance.
You will need to have a valid certificate and be on the official course list to access
the website. If you have pre-registered for 6.437, this should have already been set up;
just double-check that you can access the website (try to download a handout, for
example). Otherwise, contact one of the TAs.

Gradescope
You will submit homework and quizzes on Gradescope, which you can access via the
Gradescope tab in the navigation bar of the class website and logging in with your
MIT credentials.

Piazza
We will use Piazza to make announcements and as an online discussion forum, which
you can access (and enroll yourself in) via the Piazza Forum tab in the navigation bar of
the class website. Through Piazza you can post questions or comments (anonymously

5
to your peers) about any aspects of the material, which your peers and the staff can
respond to. It is a popular resource in the course for clarifying potential points of
confusion in class notes, problem sets, recitation handouts, the project, etc.

Email
Finally, if you have administrative questions during the term, you can send email to

[email protected]

We will be glad to help you, but please be sure that your question isn’t already
answered in this handout before you email us!

6
 Tentative Syllabus and Schedule
Date Topic Due Out

T 2/1 L1: Introduction and overview PS1 (L1,L2)

R 2/3 L2: Bayesian hypothesis testing
T 2/8 L3: NonBayesian decision theory PS1 PS2 (L3,L4)
R 2/10 L4: Minimax decision theory
T 2/15 L5: Bayesian estimation and geometry PS2 PS3 (L5-7)
R 2/17 L6: NonBayesian parameter estimation
T 2/22 Monday schedule – no class
R 2/24 L7: Exponential families
T 3/1 L8: Sufficient statistics PS3 PS4 (L8,L9)
R 3/3 L9: The EM algorithm
F 3/4 Add Date
T 3/8 L10: Inference as decision PS4 PS5 (L10,L11)
R 3/10 L11: Information geometry
T 3/15 L12: Modeling as inference PS5 PS6 (L12-14)
W 3/16 Midterm Quiz (through L11 and PS5)
7:30-10:30pm (50-340)
R 3/17 No Class
3/21-25 Spring Break
T 3/29 L13: Extensions to continuous parameters
R 3/31 L14: Max entropy priors and conjugate priors
T 4/5 L15: Alternating projections PS6 PS7 (L15,L16)
R 4/7 L16: Approx. inference: stochastic
T 4/12 L17: Markov chain Monte Carlo methods PS7 Project
R 4/14 L18: Approx. inference: deterministic
T 4/19 L19: Typical sequences, large deviations PI PS8 (L17-19)
Drop Date
R 4/21 L20: Method of Types and Sanov’s Theorem
T 4/26 L21: Asymptotics of hypothesis testing PS8 PS9 (L20-22)
R 4/28 L22: Asymptotics of estimation
T 5/3 L23: Asymptotics of inference and universality PS9 PS10† (L23-24)
R 5/5 L24: Nested classes and model order selection
F 5/6 PII
T 5/10 L25: TBA
5/13-5/18 Final Quiz (through L25 and PS10)
† Never due.