ISTA 311: Foundations of Information and

Inference

Colin Reimer Dawson

Last Revised March 21, 2014

1 Course Information

1.1 Course Description

In the information age, we are surrounded with data, much of which contains an
element of randomness, or "noise". Taking advantage of the abundance of informa-
tion requires the ability to make principled inferences and predictions under noisy
conditions.
A wide variety of inference and prediction can be understood in one way or another
as applications of conditional probability. In the first part of the course, we will
develop some of the mathematical tools we need before we can study conditional
probability rigorously. These include proof by induction, combinatorics, set algebra,
and probability spaces. In the middle section, we develop some theory of probability
and discrete random variables, with an emphasis on conditional probability and
Bayes rule. In the final segment, we look at some of the ways conditional probability
can be applied to study random environments. Topics in this section include basic
information theory, discrete state Markov chains (a kind of random process), and
basic elements of Bayesian statistics and decision theory.

1
1.2 Prerequisites

Grade of B or better in any one of ISTA 116, MATH 163, MATH 263, or equiva-
lent. One of CS 245, LING/MATH/PHIL 202, MATH 215 or MATH 243. These
requirements (particularly the second one) may be waived for students with strong
academic records with permission of the instructor. Students who do not meet the
formal course requirements should be very comfortable with mathematical functions
of one and two variables, should have at least some familiarity with basic concepts of
probability, and should be comfortable constructing logical arguments with explicit
justification of steps.

1.3 Locations and Times

Lecture: M/W 10:30-11:45, Social Sciences 312

Recitation: F 11-11:50 OR 12-12:50
Website: http://sista.arizona.edu/~cdawson/ista311/

1.4 Instructor Information

Colin Reimer Dawson, Ph.D.

Office: Gould-Simpson 850
Email: [email protected]
Office Hours: M 3-4, or right after class, or by appointment

1.5 Readings

I havent found a single textbook that covers everything we need at the right level.
Therefore, the primary readings will be my lecture notes, which I will update as we
go along.
I also recommend the following book chapters, which have complementary cover-
age:
Chapter 1 from:
Jaynes, E.T. (2003). Probability Theory: The Logic of Science. Cambridge
University Press. (Available for free electronically from UA Library)

2
Chapters 4-8 from:
Bolstad, William M. (2007). Introduction to Bayesian Statistics. Hoboken, New
Jersey: John Wiley and Sons.
Chapters 1-6 and 10 from:
Applebaum, David (2008). Probability and Information: An Integrated Approach,
2nd Edition. Cambridge University Press.

1.6 Course Goals

This course has two main purposes: First, after taking this course, students should
be accustomed to reasoning probabilistically, incorporating uncertainty in every day
inferences, and should be able to translate real-world problems into mathemati-
cal language. Second, students should become comfortable with the structure of
mathematical arguments, and should be able to assess whether conclusions follow
deductively from definitions and premises. After completing this course, students
will be prepared for 400-level classes in probability, statistics, artificial intelligence
and machine learning.

1.7 Format of the Class

The required meetings consist of two 75-minute lectures and one 50-minute discussion
/ problem-solving session per week. New topics will be presented in lectures. In
discussion/problem sections we will work through examples, and students will present
their solutions to homework exercises.

1.8 A Note on Lectures

The word lecture has its roots in the word reading. As such it is a bit of a
misnomer in an age when students have access to reading materials on their own.
Class meetings should not be a mechanism for the one-way delivery of information
thats what the reading is for. Therefore students are expected to do the relevant
reading before each class, so that class time can be spent interactively: reinforcing
the reading, clarifying difficult concepts, and discussing subtleties.

3
2 Course and Grading Policies

2.1 Components of the Grade

Grades are based on attendance and participation, approximately weekly homework

assignments, two takehome midterms, an in class final exam, and an optional final
project.
Component Weight
Letter Grade Point Range
Attendance/Participation 200 pts
A 875-1000
Homework 300 pts
B 725-874
Exam 1 150 pts
C 575-724
Exam 2 150 pts
D 500-574
Final Exam 200 pts
E 0-499
Total 1000 pts

Grade Disputes. Disputes about grades on a particular assignment or exam will

be entertained for one week from the day the assignment is returned, or one week
from the date of the last exam, whichever is sooner. These will be resolved by re-
grading the entire assignment or exam. Note that this can result in a lower grade in
the event that new mistakes are discovered. The final exam will be graded and made
available for review by students within 48 hours of its completion, to allow time for
any requested regrades.

2.2 Attendance and Participation

Most people find the material in this class quite challenging. Practicing using con-
cepts and techniques with homework problems is critical for internalizing the ma-
terial; but it is not enough on its own, particularly since it is sometimes possible
to complete homework mechanically, with only a shallow understanding of whats
going on. A deeper understanding occurs when one has to try to explain a concept
or problem to someone else. The goal of the discussion sections is for students to
engage with each other in a way that allows the kind of understanding which is
only possible through explaining to take hold. Ideally this would occur naturally;
however, sometimes concrete incentives are needed. As such 20% of the final grade
comes from attending and participating during class and discussion sections. Stu-

4
dents may miss up to 3 lectures and 3 discussion sections without adversely affecting
their grade.

2.3 Individual Homework Assignments

There will be 8-10 paper-and-pencil conceptual and mathematical problems, along

with occasional computational exercises, assigned each week, pertaining to the spe-
cific material discussed in lectures. Of these, a subset will be presented by students
during the discussion section, and the remainder will be due for homework the fol-
lowing week. The two lowest homework scores are dropped.

Collaboration Policy. Students are encouraged to work together, both in dis-

cussion sections / office hours and otherwise, to understand problems and general
approaches for solutions. However, final write-ups of graded exercises must
be done individually. Any collaboration that takes place outside section or office
hours must be identified, along with the nature of the collaboration (e.g., we worked
together, she helped me, I helped him).

Late Policy. Assignments are due at the end of class or discussion section on the
posted date. Limited extensions will be entertained, but must be requested before
the assignment is due. The first request by a given student will automatically be
granted; the probability of rejecting an extension increases with each subsequent
request.

2.4 Exams

Three exams will be given at roughly equal intervals. The first two will be takehome
exams, to be completed individually over a short time period (about 48 hours) with-
out collaboration or use of outside sources. The final exam will be in class on the
designated day during finals week.
All exams will be similar in format to the homework: some straightforward calcula-
tion problems, some problems that require representing a scenario mathematically,
and some relatively simple proofs.

5
Make-up Policy. Exams may only be made up in case of a serious, unan-
ticipated emergency. The student or an appointed proxy must contact the in-
structor in advance of the exam. Clearly worded documentation, from a doctor,
dean, etc., demonstrating physical inability to take the exam at the appointed time,
will be required to grant a make-up appointment. No make-ups will be granted
for personal reasons, including travel or personal hardship.

2.5 Optional Final Project

Students have the option of completing a term project at the end of the semester. The
grade on the project may be used to substitute for one of the two takehome midterm
exams, or as a 50 point extra credit assignment, whichever is more beneficial.
There are two options for this project. The first consists of identifying 8-10 exercises
that were neither assigned for a grade nor discussed in recitation, and writing them
up in the manner of a long homework assignment. Problems identified with asterisks
as challenging count double.
The second option involves identifying a real-world phenomenon that can repre-
sented using a probabilistic model with at least one variable that cannot be directly
observed, and at least one that can, such that the distribution of the latter depends
on the value of the former. For example, the unobserved variable could be a pa-
rameter in the distribution of the observed variable; it may be part of the transition
matrix in a Markov chain; or it may simply correspond to which in a set of compet-
ing hypotheses accurately describes the phenomenon. The goal is to (a) define the
relevant variables and explain how they relate, (b) identify a set of questions about
the problem that can be addressed with data, (c) gather or find some data relevant
to the chosen questions, and (d) attempt to answer the question either analytically
(solving equations on paper) or by computer simulation.
For those who took ISTA 116 with me, the second option is similar to the paper
assigned in that class, but taking a Bayesian perspective.

3 University Policies
Classroom Behavior. Students are expected to behave respectfully toward each
other and to the instructor. Disrespectful behavior includes the use of cell phones or

6
other electronic devices in the classroom during class hours.
The Arizona Board of Regents Student Code of Conduct is here: http://dos.web.
arizona.edu/uapolicies/scc5308abcd.html#sccphilosophy
ABOR Policy 5-308, prohibits threats of physical harm to any member of the Uni-
versity community, including to oneself. See: http://policy.web.arizona.edu/
~policy/threaten.shtml.

Special Needs and Accommodations. Students who need special accommoda-

tion or services should contact the
Disability Resources Center
1224 East Lowell Street, Tucson, AZ 85721
(520) 621-3268
FAX (520) 621-9423
email: [email protected]
web: http://drc.arizona.edu/.
You must register and request that the Center or DRC send official notification
of your accommodations needs as soon as possible. Please plan to meet with the
instructor by appointment or during office hours to discuss accommodations and how
the course requirements and activities may impact your ability to fully participate.
The need for accommodations must be documented by the appropriate office.

Student Code of Academic Integrity. Students are encouraged to share intel-

lectual views and discuss freely the principles and applications of course materials.
However, graded work/exercises must be the product of independent effort unless
otherwise instructed. Students are expected to adhere to the UA Code of Academic
Integrity as described in the UA General Catalog. See: http://dos.web.arizona.
edu/uapolicies/.

Confidentiality of Student Records. See http://www.registrar.arizona.

edu/ferpa/default.htm

Subject to Change Statement. Information contained in this syllabus, other

than the grade and absence policy, may be subject to change by the instructor, with
advance notice.

7
4 Approximate Schedule of Lecture Topics

Topic Dates Reading

Introduction and Preliminaries Jan. 15 J1 through
Introducing the Robot
MLK DAY (NO CLASS) Jan 20
Structure of Proofs; Jan 22, 27 LN 2, A1
Mathematical Induction
Counting Techniques Jan 29, Feb 3LN 3.1, A2
The Algebra of Sets Feb 3, 5 LN 3.2, A3.1-2
Probability Spaces Feb. 10, 12, 17
LN 4, B4.1-3, A4.1-2
J1: Basic Desiderata
Conditional Probability Feb 17,19,24,26 LN 5, B4.4-8,
& Bayes Theorem A4.3-4
Discrete Random Variables Mar 3,5,10,12 LN 6.1-4, B5.1, A5.1-2
SPRING BREAK Mar 17, 19
Expectation and Variance Mar 24 LN 6.5, B5.1, A5.3
Covariance and Conditional Distributions Mar 26, 31 LN 7.1-2, B5.6-7, A5.4
Collections of RVs Mar 31, Apr 2 LN 7.3-4, A5.5-5.7, B5.3,5
Bayesian Inference about Apr 2, 7 B6
a discrete parameter
Continuous Priors Apr 9, 14 LN 9.1-2, B7
Bayesian inference with Apr 16, 21 LN 9.3, B8.1-4, B10
a conjugate prior
Decision Theory Apr 23, 28, 30 LN 10
Information & Entropy May 5,7 LN 11, A6.1-3
FINAL EXAM (10:30-12:30 in SS 312) May 14
Note: LN refers to lecture notes, J refers to the Jaynes book, B to the Bolstad book
and A to the Applebaum book. The numbers are chapters and sections: for example,
B4.1-3 means section 4.1 through 4.3 from Bolstad.

8
5 Discussion Schedule and Deadlines

Date Discussion Topic Homework Due

Jan 17 NO DISCUSSION SECTION
Jan 24 Direct Proofs
Jan 31 Proof By Induction
Feb 7 Counting and Sets 1: Proofs
Feb 14 Probability 2: Counting and Sets
Feb 21 Conditional Probabilty 3: Probability
Feb 28 Conditional Prob. & Bayes TAKEHOME EXAM 1
Mar 7 Bayes / Discrete RVs 4: Conditional Probability
Mar 14 Discrete RVs 5: Bayes/Independence/Discrete RVs
Mar 21 SPRING BREAK
Mar 28 Expectation / (Co)variance 6: Discrete RVs / Expectation
Apr 4 Covariance / Conditional Dists 7: Expectation / (Co)variance
Apr 11 Binomial/Poisson 8: Covariance/Conditional Dists
Apr 18 Continuous Dists 9: Binomial/Poisson / Discrete Bayes
Apr 25 Conjugate Priors / Decision Theory 10: Continuous Dists
May 2 Decision/Info Theory 11: Conjugate Priors / Dec. Theory
May 7 TAKEHOME EXAM 2
May 9 OPTIONAL REVIEW (11-12:50 in GS 906)
May 14 FINAL EXAM (10:30-12:30 in SS 312)