\documentclass{article}

\usepackage[margin=1in]{geometry}
\usepackage{amsmath, amsfonts}
\usepackage{enumerate}
\usepackage{graphicx}
\usepackage{titling}
\usepackage{url}
\usepackage{xcolor}
\usepackage[colorlinks=true,urlcolor=blue]{hyperref}
\usepackage{bbm}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Commands for customizing the assignment %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newcommand \duedate {5 p.m. Friday, April 3, 2015}

\title{
  10-601 Machine Learning: Homework 6\\
  \vspace{0.2cm}
  \large{
    Due \duedate{}
  }
}

% Un-comment one of these lines to either include or exclude the solutions.

\newcommand \solution [1]{}
%\newcommand \solution [1]{{\sf \color{blue} Solution: #1}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Useful commands for typesetting the questions %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newcommand \expect {\mathbb{E}}
\newcommand \mle [1]{{\hat #1}^{\rm MLE}}
\newcommand \map [1]{{\hat #1}^{\rm MAP}}
\newcommand \argmax {\operatorname*{argmax}}
\newcommand \argmin {\operatorname*{argmin}}
\newcommand \code [1]{{\tt #1}}
\newcommand \datacount [1]{\#\{#1\}}
\newcommand \ind [1]{\mathbb{I}\{#1\}}
\newcommand \bs [1]{\boldsymbol{#1}}
\newcommand \R {\mathbb{R}}
\newcommand \indic[1] {\mathbbm{1}\left\lbrace #1\right\rbrace}

%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document configuration %
%%%%%%%%%%%%%%%%%%%%%%%%%%

% Don't display a date in the title and remove the white space
\predate{}
\postdate{}
\date{}

% Don't display an author and remove the white space
\preauthor{}
\postauthor{}
\author{}

%%%%%%%%%%%%%%%%%%
% Begin Document %
%%%%%%%%%%%%%%%%%%

\begin{document}

\maketitle

\section*{Instructions}
\begin{itemize}
\item {\bf Late homework policy:} Homework is worth full credit if
  submitted before the due date, half credit during the next 48 hours,
  and zero credit after that.  You {\em must} turn in at least $n-1$
  of the $n$ homeworks to pass the class, even if for zero credit.
\item {\bf Collaboration policy:} Homeworks must be done individually,
  except where otherwise noted in the assignments. ``Individually''
  means each student must hand in their own answers, and each student
  must write and use their own code in the programming parts of the
  assignment. It is acceptable for students to collaborate in figuring
  out answers and to help each other solve the problems, though you
  must in the end write up your own solutions individually, and you
  must list the names of students you discussed this with.  We will be
  assuming that, as participants in a graduate course, you will be
  taking the responsibility to make sure you personally understand the
  solution to any work arising from such collaboration.
\item {\bf Online submission:} You must submit your solutions online
  on
  \href{https://autolab.cs.cmu.edu/courses/10601b-s15/assessments/homework6}{autolab}. We
  recommend that you use \LaTeX{} to type your solutions to the
  written questions, but we will accept scanned solutions as well. On
  the Homework 6 autolab page, you can download the
  \href{https://autolab.cs.cmu.edu/courses/10601b-s15/assessments/homework6/attachments/46}{template},
  which is a tar archive containing a blank placeholder pdf for the
  written questions. Replace each pdf file with one that contains
  your solutions to the written questions. When you are ready to submit, create a new tar archive of the
  top-level directory and submit your archived solutions online by
  clicking the ``Submit File'' button. You should submit a single tar
  archive identical to the template, except with the blank pdfs replaced by your
  solutions for the written questions. You are free to submit as many
  times as you like.
  \textbf{\emph{DO NOT}} change the name of any of the files or
  folders in the submission template. In other words, your submitted
  files should have exactly the same names as those in the submission
  template. Do not modify the directory structure.
\end{itemize}


\section*{Problem 1: Expectation Maximization}
Mixture models are helpful for modeling unknown subpopulations in data. If we have a collection of data points $X = \lbrace X_1,...,X_n\rbrace$, where each $X_i$ is drawn from one of $K$ possible distributions, we can introduce a discrete-valued random variable $Z_i\in\lbrace 1,...,K\rbrace$ that indicates which distribution $X_i$ is drawn from. In lecture, we discussed Gaussian mixture models, where each sample $X_i$ is drawn from a Gaussian distribution according to the value of its mixture component. This question deals with the \emph{categorical} mixture model, where each observation $X_i$ is a discrete value drawn from a categorical distribution, rather than a continuous value drawn from a Gaussian distribution. The parameter for a categorical distribution is a $K$-dimensional vector $\pi$ that lists the probability of each of $K$ possible values (therefore, $\sum_k \pi_k = 1$). For example, suppose our categorical mixture model has 3 underlying distributions. Then, each $Z_i$ could take on one of three values, $\lbrace 1,2,3\rbrace$, with respective probabilities $\lbrace\pi_1,\pi_2,\pi_3\rbrace$; equivalently, $Z_i\sim\text{Categorical}(\pi)$, where $\pi = \lbrack \pi_1,\pi_2,\pi_3\rbrack^T\in\R^3$. The observation $X_i$ is then generated from another categorical distribution, depending on the value of $Z_i$. Equation~(\ref{eqn:gen-cmm}) summarizes the generative process for a categorical mixture model.

\begin{equation}
\begin{aligned}
Z_i&\sim \text{Categorical}(\pi)\\
X_i&\sim \text{Categorical}(\theta_{Z_i})
\end{aligned}
\label{eqn:gen-cmm}
\end{equation}
For this model, where we observe $X$ but not $Z$, we want to learn the parameters of the $K$ categorical components $\Theta = \lbrace\pi,\theta_1,...,\theta_K\rbrace$, where each $\theta_k\in\R^M$ is the parameter for the categorical distribution associated with the $k$-th mixture component (this means that each $X_i$ can take on one of $M$ possible values). We can use the EM algorithm to accomplish this.

A note on notation and a hint: it is helpful to use indicator variables when working with categorical distributions. The indicator function $\indic{x=j}$ has value 1 when $x=j$ and 0 otherwise. With this notation, we can express the probability that a random variable drawn from a categorical distribution (e.g., $Y\sim\text{Categorical}(\phi)$, where $\phi\in\R^N$) takes on a particular value as
\begin{equation*}
\begin{aligned}
P(Y) = \prod\limits_{i=1}^{N} \phi_i^{\indic{Y=i}}.
\end{aligned}
\end{equation*}

\begin{enumerate} [(a)]
\item {\bf [6 points]} What is the joint distribution $P(X,Z;\Theta)$?
\item {\bf [6 points]} What is the posterior distribution of the latent variables, $P(Z|X;\Theta)$?
\item {\bf [8 points]} What is the expectation of the log-likelihood, $Q(\Theta'|\Theta) := \expect_{Z|X;\Theta} \log P(X,Z;\Theta')$?
\item {\bf [Extra Credit, 5 points]} What is the update step for $\Theta$? That is, what $\Theta$ maximizes $Q(\Theta'|\Theta)$? (Remember that $\Theta$ includes the categorical parameter $\pi$ for $Z$ and the categorical parameters $\theta_1,...,\theta_K$ for $X$, from the generative process described in (\ref{eqn:gen-cmm}). Make sure your solution enforces the constraint that parameters to categorical distributions must sum to 1\textemdash Lagrange multipliers are a great way to solve constrained optimization problems.)
\end{enumerate}


\section*{Problem 2: AdaBoost}
Consider the following dataset, plotted in Figure~\ref{fig:adaboost-ex}: 
%$$X_1 = (-1,0,+), X_2 = (0,1,-), X_3 = (1,0,+), X_4 = (0,-1,-).$$
\begin{equation*}
\begin{aligned}
X_1 = (-1,0,+), X_2 = (-0.5,0.5,+), X_3 = (0,1,-), X_4 = (0.5,1,-),\\ X_5 = (1,0,+), X_6 = (1,-1,+), X_7 = (0,-1,-), X_8 = (0,0,-).
\end{aligned}
\end{equation*}
In this problem, you'll run through $T=3$ iterations of AdaBoost with decision stumps (axis-aligned half-planes) as weak learners.
\begin{figure}[h]
	\centering
	\includegraphics[scale=0.5]{adaboost_ex.pdf}
	\caption{A small dataset, for binary classification with AdaBoost.}
	\label{fig:adaboost-ex}
\end{figure}
\begin{enumerate}[(a)]
\item {\bf [20 points]} For each iteration $t=1,2,3$, compute $\epsilon_t,\alpha_t,Z_t,D_t(i)\,\forall i$ (in Table~\ref{tab:adaboost-vals}) and draw the decision stump (on Figure~\ref{fig:adaboost-ex}). Recall that $Z_t$ is the normalization factor to ensure that the weights $D_t$ sum to one.
\item {\bf [5 points]} What is the training error of AdaBoost? Give a one-sentence reason for why AdaBoost outperforms a single decision stump.
\end{enumerate}

\begin{table}[h]
\caption{Values of AdaBoost parameters at each timestep.}
\label{tab:adaboost-vals}
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline
$t$&$\;\epsilon_t\;$&$\;\alpha_t\;$&$\;Z_t\;$&$D_t(1)$&$D_t(2)$&$D_t(3)$&$D_t(4)$&$D_t(5)$&$D_t(6)$&$D_t(7)$&$D_t(8)$\\
\hline
1&&&&&&&&&&&\\
2&&&&&&&&&&&\\
3&&&&&&&&&&&\\
\hline
\end{tabular}
\end{center}
\end{table}

\section*{Problem 3: Perceptron}
Consider running the Perceptron algorithm on some sequence of examples $S$ (an example is a data point and its label). Let $S'$ be the same set of examples as $S$, but presented in a different order.
\begin{enumerate}[(a)]
\item {\bf [4 points]} Does the Perceptron algorithm necessarily make the same number of mistakes on $S$ as it does on $S'$?
\item {\bf [8 points]} If so, why?  If not, show such an $S$ and $S'$ where the Perceptron  algorithm makes a different number of mistakes on $S'$ than it does on $S$.
\item {\bf [8 points]} We know that in $\R^d$ we can shatter $d+1$ points  with linear separators, but not $d+2$ points (because the VC-dimension  of linear separators is $d+1$).  But what if we require that the points be separated by margin $\gamma$?  Show that you can have at most $(R/\gamma)^2$ points inside the ball of radius $R$ that can be ``shattered at margin $\gamma$," meaning that every labeling is achievable by a separator of margin $\gamma$.\\
{\bf Hint:} Suppose for contradiction you had $M=(R/\gamma)^2+1$ such points.  What would happen if you gave them in some order to the Perceptron algorithm, with labels exactly opposite of what Perceptron predicts?
\end{enumerate}

\section*{Problem 4: Kernels}
\subsection*{4.1: A proposed kernel}
Consider the following kernel function:
\begin{equation*}
K(x,x') = 
\begin{cases}
1, \; \text{if } x=x'\\
0, \; \text{otherwise}
\end{cases}
\end{equation*}
\begin{enumerate}[(a)]
\item {\bf [8 points]} Prove this is a legal kernel. That is, describe an implicit mapping $\Phi: X\rightarrow \R^m$ such that $K(x,x') = \Phi(x)\cdot\Phi(x')$. (You may assume the instance space $X$ is finite.)
\item {\bf [6 points]} In this kernel space, any labeling of points in $X$ will be linearly separable. Justify this claim.
\item {\bf [6 points]} Since all labelings are linearly separable, this kernel seems perfect for learning any target function. Why is this actually a bad idea?
\end{enumerate}
\subsection*{4.2: Composition of kernels}
It is possible to use previously-defined kernels to construct new ones. For this problem, you will prove why the following proposed functions are or are not valid kernels The notation $<x,z>$ indicates the dot product $x^Tz)$.
\begin{enumerate}[(a)]
\item {\bf [3 points]} $K(x,z) = 5<x,z>$
\item {\bf [4 points]} $K(x,z) = <x,z>^3 + (<x,z>+1)^2$
\item {\bf [8 points]} $K(x,z) = <x,z>^2 + \exp(-||x||^2)\exp(-||z||^2)$
\end{enumerate}

\section*{Problem 5: Extra Credit}
\subsection*{Support Vector Machines}
Suppose we have a set $S = \lbrace(x_1,y_1),...,(x_m,y_m)\rbrace$ of labeled examples in $\R^n$, and assume $||x_i||=1,\forall i$. It is NP-hard to find a linear separator that minimizes the number of points misclassified, so learning algorithms optimize other related quantities that are easier to solve. SVM solves the optimization problem
\begin{equation}
\begin{aligned}
&\underset{w}{\min}&&||w||^2_2 + C\sum\limits_{i=1}^m\xi_i\\
&\text{subject to } &&y_i(w^Tx_i) \geq 1-\xi_i &\forall i\\
&&&\xi_i\geq 0 &\forall i
\end{aligned}
\end{equation}
\begin{enumerate}[(a)]
\item {\bf [5 points]} Suppose that $S$ has the property that the total distance one would need to move the points in order to make them separable by margin $\gamma$ is $d_\gamma$. (By ``total distance" we mean $\sum_i d_i$ where $d_i$ is the distance that point $x_i$ is moved. By ``separable by margin $\gamma$" we mean that, for some hyperplane through the origin, all points are on the correct side and at distance at least $\gamma$ from it.) Show that, for an appropriate value of $C$, the number of misclassifications made on $S$ by the separator produced by SVM is at most $\frac12 + d_\gamma/\gamma$.\\
{\bf Hint:} We can find the value of $d_\gamma$ with the following optimization problem:
\begin{equation}
\label{eqn:d-gamma}
\begin{aligned}
&\underset{w,\tilde{\gamma}_i,\tilde{\gamma}}{\min}&& \sum\limits_{i=1}^m\tilde{\gamma}_i\\
&\text{subject to } &&||w||=1\\
&&&y_i(w^Tx_i) \geq \tilde{\gamma}-\tilde{\gamma}_i &\forall i\\
\end{aligned}
\end{equation}
where $\gamma_i$ indicates the distance between point $x_i$ and the appropriate margin.
\end{enumerate}

\subsection*{Boosting}
Suppose that, instead of computing $\alpha_t = \frac12\ln\left(\frac{1-\epsilon_t}{\epsilon_t}\right)$, we instead fixed $\alpha>0$ ahead of time and set $\alpha_t = \alpha$ for each iteration $t$ in this Boost($\alpha$) algorithm. Assume that on every round $t$ of boosting, we know that $\epsilon_t$ will be at most $1/2-\gamma$, for some $\gamma>0$. We choose to set $\alpha = \frac12\ln\left(\frac{1+2\gamma}{1-2\gamma}\right)$.
\begin{enumerate}[(a)]
\item {\bf [5 points]} Show how to modify the training error analysis of AdaBoost to derive an upper bound on the training error of the final hypothesis $H$ produced by Boost($\alpha$) after $T$ rounds. Your bound should be in terms of $\gamma$ and $T$ only (it should not depend on $\alpha$,$\epsilon_t$, etc.).\\
{\bf Hint:} Begin by proving, as we did in class, the bound on the error $\text{err}_S(H_\text{final})$ in terms of the normalization factors $Z_t$.
\item {\bf [5 points]} Use the previous result to show that the final hypothesis $H$ will be consistent with $m$ training examples (i.e., have zero training error) after $T$ rounds if $T>\frac{\ln m}{2\gamma^2}$.
\item {\bf [5 points]} Assume that the weak learning algorithm generates hypotheses $h_t$ which belong to a finite class $\mathcal{H}$. Use error bounds from the class to show that if we choose $T$ as in part (b), then with probability $1-\delta$, the generalization error of $H$ is at most $\frac{T\ln|\mathcal{H}|+\ln(1/\delta)}{m}$.
\end{enumerate}

\end{document}
