\documentclass{article}

\usepackage[margin=1in]{geometry}
\usepackage{amsmath, amsfonts}
\usepackage{enumerate}
\usepackage{graphicx}
\usepackage{titling}
\usepackage{url}
\usepackage{xcolor}
\usepackage[colorlinks=true,urlcolor=blue]{hyperref}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Commands for customizing the assignment %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newcommand \duedate {5 p.m. Monday, February 16, 2015}

\title{
  10-601 Machine Learning: Homework 4\\
  \vspace{0.2cm}
  \large{
    Due \duedate{}
  }
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Useful commands for typesetting the questions %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newcommand \expect {\mathbb{E}}
\newcommand \mle [1]{{\hat #1}^{\rm MLE}}
\newcommand \map [1]{{\hat #1}^{\rm MAP}}
\newcommand \argmax {\operatorname*{argmax}}
\newcommand \argmin {\operatorname*{argmin}}
\newcommand \code [1]{{\tt #1}}
\newcommand \datacount [1]{\#\{#1\}}
\newcommand \ind [1]{\mathbb{I}\{#1\}}
\newcommand \bs [1]{\boldsymbol{#1}}

%%%%%%%%%%%%%%%%%%%%%%%%%%
% Document configuration %
%%%%%%%%%%%%%%%%%%%%%%%%%%

% Don't display a date in the title and remove the white space
\predate{}
\postdate{}
\date{}

% Don't display an author and remove the white space
\preauthor{}
\postauthor{}
\author{}

%%%%%%%%%%%%%%%%%%
% Begin Document %
%%%%%%%%%%%%%%%%%%

\begin{document}

\maketitle

\section*{Instructions}
\begin{itemize}
\item {\bf Late homework policy:} Homework is worth full credit if
  submitted before the due date, half credit during the next 48 hours,
  and zero credit after that.  You {\em must} turn in at least $n-1$
  of the $n$ homeworks to pass the class, even if for zero credit.
\item {\bf Collaboration policy:} Homeworks must be done individually,
  except where otherwise noted in the assignments. ``Individually''
  means each student must hand in their own answers, and each student
  must write and use their own code in the programming parts of the
  assignment. It is acceptable for students to collaborate in figuring
  out answers and to help each other solve the problems, though you
  must in the end write up your own solutions individually, and you
  must list the names of students you discussed this with.  We will be
  assuming that, as participants in a graduate course, you will be
  taking the responsibility to make sure you personally understand the
  solution to any work arising from such collaboration.

\item {\bf Online submission:} You must submit your solutions online
  on
  \href{https://autolab.cs.cmu.edu/courses/27/assessments/292}{autolab}. We
  recommend that you use \LaTeX{} to type your solutions to the
  written questions, but we will accept scanned solutions as well. On
  the Homework 4 autolab page, you can download the
  \href{https://autolab.cs.cmu.edu/courses/27/assessments/292/attachments/20}{template},
  which is a tar archive containing a blank placeholder pdf for the
  written questions and one Octave source file for each of the
  programming questions. Replace each pdf file with one that contains
  your solutions to the written questions and fill in each of the Octave source
  files with your code. When you are ready to submit, create a new tar archive of the
  top-level directory and submit your archived solutions online by
  clicking the ``Submit File'' button. You should submit a single tar
  archive identical to the template, except with each of the Octave
  source files filled in and with the blank pdf replaced by your
  solutions for the written questions. You are free to submit as many
  times as you like (which is useful since you can see the autograder
  feedback immediately).

  \textbf{\emph{DO NOT}} change the name of any of the files or
  folders in the submission template. In other words, your submitted
  files should have exactly the same names as those in the submission
  template. Do not modify the directory structure.
\end{itemize}

\section*{Problem 1: Gradient Descent}
In class we derived a gradient descent learning algorithm for simple linear regression where we assumed
\[y= w_0 + w_1 x_1 + \epsilon  \hspace{1cm}  \text{where } \epsilon \sim \mathcal{N}(0, \sigma^2) \]
In this question, you will do the same for the following model
$$ y = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1^2 + \epsilon \hspace{1cm} \text{where } \epsilon \sim \mathcal{N}(0, \sigma^2) $$
where your learning algorithm will estimate the parameters $w_0$, $w_1$, $w_2$, and $w_3$.
\begin{enumerate}[(a)]
\item {\bf [8 Points]} Write down an expression for $P(y \vert x_1, x_2)$.
\item {\bf [8 Points]} Assume you are given a set of training observations $(x_1^{(i)}, x_2^{(i)}, y^{(i)})$ for $i = 1,\ldots,n$.   Write down the conditional log likelihood of this training data.   Drop any constants that do not depend on the parameters $w_0, \ldots, w_3$.
\item {\bf [8 Points]} Based on your answer above, write down a function $f(w_0, w_1, w_2, w_3)$ that can be \emph{minimized} to find the desired parameter estimates.  
\item {\bf [8 Points]} Calculate the gradient of $f(\bs{w})$ with respect to the parameter vector $\bs{w} = [w_0, w_1, w_2, w_3]^T$. Hint: The gradient can be written as $$
 \nabla_{\bs{w}} f(\bs{w}) = \left[ \begin{array}{cccc}
\dfrac{\partial f(\bs{w})}{\partial w_0} & \dfrac{\partial f(\bs{w})}{\partial w_1}  & \dfrac{\partial f(\bs{w})}{\partial w_2} & \dfrac{\partial f(\bs{w})}{\partial w_3}
\end{array} \right]^T $$
\item {\bf [8 Points]} Write down a gradient descent update rule for $\bs{w}$ in terms of $ \nabla_{\bs{w}} f(\bs{w})$.
\end{enumerate}

\section*{Problem 2: Logistic Regression}

In this question, you will implement a logistic regression classifier and apply it to a two-class classification problem. As in the last assignment, your code will be autograded by a series of test scripts that run on Autolab. To get started, download the template for Homework 4 from the Autolab website, and extract the contents of the compressed directory. In the archive, you will find one~\code{.m} file for each of the functions that you are asked to implement, along with a file called \code{HW4Data.mat} that contains the data for this problem. You can load the data into Octave by executing \code{load("HW4Data.mat")} in the Octave interpreter. Make sure not to modify any of the function headers that are provided.

\begin{enumerate}[(a)]
\item {\bf [5 Points]} In logistic regression, our goal is to learn a set of parameters by maximizing the conditional log likelihood of the data. Assuming you are given a dataset with $n$ training examples and $p$ features, write down a formula for the conditional log likelihood of the training data in terms of the the class labels $y^{(i)}$, the features $x^{(i)}_1, \ldots, x^{(i)}_p$, and the parameters $w_0, w_1, \ldots, w_p$, where the superscript $(i)$ denotes the sample index. This will be your objective function for gradient ascent. 
\item {\bf [5 Points]} Compute the partial derivative of the objective function with respect to $w_0$ and with respect to an arbitrary $w_j$, i.e.~derive $\partial f / \partial w_0$ and $\partial f / \partial w_j$, where $f$ is the objective that you provided above.
\item {\bf [30 Points]} Implement a logistic regression classifier using gradient ascent by filling in the missing code for the following functions: 
\begin{itemize}
\item Calculate the value of the objective function: \code{obj = LR\_CalcObj(XTrain,yTrain,wHat)}
\item Calculate the gradient: \code{grad = LR\_CalcGrad(XTrain,yTrain,wHat)}
\item Update the parameter value: \code{wHat = LR\_UpdateParams(wHat,grad,eta)}
\item Check whether gradient ascent has converged: \code{hasConverged = LR\_CheckConvg(oldObj,newObj,tol)}
\item Complete the implementation of gradient ascent: \code{[wHat,objVals] = LR\_GradientAscent(XTrain,yTrain)}
\item Predict the labels for a set of test examples: \code{[yHat,numErrors] = LR\_PredictLabels(XTest,yTest,wHat)}
\end{itemize}
where the arguments and return values of each function are defined as follows:
\begin{itemize}
\item \code{XTrain} is an $n \times p$ dimensional matrix that contains one training instance per row
\item \code{yTrain} is an $n \times 1$ dimensional vector containing the class labels for each training instance
\item \code{wHat} is a $p+1 \times 1$ dimensional vector containing the regression parameter estimates $\hat{w}_0, \hat{w}_1, \ldots, \hat{w}_p$
\item \code{grad} is a $p+1 \times 1$ dimensional vector containing the value of the gradient of the objective function with respect to each parameter in \code{wHat}
\item \code{eta} is the gradient ascent step size that you should set to \code{eta = 0.01}
\item \code{obj}, \code{oldObj} and \code{newObj} are values of the objective function
\item \code{tol} is the convergence tolerance, which you should set to \code{tol = 0.001}
\item \code{objVals} is a vector containing the objective value at each iteration of gradient ascent
\item \code{XTest} is an $m \times p$ dimensional matrix that contains one test instance per row
\item \code{yTest} is an $m \times 1$ dimensional vector containing the true class labels for each test instance
\item \code{yHat} is an $m \times 1$ dimensional vector containing your predicted class labels for each test instance
\item \code{numErrors} is the number of misclassified examples, i.e.~the differences between \code{yHat} and \code{yTest}
\end{itemize}
To complete the \code{LR\_GradientAscent} function, you should use the helper functions \code{LR\_CalcObj}, \code{LR\_CalcGrad}, \code{LR\_UpdateParams}, and \code{LR\_CheckConvg}.
\item {\bf [0 Points]} Train your logistic regression classifier on the data provided in \code{XTrain} and \code{yTrain} with \code{LR\_GradientAscent}, and then use your estimated parameters \code{wHat} to calculate predicted labels for the data in \code{XTest} with \code{LR\_PredictLabels}. 
\item {\bf [5 Points]} Report the number of misclassified examples in the test set.
\item {\bf [5 Points]} Plot the value of the objective function on each iteration of gradient descent, with the iteration number on the horizontal axis and the objective value on the vertical axis. Make sure to include axis labels and a title for your plot. Report the number of iterations that are required for the algorithm to converge.
\item {\bf [10 Points]} Next, you will evaluate how the training and test error change as the training set size increases. For each value of $k$ in the set $\{10, 20, 30, \ldots, 480, 490, 500\}$, first choose a random subset of the training data of size $k$ using the following code:
\begin{align*}
&\code{subsetInds = randperm(n,k)} \\
&\code{XTrainSubset = XTrain(subsetInds,:)} \\
&\code{yTrainSubset = yTrain(subsetInds)}
\end{align*}
Then re-train your classifier using \code{XTrainSubset} and \code{yTrainSubset}, and use the estimated parameters to calculate the number of misclassified examples on both the training set \code{XTrainSubset} and \code{yTrainSubset} and on the original test set \code{XTest} and \code{yTest}. Finally, generate a plot with two lines: in blue, plot the value of the training error against $k$, and in red, pot the value of the test error against $k$, where the error should be on the vertical axis and training set size should be on the horizontal axis. Make sure to include a legend in your plot to label the two lines. Describe what happens to the training and test error as the training set size increases, and provide an explanation for why this behavior occurs.
\end{enumerate}
{\bf Extra Credit:}
\begin{enumerate}[(a)]
\setcounter{enumi}{7}
\item {\bf [5 Points]}  Based on the logistic regression formula you learned in class, derive the analytical expression for the decision boundary of the classifier in terms of $w_0, w_1, \ldots, w_p$ and $x_1, \ldots, x_p$. What can you say about the shape of the decision boundary? 
\item {\bf [5 Points]} In this part, you will plot the decision boundary produced by your classifier. First, create a two-dimensional scatter plot of your test data by choosing the two features that have highest absolute weight in your estimated parameters \code{wHat} (let's call them features $j$ and $k$), and plotting the $j$-th dimension stored in \code{XTest(:,j)} on the horizontal axis and the  $k$-th dimension stored in \code{XTest(:,k)} on the vertical axis. Color each point on the plot so that examples with true label $y = 1$ are shown in blue and label $y = 0$ are shown in red. Next, using the formula that you derived in part (h), plot the decision boundary of your classifier in black on the same figure, again considering only dimensions $j$ and $k$. 
\end{enumerate}

\end{document}
