Unsupervised Face Clustering	
	
	Aaron Jaech	Andy Strat	Chenkai Xie

Abstract
This paper outlines a simple methodology that can replicate the user experience of tagging a photo on Facebook or iPhoto for video. The method can take a video from any source (e.g., youtube.com, camcorder, webcam, television), detect all the frames with faces, tag scenes, and label participants. We achieve this with a combination of face and scene detection, tracking and clustering. The system presents the user with a representative image for each distinct individual in the video. The user can quickly label the representative images and the algorithm automatically associates that label with the rest of the images in the cluster. Upon completion, the video has been indexed by person and by scene for easy searching, categorizing and summarization. 
1	Introduction
Large collections of digital videos are found in surveillance applications as well as social media. In both applications, the capability of indexing or summarizing the video is highly desirable yet the size of these video collections makes manual labeling unfeasible.  Unsupervised face clustering is the ability of a system to determine the number of distinct individuals in each video and the scenes or times in which they appear. This paper outlines a method of detecting and extracting faces from video in such a way that a simple clustering algorithm is able to associate images from the same individual together. The method is robust to changes in pose, illumination, expression and occlusion.
The video is processed in two main steps. In the first phase, the Viola-Jones algorithm is used to detect all the faces in the video. Simultaneously, we run a scene detection algorithm and a face-tracking algorithm. In the second phase, the cropped faces are normalized and clustered. The only manual input required from the user is the labeling of one image of each distinct individual in the video, which can be performed quickly. In other words, this process replicates the familiar user experience of tagging faces in a photo. 

2	Face Detection
The Viola-Jones algorithm provides real time face detection. The algorithm operates on local rectangular features (called “Haar-like features” for their resemblance to the 2D-Haar basis) that can be extracted from the image with constant time operations. These features capture changes in intensity of the image across different regions i.e. the area encompassing the eyes and the eyebrows tends to be darker than the forehead.

Viola and Jones used AdaBoost to learn a classifier based on these features. AdaBoost is an iterative learning method that builds a strong classifier from the combination of many weak classifiers. Initially, all of the training samples are weighted equally. In each iteration, AdaBoost finds the weak classifier that minimizes the weighted classification error. In the case of Viola-Jones, the weak classifier is restricted to using just one of the Haar-like features. The training samples which are classified correctly are down-weighted before the next iteration so that subsequently learned weak classifiers will focus on the misclassified samples. The final strong classifier is a linear combination of all the weak classifiers each weighted by the training error on its iteration.

Figure 1: One frame from a test video with five detected faces.

The strength of a face detector primarily depends on the quality of the training images. Gathering good positive and negative training images is no easy task. For our application we used an open source implementation of the Viola-Jones algorithm. This detector works well on frontal faces but has difficulty with off-angles poses. Off-angle poses make up a significant part of our test videos.

In order to better compare one face to another, detected faces are cropped, resized and scaled to unit-norm.

3	Scene Detection
Scene detection is used as a preprocessing step for the face tracking since rapid scene changes tend to confuse the tracker. Typically, adjacent frames in the video will be similar to each other. When this is not the case it is likely the result of a scene change. For each frame we calculate 256 bin histograms for the red, green and blue channels. Changes in the histograms are measured by taking the norm of the difference of histograms between subsequent frames and normalizing this by the norm of the channel histogram for the current frame. If this value is greater than a fixed threshold than we assume a change in scene has taken place. We increase the accuracy of the scene detector by examining only every third or fourth frame.

4	Tracking
The goal of implementing a tracking algorithm is to compensate for some of the weaknesses of the face detector. In our test videos, the face detector would often detect a face at slightly different locations and scales from frame to frame even though the face remained in essentially the same position. These variations make crops from adjacent frames less similar than they might otherwise be. The face detector we used struggled to detect people wearing glasses—only correctly detecting about one out of every five frames. The tracker bridges the gaps between these sporadic detections. The face detector works best on frontal faces. When an individual looks straight at the camera and then looks away we’d like to capture that whole sequence. The tracker can initialize on the frontal detection and then follow the person during off-angles poses.

Tracking connects faces through their displacement throughout time. We initialize tracking on each face detected unless the detected face is already within close proximity to an existing tracking stream. Problems occur with this method primarily on scene change. If the scene abruptly changes the tracking streams will lock on to whatever happens to be in that location in the new scene. For this reason, it is important that we restart all the tracking streams when the scene changes. The scene detection algorithm is discussed above.

The first tracking method we considered was the ever-popular Lucas Kanade [2] template tracker. This tracker uses an initial frame and compares each subsequent image with that same template, and returns the highest response to that template. This works well when the object in the image holds its shape.  Alternatively, you can update the template at every iteration such that small changes, such as turning one’s head, will be reflected in the new template. This method is vulnerable to drifting.

The choice of how and when to update the template is an ongoing research problem. On one extreme, the tracker can update the target object at every frame. This technique tends to caused trackers to drift off the target object. On the other extreme, the tracker is never updated such that the original object is carrier through indefinitely. This method is prone to failure when the object changes shape, size and orientation. We opted to attempt to find a happy medium. 

We run face detection on every frame, so we know when the track stream intersects with a detected face. This allows us to update the tracker at every frame. Drift is accounted for by noticing that any track stream which drifts off the face will not be updated with a newly detected face. After a few frames, a track that has drifted off the face will be canceled. Since these stray tracks are short lived, the images they produce can be clustered and then discarded by eliminating all clusters under a certain size.

Ultimately we opted not to use the Lucas Kanade template matching because of its aversion to rotation and deformation, as a head moving in three dimensions will have. Instead, we implemented a discriminative tracker [3]. This tracker draws a distinction between foreground (face) pixels and background pixels. The way we measure this distinction is based on color histograms. We first find a number of linear combinations of pixel color values with coefficients between -2 and 2. This leaves 49 linearly independent features (such as val = -2*R + G - B. Such a feature would return high for large green values and small other values). We then compute histograms based on these 49 features and choose the features which best separate the foreground from the background. Once we know the features to use for tracking (which are updated at every iteration) we test the next image in the sequence with the chosen features. We then find the centroid of the response, which represents the center of the group of pixels closest in feature value to the foreground pixels, and update the tracker with the new location. 

5	Clustering
The general problem of determining whether two face images with differing pose and expression come from the same individual is known to be challenging. However, there are some things about the nature of video that make clustering viable. Since video is recorded at approximately fifteen frames per second, the individual has little time in between frames to change their pose or facial expression. Thus, adjacent frames tend to produce highly similar images. As the high frame rate indicates, videos contain an abundance of data. A short five-minute video contains approximately 4,500 frames! We assume that after a certain number of faces from an individual have been observed each subsequent face will be similar to at least one face that has already been seen. The multi-modal distribution in Figure 2 supports this assumption. The first mode is composed of pair-wise comparisons of images from the same person. The second peak represents pairs of images from distinct individuals. 

Figure: 2 Histogram of the Euclidean distance between all pairs of faces in 4,000 frames.

We used a simple agglomerative clustering algorithm. This algorithm is non-parametric and makes no assumptions about the number of clusters. (Note that the true clusters are almost certainly not normally distributed.) The algorithm works like so: Every point starts in its own cluster. In each iteration, the algorithm merges the closest two clusters. We define the distance between two clusters to be the minimum Euclidean distance between any two points in each of the clusters. This process is repeated until the distance between the two nearest clusters is greater than some fixed threshold. The user can adjust the threshold after viewing the results to make the clustering more or less aggressive. 

Figure 3: Six face crops that were correctly associated during clustering

The clustering termination threshold must be set empirically. It is dependent on the resolution of the cropped images. There is a trade-off between the purity (an impure cluster has images from more than one person) of the clusters and the number of clusters. Ideally, we would have exactly one cluster per distinct individual. If the threshold is increased then the number of clusters will be reduced but the purity of the clusters may suffer.

Once the clustering is finished, the user can be presented with a representative face from each cluster for identification. This allows the video to be indexed by person and by scene for easy searching, categorization and summarization. 

6	Conclusion
This method works well in practice. In our test videos, which were about five minutes long each, we were able to set a threshold that maintains perfect purity of the clusters with about two clusters per person.  As shown in Figure 3, the clustering algorithm is able to correctly associate faces with different pose, lighting and expression. It even handles images where a significant portion of the face is occluded. We expect the performance of this method to improve on longer videos since the algorithm performed best on the individuals who appeared more frequently in the test videos. In one of our test videos the camera panned across a group detecting at least twenty distinct individuals. The algorithm was able to handle this case correctly although we expect that it would have some difficulty for large crowds.

References

[1] Viola, P. and Jones, M. (2004), “Robust Real-Time Face Detection”, International Journal of Computer Vision, 57(2), 137-154, 2004

[2] S. Baker and I. Matthews. Lucas-kanade 20 years on: A unifying framework. International Journal of
Computer Vision, 56(3):221–255, 2004.

[3] Robert T. Collins, Yanxi Liu, and Marius Leordeanu. On-Line Selection of Discriminative Tracking
Features. Pattern Analysis and Machine Intelligence (PAMI), 2005.