Proceedings of the Eighteenth International Conference on Machine Learning, 2001, p. 577–584.

Constrained K-means Clustering with Background Knowledge

Kiri Wagstaff [email protected]

Claire Cardie [email protected]
Department of Computer Science, Cornell University, Ithaca, NY 14853 USA
Seth Rogers [email protected]
Stefan Schroedl [email protected]
DaimlerChrysler Research and Technology Center, 1510 Page Mill Road, Palo Alto, CA 94304 USA

ter placement (Wagstaff & Cardie, 2000). K-means

Abstract
is another popular clustering algorithm that has been
Clustering is traditionally viewed as an un- used in a variety of application domains, such as image
supervised method for data analysis. How- segmentation (Marroquin & Girosi, 1993) and infor-
ever, in some cases information about the mation retrieval (Bellot & El-Beze, 1999). Due to its
problem domain is available in addition to widespread use, we believe that developing a modified
the data instances themselves. In this paper, version that can make use of background knowledge
we demonstrate how the popular k-means can be of significant use to the clustering community.
clustering algorithm can be profitably modi- The major contributions of the current work are two-
fied to make use of this information. In ex- fold. First, we have developed a k-means variant that
periments with artificial constraints on six can incorporate background knowledge in the form
data sets, we observe improvements in clus- of instance-level constraints, thus demonstrating that
tering accuracy. We also apply this method this approach is not limited to a single clustering al-
to the real-world problem of automatically gorithm. In particular, we present our modifications
detecting road lanes from GPS data and ob- to the k-means algorithm and demonstrate its perfor-
serve dramatic increases in performance. mance on six data sets.
Second, while our previous work with COBWEB was
1. Introduction restricted to testing with random constraints, we
demonstrate the power of this method applied to a
Clustering algorithms are generally used in an un- significant real-world problem (see Section 6).
supervised fashion. They are presented with a set of
In the next section, we provide some background on
data instances that must be grouped according to some
the k-means algorithm. Section 3 examines in de-
notion of similarity. The algorithm has access only to
tail the constraints we propose using and presents our
the set of features describing each object; it is not given
modified k-means algorithm. Next, we describe our
any information (e.g., labels) as to where each of the
evaluation methods in Section 4. We present experi-
instances should be placed within the partition.
mental results in Sections 5 and 6. Finally, Section 7
However, in real application domains, it is often the compares our work to related research and Section 8
case that the experimenter possesses some background summarizes our contributions.
knowledge (about the domain or the data set) that
could be useful in clustering the data. Traditional clus-
2. K-means Clustering
tering algorithms have no way to take advantage of this
information even when it does exist. K-means clustering (MacQueen, 1967) is a method
commonly used to automatically partition a data set
We are therefore interested in ways to integrate back-
into k groups. It proceeds by selecting k initial cluster
ground information into clustering algorithms. We
centers and then iteratively refining them as follows:
have previously had success with a modified version of
COBWEB (Fisher, 1987) that uses background infor-
mation about pairs of instances to constrain their clus- 1. Each instance di is assigned to its closest cluster
 center.
Table 1. Constrained K-means Algorithm
2. Each cluster center Cj is updated to be the mean cop-kmeans(data set D, must-link constraints Con= ⊆
of its constituent instances. D × D, cannot-link constraints Con6= ⊆ D × D)
1. Let C1 . . . Ck be the initial cluster centers.
The algorithm converges when there is no further
change in assignment of instances to clusters. 2. For each point di in D, assign it to the closest cluster
Cj such that violate-constraints(di , Cj , Con= ,
In this work, we initialize the clusters using instances Con6= ) is false. If no such cluster exists, fail
chosen at random from the data set. The data sets we (return {}).
used are composed solely of either numeric features 3. For each cluster Ci , update its center by averaging all
or symbolic features. For numeric features, we use a of the points dj that have been assigned to it.
Euclidean distance metric; for symbolic features, we
4. Iterate between (2) and (3) until convergence.
compute the Hamming distance.
5. Return {C1 . . . Ck }.
The final issue is how to choose k. For data sets
where the optimal value of k is already known (i.e., all violate-constraints(data point d, cluster C, must-
link constraints Con= ⊆ D × D, cannot-link constraints
of the UCI data sets), we make use of it; for the real- Con6= ⊆ D × D)
world problem of finding lanes in GPS data, we use
1. For each (d, d= ) ∈ Con= : If d= 6∈ C, return true.
a wrapper search to locate the best value of k. More
details can be found in Section 6. 2. For each (d, d6= ) ∈ Con6= : If d6= ∈ C, return true.

3. Otherwise, return false.

3. Constrained K-means Clustering
We now proceed to a discussion of our modifications
to the k-means algorithm. In this work, we focus on
background knowledge that can be expressed as a set
of instance-level constraints on the clustering process.
After a discussion of the kind of constraints we are
rived from partially labeled data (cf. Section 5) or from
using, we describe the constrained k-means clustering
background knowledge about the domain or data set
algorithm.
(cf. Section 6).

3.1 The Constraints

3.2 The Constrained Algorithm
In the context of partitioning algorithms, instance-
Table 1 contains the modified k-means algorithm
level constraints are a useful way to express a priori
(COP-KMEANS) with our changes in bold. The algo-
knowledge about which instances should or should not
rithm takes in a data set (D), a set of must-link con-
be grouped together. Consequently, we consider two
straints (Con= ), and a set of cannot-link constraints
types of pairwise constraints:
(Con6= ). It returns a partition of the instances in D
that satisfies all specified constraints.
• Must-link constraints specify that two instances
have to be in the same cluster. The major modification is that, when updating clus-
ter assignments, we ensure that none of the speci-
• Cannot-link constraints specify that two in- fied constraints are violated. We attempt to assign
stances must not be placed in the same cluster. each point di to its closest cluster Cj . This will suc-
ceed unless a constraint would be violated. If there
The must-link constraints define a transitive binary re- is another point d= that must be assigned to the
lation over the instances. Consequently, when making same cluster as d, but that is already in some other
use of a set of constraints (of both kinds), we take a cluster, or there is another point d6= that cannot be
transitive closure over the constraints.1 The full set grouped with d but is already in C, then d cannot
of derived constraints is then presented to the clus- be placed in C. We continue down the sorted list
tering algorithm. In general, constraints may be de- of clusters until we find one that can legally host d.
1 Constraints are never broken; if a legal cluster cannot
Although only the must-link constraints are transitive,
the closure is performed over both kinds because, e.g, if di be found for d, the empty partition ({}) is returned.
must link to dj which cannot link to dk , then we also know An interactive demo of this algorithm can be found at
that di cannot link to dk . http://www.cs.cornell.edu/home/wkiri/cop-kmeans/.
4. Evaluation Method ing algorithm). If they had the same label, we gener-
ated a must-link constraint. Otherwise, we generated
The data sets used for the evaluation include a “cor- a cannot-link constraint. We conducted 100 trials on
rect answer” or label for each data instance. We use each data set (where a trial is one 10-fold cross valida-
the labels in a post-processing step for evaluating per- tion run) and averaged the results.
formance.
In our previous work with COP-COBWEB, a con-
To calculate agreement between our results and the strained partitioning variant of COBWEB, we made
correct labels, we make use of the Rand index (Rand, use of three UCI data sets (soybean, mushroom, and
1971). This allows for a measure of agreement between tic-tac-toe) and one “real-world” data set that involved
two partitions, P1 and P2 , of the same data set D. part of speech data (Wagstaff & Cardie, 2000). In this
Each partition is viewed as a collection of n ∗ (n − 1)/2 work, we replicated our COP-COBWEB experiments
pairwise decisions, where n is the size of D. For each for the purpose of comparison with COP-KMEANS.
pair of points di and dj in D, Pi either assigns them COBWEB is an incremental algorithm, while k-means
to the same cluster or to different clusters. Let a be is a batch algorithm. Despite their significant algo-
the number of decisions where di is in the same cluster rithmic differences, we found that both algorithms im-
as dj in P1 and in P2 . Let b be the number of deci- proved almost identically when supplied with the same
sions where the two instances are placed in different amount of background information.
clusters in both partitions. Total agreement can then
be calculated using

a+b
Rand(P1 , P2 ) = .
n ∗ (n − 1)/2

We used this measure to calculate accuracy for all of

our experiments. We were also interested in testing
our hypothesis that constraint information can boost
performance even on unconstrained instances. Conse-
quently, we present two sets of numbers: the overall
accuracy for the entire data set, and accuracy on a
held-out test set (a subset of the data set composed of
instances that are not directly or transitively affected
by the constraints). This is achieved via 10-fold cross-
validation; we generate constraints on nine folds and
evaluate performance on the tenth. This enables a true
Figure 1. COP-KMEANS results on soybean
measurement of improvements in learning, since any
improvements on the held-out test set indicate that
the algorithm was able to generalize the constraint in- The first data set of interest is soybean, which has
formation to the unconstrained instances as well. 47 instances and 35 attributes. Four classes are rep-
resented in the data. Without any constraints, the
k-means algorithm achieves an accuracy of 87% (see
5. Experimental Results Using Figure 1). Overall accuracy steadily increases with
Artificial Constraints the incorporation of constraints, reaching 99% after
In this section, we report on experiments using six 100 random constraints.
well-known data sets in conjunction with artificially- We applied the Rand index to the set of constraints
generated constraints. Each graph demonstrates the vs. the true partition. Because the Rand index views
change in accuracy as more constraints are made avail- a partition as a set of pairwise decisions, this allowed us
able to the algorithm. The true value of k is known to calculate how many of those decisions were ’known’
for these data sets, and we provided it as input to our by the set of constraints.2 For this data set, 100 ran-
algorithm. dom constraints achieve an average accuracy of 48%.
The constraints were generated as follows: for each We can therefore see that combining the power of clus-
constraint, we randomly picked two instances from the tering with background information achieves better
data set and checked their labels (which are available performance than either in isolation.
2
for evaluation purposes but not visible to the cluster- For clarity, these numbers do not appear in the figure.
Held-out accuracy also improves, achieving 98% with COBWEB starts at 56% accuracy with no constraints,
100 constraints. This represents a held-out improve- reaching a held-out accuracy of 72% with 100 con-
ment of 11% over the baseline (no constraints). Simi- straints.
larly, COP-COBWEB starts at 85% accuracy with no
constraints, reaching a held-out accuracy of 96% with
100 random constraints.3

Figure 3. COP-KMEANS results on tic-tac-toe

Figure 2. COP-KMEANS results on mushroom

Finally, we focus on the tic-tac-toe data set.5 There
are 100 instances in this data set, each described by
9 attributes. There are two classes in this data set.
We next turn to the mushroom data set, with 50 in- Without constraints, the k-means algorithm achieves
stances and 21 attributes.4 It contains two classes. an accuracy of 51% (Figure 3). After incorporating
In the absence of constraints, the k-means algorithm 500 random constraints, overall accuracy is 92%. This
achieves an accuracy of 69% (Figure 2). After incor- set of constraints achieves 80% accuracy in isolation.
porating 100 random constraints, overall accuracy im- Held-out accuracy reaches 56%, achieving a 5% in-
proves to 96%. In this case, 100 random constraints crease in accuracy.
achieve 73% accuracy before any clustering occurs.
COP-COBWEB behaves somewhat worse on this data
Held-out accuracy climbs to 82%, yielding an improve-
set, with held-out performance staying roughly at the
ment of 13% over the baseline. COP-COBWEB starts
49% mark. We believe that this data set is particu-
at 67% accuracy with no constraints, with held-out
larly challenging because the classification of a board
accuracy reaching 83% with 100 constraints.
as a win or a loss for the X player requires extracting
The third data set under consideration is the part-of- relational information between the attributes — infor-
speech data set (Cardie, 1993). A subset of the full mation not contained in our instance-level constraints.
data set, it contains 50 instances, each described by
In contrast to the COP-COBWEB experiments, which
28 attributes. There are three classes in this data set.
made use of data sets with symbolic (categorical)
Without constraints, the k-means algorithm achieves
attributes, we also experimented with using COP-
an accuracy of 58% (We have omitted the graph for
KMEANS on two UCI data sets with numeric (contin-
this data set, as COP-KMEANS and COP-COBWEB
uous) attributes. On the iris data set (150 instances,
have very similar performance, just as shown in the
four attributes, three classes), incorporating 400 ran-
previous two figures.). After incorporating 100 ran-
dom constraints yielded a 7% increase in held-out ac-
dom constraints, overall accuracy improves to 87%.
curacy. Overall accuracy climbed from 84% to 98%,
Here, 100 random constraints attain 56% accuracy.
and the set of constraints achieved 66% accuracy. Be-
Held-out accuracy climbs to 70%, yielding an im-
havior on the wine data set (178 instances, 13 at-
provement of 12% over the baseline. Likewise, COP-
tributes, three classes) was similar to that of the tic-
3
The COP-COBWEB results are not reproduced in tac-toe data set, with only marginal held-out improve-
graph form here, but can be found in full detail in Wagstaff ment (although overall accuracy, as usual, increased
and Cardie (2000).
4 5
This is a subset of the full mushroom data set, to match Because this data set is larger, we experimented with
the COP-COBWEB experiments. more constraints.
dramatically, from 71% to 94%). The constraint set centerline.6 For evaluation purposes, we asked the
achieved 68% in isolation. drivers to indicate which lane they occupied and any
lane changes. This allowed us to label each data point
What we can conclude from this section is that even
with its correct lane.
randomly-generated constraints can improve cluster-
ing accuracy. As one might expect, the improvement
obtained depends on the data set under consideration. 6.2 Background Knowledge as Constraints
If the constraints are generalizable to the full data set, For the problem of automatic lane detection, we fo-
improvements can be observed even on unconstrained cused on two domain-specific heuristics for generat-
instances. ing constraints: trace contiguity and maximum sepa-
ration. These represent knowledge about the domain
6. Experimental Results on GPS Lane that can be encoded as instance-level constraints.
Finding Trace contiguity means that, in the absence of lane
changes, all of the points generated from the same ve-
In all of the above experiments, the constraints we ex-
hicle in a single pass over a road segment should end
perimented with were randomly generated from the
up in the same lane.
true data labels. To demonstrate the utility of con-
strained clustering with real domain knowledge, we Maximum separation refers to a limit on how far apart
applied COP-KMEANS to the problem of lane finding two points can be (perpendicular to the centerline)
in GPS data. In this section, we report on the results while still being in the same lane. If two points are
of these experiments. More details can be found in separated by at least four meters, then we generate
Schroedl et al. (2001), which focuses specifically on a constraint that will prevent those two points from
the problem of map refinement and lane finding. being placed in the same cluster.
As we will show, the unconstrained k-means algorithm To better analyze performance in this domain, we
performs abysmally compared to COP-KMEANS, modified the cluster center representation. The usual
which has access to additional domain knowledge way to compute the center of a cluster is to average
about the problem. Section 6.2 describes how we all of its constituent points. There are two significant
transformed this domain knowledge into a useful set drawbacks of this representation. First, the center of
of instance-level constraints. a lane is a point halfway along its extent, which com-
monly means that points inside the lane but at the far
6.1 Lane Finding Explained ends of the road appear to be extremely far from the
cluster center. Second, applications that make use of
Digital road maps currently exist that enable several the clustering results need more than a single point to
applications, such as generating personalized driving define a lane.
directions. However, these maps contain only coarse
information about the location of a road. By refining Consequently, we instead represented each lane clus-
maps down to the lane level, we enable a host of more ter with a line segment parallel to the centerline. This
sophisticated applications such as alerting a driver who more accurately models what we conceptualize as “the
drifts from the current lane. center of the lane”, provides a better basis for measur-
ing the distance from a point to its lane cluster cen-
Our approach to this problem is based on the obser- ter, and provides useful output for other applications.
vation that drivers tend to drive within lane bound- Both the basic k-means algorithm and COP-KMEANS
aries. Over time, lanes should correspond to “densely make use of this lane representation (for this problem).
traveled” regions (in contrast to the lane boundaries,
which should be “sparsely traveled”). Consequently,
6.3 Experiment 1: Comparison with K-means
we hypothesized that it would be possible to collect
data about the location of cars as they drive along a Table 2 presents accuracy results7 for both algorithms
given road and then cluster that data to automatically over 20 road segments. The number of data points for
determine where the individual lanes are located. each road segment is also indicated. These data sets
6
We collected data approximately once per second from The centerline parallels the road but is not necessarily
several drivers using GPS receivers affixed to the top located in the middle of the road.
7
These results represent overall accuracy rather than
of the vehicle being driven. Each data point is repre-
held-out accuracy, since determining the right set of con-
sented by two features: its distance along the road straints is part of the problem (they are not artificially
segment and its perpendicular offset from the road generated from the true labels).
 and averaging 98.6% overall. The unconstrained ver-
Table 2. Lane Finding Performance (Rand Index)
sion performed much worse, averaging 58.0% accuracy.
Segment K-means COP- Constraints
The clusters the latter algorithm produces often span
(size) KMEANS alone
multiple lanes and never cover the entire road segment
1 (699) 49.8 100 36.8
lengthwise. Lane clusters have a very specific shape:
2 (116) 47.2 100 31.5
they are greatly elongated and usually oriented hor-
3 (521) 56.5 100 44.2
izontally (with respect to the road centerline). Yet
4 (526) 49.4 100 47.1
even when the cluster center is a line rather than a
5 (426) 50.2 100 29.6
point, k-means seeks compact, usually spherical clus-
6 (503) 75.0 100 56.3
ters. Consequently, it does a very poor job of locating
7 (623) 73.5 100 57.8
the true lanes in the data.
8 (149) 74.7 100 53.6
9 (496) 58.6 100 46.8 For example, Figure 4 shows the output of the regular
10 (634) 50.2 100 63.4 k-means algorithm for data set 6.9 The horizontal axis
11 (1160) 56.5 100 72.3 is the distance along the road (in meters) and the ver-
12 (427) 48.8 96.6 59.2 tical axis is the centerline offset. There are four true
13 (587) 69.0 100 51.5 lanes. The points for each of the four clusters found by
14 (678) 65.9 100 59.9 k-means are represented by different symbols. Clearly,
15 (400) 58.8 100 39.7 these lanes do not correspond to the true lanes.
16 (115) 64.0 76.6 52.4
17 (383) 60.8 98.9 51.4
18 (786) 50.2 100 73.7
19 (880) 50.4 100 42.1
20 (570) 50.1 100 38.3
Average 58.0 98.6 50.4

are much larger than the UCI data sets, providing a

chance to test the algorithms’ scaling abilities.
In these experiments, the algorithms were required to
select the best value for the number of clusters, k. To
this end, we used a second measure that trades off
cluster cohesiveness against simplicity (i.e., number of
clusters).8 Note that this measure differs from the ob- Figure 4. K-means output for data set 6, k=4
jective function used by k-means and COP-KMEANS
while clustering. In the language of Jain and Dubes The final column in Table 2 is a measure of how much
(1988), the former is a relative criterion, while the lat- is known after generating the constraints and before
ter is an internal criterion. doing any clustering. It shows that an average accu-
In the lane finding domain, the problem of selecting k racy of 50.4% can be achieved using the background
is particularly challenging due to the large amount of information alone. What this demonstrates is that
noise in the GPS data. Each algorithm performed 30 neither general similarity information (k-means clus-
randomly-initialized trials with each value of k (from 1 tering) nor domain-specific information (constraints)
to 5). COP-KMEANS selected the correct value for k alone perform very well, but that combining the two
for all but one road segment, but k-means never chose sources of information effectively (COP-KMEANS)
the correct value for k (even though it was using the can produce excellent results.
same method for selecting k). An analysis of the errors made by COP-KMEANS on
As shown in Table 2, COP-KMEANS consistently the lane-finding data sets showed that each mistake
outperformed the unconstrained k-means algorithm, arose for a different reason. For data set 12, the algo-
attaining 100% accuracy for all but three data sets rithm incorrectly included part of a trace from lane 4
8 in lane 3. This appears to have been caused by noise
More precisely, it calculates the average squared
distance from each point to its assigned cluster cen- in the GPS points in question: they are significantly
ter and penalizes for the complexity of the solution: 9
The correct value of k was specified. Without it, the
Σi dist(di ,di .cluster)2
n
∗k2 . The goal is to minimize this value. algorithm selects k = 1.
closer to lane 3 than lane 4. On data set 16, COP- glomerative clustering algorithms and contiguity con-
KMEANS chose the wrong value for k (it decided on straints (similar to the above trace contiguity con-
three lanes rather than four). This road segment con- straint). In particular, no accommodation is provided
tains very few traces, which possibly contributed to for constraints that dictate the separation of two items.
the difficulty. Since COP-KMEANS made so few er- In addition, the contiguity relation is assumed to cover
rors on this data set, it is not possible to provide a all data items. This contrasts with our approach,
more general characterization of their causes. which can easily handle partial constraint relations
that only cover a subset of the instances.
It might be argued that k-means is simply a poor
choice of algorithm for this problem. However, In the machine learning literature, Thompson and
the marked improvements we observed with COP- Langley (1991) performed experiments with provid-
KMEANS suggest another advantage of this method: ing an initial “priming” concept hierarchy to several
algorithm choice may be of less importance when you incremental unsupervised clustering systems. The al-
have access to constraints based on domain knowledge. gorithms were then free to modify the hierarchy as
For this task, even a poorly-performing algorithm can appropriate. In contrast to these soft constraints, our
boost its performance to extremely high levels. In approach focuses on hard, instance-level constraints.
essence, it appears that domain knowledge can make
Additionally, Talavera and Béjar incorporated domain
performance less sensitive to which algorithm is cho-
knowledge into an agglomerative algorithm, ISAAC
sen.
(Talavera & Bejar, 1999). It is difficult to classify
ISAAC’s constraints as uniformly hard or soft. The
6.4 Experiment 2: Comparison with agglom final output is a partition (from some level of the hi-
Rogers et al. (1999) previously experimented with a erarchy), but the algorithm decides at which level of
clustering approach that viewed lane finding as a one- the hierarchy each constraint will be satisfied. Conse-
dimensional problem. Their algorithm (agglom) only quently, a given constraint may or may not be satisfied
made use of the centerline offset of each point. They by the output.
used a hierarchical agglomerative clustering algorithm It is possible for the k-means algorithm to evolve
that terminated when the two closest clusters were empty clusters in the course of its iterations. This
more than a given distance apart (which represented is undesirable, since it can produce a result with fewer
the maximum width of a lane). than k clusters. Bradley et al. (2000) developed a
This approach is quite effective when there are no lane method to ensure that this would never happen by
merges or splits across a segment, i.e., each lane con- imposing a minimum size on each cluster. Effec-
tinues horizontally from left to right with no inter- tively, these act as cluster-level constraints. Like our
ruptions. For the data sets listed in Table 2, their instance-level constraints, they can be used to incor-
algorithm obtains an average accuracy of 99.4%.10 porate domain knowledge about the problem. For ex-
ample, we know that road lanes must be separated
However, all of these data sets were taken from a free- by some minimum distance. However, we have not
way, where the number of lanes is constant over the yet incorporated this type of constraint as an input
entirety of each road segment. In cases where there are to the clustering algorithm; rather, we simply discard
lane merges or splits, the one-dimensional approach is solutions that contain lanes that are deemed too close
inadequate because it cannot represent the extent of a together. We are interested in exploring these cluster-
lane along the road segment. We are currently in the level constraints and integrating them more closely
process of obtaining data for a larger variety of roads, with the clustering algorithm itself.
including segments with lane merges and splits, which
we expect will illustrate this difference more clearly.
8. Conclusions and Future Directions
7. Related Work We have developed a general method for incorporat-
ing background knowledge in the form of instance-
A lot of work on certain varieties of constrained clus- level constraints into the k-means clustering algorithm.
tering has been done in the statistical literature (Gor- In experiments with random constraints on six data
don, 1973; Ferligoj & Batagelj, 1983; Lefkovitch, sets, we have shown significant improvements in accu-
1980). In general, this work focuses exclusively on ag- racy. Interestingly, the results obtained with COP-
10
A maximum merging distance of 2.5 meters was spec- KMEANS are very similar to those obtained with
ified. COP-COBWEB. In addition, we have demonstrated
how background information can be utilized in a real Ferligoj, A., & Batagelj, V. (1983). Some types of clus-
domain, GPS lane finding, and reported on impressive tering with relational constraints. Psychometrika,
gains in accuracy. 48, 541–552.
We see several avenues for improvements and future Fisher, D. (1987). Knowledge acquisition via incre-
work. The use of constraints while clustering means mental conceptual clustering. Machine Learning, 2,
that, unlike the regular k-means algorithm, the assign- 139–172.
ment of instances to clusters can be order-sensitive. If
a poor decision is made early on, the algorithm may Gordon, A. D. (1973). Classification in the presence
later encounter an instance di that has no possible of constraints. Biometrics, 29, 821–827.
valid cluster (e.g., it has a cannot-link constraint to at Jain, A. K., & Dubes, R. C. (1988). Algorithms for
least one item in each of the k clusters). This occa- clustering data. Prentice Hall.
sionally occurred in our experiments (for some of the
random data orderings). Ideally, the algorithm would Lefkovitch, L. P. (1980). Conditional clustering. Bio-
be able to backtrack, rearranging some of the instances metrics, 36, 43–58.
so that di could then be validly assigned to a cluster. MacQueen, J. B. (1967). Some methods for classifica-
Additionally, we are interested in extending the con- tion and analysis of multivariate observations. Pro-
strained clustering approach to include hierarchical al- ceedings of the Fifth Symposium on Math, Statistics,
gorithms. COP-KMEANS and COP-COBWEB both and Probability (pp. 281–297). Berkeley, CA: Uni-
generate a partition of the data and therefore are well- versity of California Press.
situated to take advantage of hard, instance-level con-
Marroquin, J., & Girosi, F. (1993). Some extensions of
straints. A different constraint formulation will be re-
the k-means algorithm for image segmentation and
quired for hierarchical algorithms.
pattern recognitionAI Memo 1390). Massachusetts
Finally, we would like to explore an alternative to the Institute of Technology, Cambridge, MA.
hard constraints presented here. Often domain knowl-
Rand, W. M. (1971). Objective criteria for the evalua-
edge is heuristic rather than exact, and it is possi-
tion of clustering methods. Journal of the American
ble that it would be better expressed by a “soft” con-
Statistical Association, 66, 846–850.
straint.
Rogers, S., Langley, P., & Wilson, C. (1999). Min-
Acknowledgements ing GPS data to augment road models. Proceedings
of the Fifth International Conference on Knowledge
We thank Pat Langley for his assistance and sugges- Discovery and Data Mining (pp. 104–113). San
tions and for access to the GPS data sets. We would Diego, CA: ACM Press.
especially like to thank Westley Weimer for advice and
suggestions on the work as it progressed. We also Schroedl, S., Wagstaff, K., Rogers, S., Langley, P., &
thank Marie desJardins for her insightful comments. Wilson, C. (2001). Mining GPS traces for map re-
finement. (in preparation).

References Talavera, L., & Bejar, J. (1999). Integrating declara-

tive knowledge in hierarchical clustering tasks. Pro-
Bellot, P., & El-Beze, M. (1999). A clustering ceedings of the International Symposium on Intelli-
method for information retrieval (Technical Report gent Data Analysis (pp. 211–222). Amsterdam, The
IR-0199). Laboratoire d’Informatique d’Avignon, Netherlands: Springer-Verlag.
France.
Thompson, K., & Langley, P. (1991). Concept forma-
Bradley, P. S., Bennett, K. P., & Demiriz, A. (2000). tion in structured domains. In D. H. Fisher, M. J.
Constrained k-means clustering (Technical Report Pazzani and P. Langley (Eds.), Concept formation:
MSR-TR-2000-65). Microsoft Research, Redmond, Knowledge and experience in unsupervised learning,
WA. 127–161. San Mateo, CA: Morgan Kaufmann.
Wagstaff, K., & Cardie, C. (2000). Clustering with
Cardie, C. (1993). A case-based approach to knowl-
instance-level constraints. Proceedings of the Seven-
edge acquisition for domain-specific sentence analy-
teenth International Conference on Machine Learn-
sis. Proceedings of the Eleventh National Conference
ing (pp. 1103–1110). Palo Alto, CA: Morgan Kauf-
on Artificial Intelligence (pp. 798–803). Washing-
mann.
ton, DC: AAAI Press / MIT Press.