OPenCV SeeingWith

in lmages Faces Finding

Figure1 what this mouthfulmeans. face of OPenCV's showsan example in action. detector - which cancome an image Given - the face from a fileor from livevideo ptqrcmrrrs.lt detector for C/C++, libnary location eachimage examines "Face" "NotFace." or it as classifies and thebasics downloading covercd for a fixedscale assumes and Classification reading OpenCV, installing and faces Since the face,say50 x 50 pixels. video, in an imagemightbe smaller capturing files, image writing or larger the over runs classifier the this, than data lPllmage withthe working and times, to searchfor image several structufe. Thismay of scales. a range across faces amount of seem like an enormous to algorithmic processing, but thanks rFhis month, l'll show You how to classifi(explained the sidebar), in tricks l'll faces. detect to I use OpenCV I exolainhow the face detection cation is very fast, even when it's scales. at several applied works,and giveyou tips for algorithm usesdata storedin The classifier gettingthe mostout of it. an XML file to decidehow to classify each image location.The OPenCV of XML four flavors includes download and data for frontal face detection, OpenCV uses a tYPe of face one for orofilefaces.lt also includes XML files:one for full detector called a Haar Cascade three non-face "How Face body (pedestrian) one for detection, ier. The sidebar, classif bodY. for lower and one Works, or What's a Haar upperbody, Detection classifier the You'll need to tell explains Anyhow?" Classifier, Cascade where to find the data file You want it to use. The one l'll be rontal haarcascade-f is called using OPenCV In face-default.xml. at: 1.0, it's located version the defaultinstallaand you selected you'duse: tion location,
iOPil{CV-ROOTI @enCV" "C: Ptogram Filesr

and Background Preliminaries

16-bit with an older, (lf you're working '\' as use of Windows, version You'd 'l'-) of instead separator, the directory the XML It'sa good ideato locate file you want to use and make sure your path to it is correctbeforeyou code the rest of your face detectron pro9ram. You'll also need an image to is a good process. Theimagelena.jpg one to test with. lt's locatedin the lf you directory. OpenCVsamples/c copy it to your program'sworking directory,you'll easilYbe able to your Program's outPutwith compare 2. the outputfromthe codein Figure

Face lmolementing by SteP StEp Detection,

codeto the source 2 shows Figure load an image from a file, detect the imagewrth in it, and display faces detectedfaces outlined in 9reen. produced by the display 1 shows Frgure this programwhen it's run from the using: line, command
DeLectFaces lena.jPg

Initializing(and running)the Detector The variableCvHaarClassifier where loPn'icv-noor]is the Path (Line2) holdsthe For cascade* pcascade installation. to your OpenCV XP data from the XML file You located if you'reon Windows example, To load the XML data into earlier. you can use the cvloado pCascade, 1. Facedetection with OpenCV FIGURE cvloado as in Lines11-13. using default parameters.The input function, for samPles/c function in the general-purpose is lenaiPg, image is a directory. to up lt takes files. from data loading

/data/haarcascades / IoPE]rCv-RoOTl haarcascade-f rontal f ace-def au1t . xnl

oe.2ool 48 srnvo

FIGURE9. Source code to detect faces in one image. Usaga Detect <image Faces file>.

three input parameters. For this you'llonly needthe first example, parameter. This is the path to an XML file containing a valid Haar Cascade. Here, l've loaded the default frontal face detector includedwith OpenCV.lf you're coding in C, set the remaining parameters to 0. lf you're coding in C++, you can simply omit the unused parametersf rom your function call. Before detecting faces in images, you'llalsoneedto instantiate a CvMemstorage object (pStorase, declared at Line3). This is a memorybuffer that expands automatically, as needed. The face detector will put the listof detected faces into this buffer. Sincethe buffer is expandable, you won't needto worryaboutoverflowing it. All you'llhave to do iscreate it (Line 10), then release it when you're finished (Line 5a). You'lloften needto loaddata from files with OpenCV. Sinceit's easy to get a path wrong, it's a good ideato insert a quickcheck to make sure everythingloaded '16-24 and initialized properly. Lines do a simpleerror check,print a diagnostic message, and exit if initialization fails.

1 2 3 4 5 6 l 8 9 10 11 12 13
1 A

// declarations CvHaarClassifierCascade CvMemstorage * pStorage

.\/qad * nE:.aPa-fe6^.

* pCascade = 0; = 0;

/ / Lhe face detector // e>pandable memory buffer / / hsL of detected faces

int

i;

/ / initializaLions Ipllrnage * pTnpInE = (argc > 1) ? cvloadlrnage(argrvFl, CV_LOAD_IMAGE_COIOR) : 0; pstoraqre - clcreateMenistorage{0) ; *)cvload pCascade = (CvHaarClassifierCascade " /data/haarcascades ( (OPNCV_ROOT /haarcascade_frontal 0, 0, 0 ); valj-date that everything properly initialized !plnplmg l1 lpStorage ll lpCascade ) printf ( "Initial-ization failed: ?s \n,,, 'didn't (lplnplmg)? load image file" : "didn't (lpCascade)? load Haar cascade -- ,, "make sure path is correct" : "failed to allocate memory for data storage,,); exir(-1);

face_defauLt

. xnl " ) ,

1 5 //
ao

if(

I] 18 19

20 2T 22 23
1 A

25 2 6 / / detect faces in image Sao = .\/H^Arf)ol-cef Ohiocf e 2 1 nFafeRpff (plnplmg, pCascade, pstorage, 28 29 1.1, // increase search scale by 1Og each pass 30 3, / / drop groups of fewer tharl chree detections 31 CV_IAAR_DO_CANM_PRUNING, / / skip regions unlikely to contain a face 32 cvSize(0,O) ); / / use )o{L default for smallest search scale 33 3 4 / / crea|e a window to display detected faces 3 5 cvNamedwindow ( "Haar Window,,, CV_WINDOW_AU|OSIZE) ; 36 3 1 / / draw a rectangular outline around each detection 3 8 for(i=Q;iq(pFaceRectseq? pFaceRectseq->total:0) ; i++ ) 39 { 40 CvRect * r = (CvRect*)cvcetseqElern(pFaceRectSeq, i) ; 4L CvPoint pt1 = { r->x, r->y }; 43 44
cl
// dicnl:rr f 6, Aat r '- 1 0 n s -^*. * -d-. ( "Haar Window., plnplmgf) ; cvShowlnr,age 48 cwvaltKey(0); 4 9 cvDestroywindow( "Haar Window") ; 50 5 1 / / clean up and release resources 5 2 cvReleaselmage (&plnpIng ) ; 5 3 i f (pCascade ) cvReleaseHaarClassif ierCascade(&pcascade ); 5 4 i f (pStorage ) cvRel-easeMemstorage (&pstorage ) ;

CvPoint pt2 = 1 r->x + r->wid[h, r->y + r->height ); cvRectangle(plnpImg, pt1, pt2, CV_RGB(0,255,A),3, 4, 0l ;

46

r j
l

) )
tf

Runningthe Detector Lines27-32 call cvHaarDerect objecrs ( ) to run the facedetector. This function takes up to seven parameters. The f irst three are the imagepointer, XML data, and memory buffer. Theremaining four parameters are set to their C++ tells the window to automatically defaults. Theselast four parameters resize itselfto fit the imageyou giveit are described below, in the section, to display. "Parameters andTuning." To passan imagefor display, call cvShowrmageO with the nameyou preViewing the Results viously assigned the window,and the A quick way to check if your image you want it to display. The program worksis to display the results c\,r'/airKeyO callat Line48 pauses the in an OpenCV window. Youcancreate application untilyouclose the window. a display window using the lf the windowfailsto closeby clicking :vNamedWindowo function,as in Line its close icon,clickinside the window's 35. The first parameter is a string, display area, thenpress a keyboard key. with a window name.The second, Also, make sure your programcalls --lr_wrNDow_AurosrzE, is a flag that (Line49) to close cvDestroywindowO

the window. Face detections arestored as a list of CvRectsrrucr pointers. Lines 38-44 access each detectionrectangle and add its outline to the prnprms variable, w h i c h h o l d st h e i n - m e m o r y image loaded fromthe file. Releasing Resources Lines52-54release the resources used by the input image,the XML data, and the storage buffer. lf you'll be detecting facesin multiple you don't needto release images, the SERVO O2.2oO7 49

Etrtr
lmage trick.After B.The Integral FIGURE . A cascaded to combinemany classifier the pixelat (x,y)contains integrating, i n t h e s h a d e d featuresefficiently. t h e s u mo f a l l p i x e lv a l u e s in Thesumof pixelvalues rectangle. rectangleD is (x4, y4) - (x2,Y2)- (x3, The features that Viola and Jones y 3 ) + ( x 1 ,y 1 ) . used are based on Haar wavelets.Haar square wavelets are single-wavelength OpenCV's face detector uses a waves (one high interval and one low method that Paul Viola and Michael interval). In two dimensions,a sguare called Jonespublishedin 2001' Usually wave is a pair of adjacent rectangles simply the Viola-Jones method, or lightand one dark. one this approach to even just Viola-Jones, The actual rectangle combinations detecting objects in images combines used for visualobject detection are not four key concepts: tru Haarwavelets.lnstead,they contain better suited to . Simple rectangularfeatures, called rctanglecombinations of that visual recognitiontasks.Because Haarfeatures. thesefeaturesare called Haar difference, ratherthan features, or Haarlike . An Integral lmage for rapid feature features, Haar wavelets. FigureA shows the detection. featuresthat OpenCVuses. The presence of a Haar feature is . TheAdaBoost method. machine-learning determined by subtractingthe average pixel value from the average dark-region i sa cra s c a d e FIGUR C E .T h ec l a s s i f i e pixelvalue.lf the differenceis light-region lmage filters. chainof single-feature above a threshold (set during learning), the that makeit through subregions "Face." featureis said to be Present. that as are classified entirecascade To determine the Presenc or as "Not Face." All othersare classified absence of hundreds of Haar features at everY lmage imagelocation and at sevsubregion Viola efficiently, eral scales and Jones used a technique called an Integral "inte' lmage. ln general, grating" means adding In this smallunitstogether. cas, the small units are pixel values. The integral value for each Pixel is the sum of all the pixelsabove it and to its left. Startingat the top left and traversing to the right and down, the \-7 o a o

A. Examplesof the Haar FIGURE features used in OPenCV.

entire imagecan be integratedwith a tew integer operations Per Pixel. As Figure81 shows,after integration, the valueat each pixel location, (x,y) contains the sum of all Pixel rgionthat valueswithin a rectangular has one corner at the top left of the image and the other at location (x,y)' To find the average pixel value in this you'd only need to divide the rectangle, area. valueat (x,y)by the rectangle's But what if You want to know the summedvaluesfor someother rectangle, one that doesn't have one corner at the FigureB9 shows upper left of the image? the solution to that problem. Suppose you want the summed values in D. You can think of that as beingthe sumof pixel values in the combined rectangle,A+B A+B +C+D, minusthe sums in rectangles and A+C,plus the sum of pixel valuesin A. In other wordt - (A+B)- (A+C)+ 4. D = A+B+C+D isthe Integral A+B+C+D Conveniently, lmage'svalue at location 4 A+B is the value at location 2, A+C is the value at location 3, and A is the valueat location you canfind lmage, 1.So,with an lntegral the sum of pixel valuesfor any rectangle in the original image with just three integeroperations: (x4,y4)- (x9,Y9)- (x3,Y3)+ (xl, Y1). To select the specific Haar features to use and to set threshold levels,Viola and Jones use a machine-learning method called AdaBoost. AdaBoost "weak" classifiersto combines many "!(eak" here "strong" classifier. crateone gets the right only means the classifier answer a little more often than random guessing would. That'snot very good. But if you had a whole lot of these weak "pushed" the and each one classifiers in the right b i t l i t t l e a final answer direction,you'd havea strong,combined lorce for arrivingat the correct solution. AdaBoost selects a set of weak to combinand assigns classifiers a weight to each. This weighted combination is the strong ier. classif Viola and Jones combined as a filter chain, weak classifiers shown in FigureC, that's esPecialD.Thefirsttwo FIGURE intheoriginal H a a rf e a t u r e s cascade. Viola-Jones

n*S
ffinot
\_7

ftr,rotr""e

r""" ,

flafl

trtr

oe.2ool 50 sEnvo

ly elticientfor classifying image regions. Eachfilter is a weak classifier consisting of one Haar feature.The thresholdfor each filter is set low enoughthat it passes all, or nearlyall,faceexamples in the training set.(The training set isa large database of faces, maybea thousand or so.)During use,if anyone of thesefilters failsto passan image region, that region XML data or the buffer until after you're donedetecting faces.

is immediatelyclassifiedas "Not Face." When a filter passesan image region, it goes to the next filter in the chain. lmage regionsthat passthrough all filters in the chain are classified as "Face." Viola and Jones dubbed this filtering chain a cascade. The order of filters in the cascade is determined by weights that AdaBoost

assigns. Themoreheavily weighted filters come first to eliminate non-face image regions as quicklyas possible. Figure D shows the first two featuresfrom the original Viola-Jonescascade superimposed on myface.Thefirstone keys off than the eye the cheekareabeinglighter region. Theseconduses the fact th6t th bridgeof the noseis lighter thanthe eyes. detections f rom the Haar classif ier. you're yourfacedetector, While tuning it'shelpful to do thisjustto seewhat's g o i n g o n i n s r d eO p e n C VV . iewing the raw detections your will improve intuition aboutthe effects of changing otherparameters, whichwill helpyou t u n et h e m . ScaleIncrease Rate The fourth input parameter to
cvHaar DecectObj ecries how s I i Specif q u i c k l y O p e n C V s h o u l d i n c r e a s et h e scale for face detections with each pass i t m a k e s o v e r a n i m a g e . S e t t i n gt h i s higher makes the detector run faster ( b y r u n n i n g f e w e r p a s s e s )b , ut if it's too high, you may jump too quickly between scales and miss faces. The d e f a u l t i n O p e n C V i s i . 1, i n o t h e r words, scale increases by a factor of 1. 1 ( 1 0 % ) e a c h p a s s . Canny Pruning Flag The sixth parameter to cvHaar ) is a flag variable. D e L e c L o b j e c r(s Thereare currentlyonly two options:0 or cv_nAAR_Do_cANNy_pRUNrNG. lf the C a n n y P r u n i n go p t i o n i s s e l e c t e dt,h e d e t e c t o rs k i p si m a g e r e g i o n st h a t a r e u n l i k e l yt o c o n t a i n a f a c e , r e d u c i n g c o m p u t a t i o n ao l v e r h e a da n d p o s s i b l y

Parameters andTuning
-.^ -^. ,^-- | I rteredre paramelers you 5everdt canadjust to tunethe facedetector for yourapplication. Th^-^

Minimum DetectionScale parameter Theseventh in the call to cvHaarDereeLobjecLsl) is the size of the smallest faceto search for. In C, you can selectthe defaultfor this by setting the scale to 0x0,asin Figure 2, L i n e 3 2 . ( l n C + + , s i m p l yo m i t t h i s parameter to use the default.) But what is the default? Youcanfind out byopening theXMLfileyou'll be using. Look for the <size> tag. In the default frontalfacedetector. it's:
<size> 24 24 </size>.

So,for thiscascade, the default minimumscale is 24 x 24. Depending on the resolution you're using, thisdefault size maybe a very small portion of your overall image. A faceimage thissmall maynot be meaningful or useful, anddetecting youcouldusefor it takes up CPU cycles other purposes. For these reasons and alsoto minimize the numberof your own code needs face detections to process- it's best to set the minimum detection scale onlyas small asyoutrulyneed. To set the minimum scale higher than the default value, set this parameter to the size you want. A good rule of thumb is to use some fraction of yourinputimage's widthor heightas the minimumscale- for example, 1/4 of the rmage width. lf you specify a minimumscaleother than the default,be sure its aspect ratio(the ratioof width to height) is FIGURE 3. OpenCV's internal the same asthe default's. In thiscase. detection rectangles.To see these, = 0. use min_neighbors a s p e crta t i o is1:1.

Minimum Neighbors Threshold One of the thingsthat happens "behind the scenes" whenyoucallthe facedetector is that eachpositive face generates regionactually many hits fromthe Haar detector. Figure 3 shows OpenCV's internal rectangle listfor the example i m a g e ,l e n a . j p gT . he face region itself generates the largest clusterof rectangles. Theselargely overlap. In addition, there's one small detectionto the (viewer's) left, and two larger detections slightlyabove and leftof the mainfacecluster. Usually, isolateddetections are falsedetections, so it makes sense to discardthese. lt also makessense t o s o m e h o wm e r g e t h e m u l t i p l e detections for eachfaceregion into a singledetection. OpenCV does both these before returning its list of detected faces.The mergestep first groups rectangles that contain a large amount of overlap, then finds the average rectangle for the group. lt then replaces all rectangles in the group with the average rectangle. Between isolated rectangles and largegroupings are smaller groupings that may be faces, or may be false detections. The minimumneighbors threshold sets the cutoff levelfor discarding or keeping groupsbased rectangle on how many raw detections are in the group.The C++ defaultfor this parameter is three, whichmeans to merge groups of three or more and discardgroupswith fewer rectangles. lf you f ind that yourfacedetector is missing a lot of faces, you might try Ioweringthis thresholdto two or one. lf you set it to 0, OpenCV will return the completelist of raw

SERVO O2.2oO7 51

. OpenCV on Sourceforge http://sourc etorge.nd/ groiecls / opencvlibrary . Official OpenCVusergroup hfip:. / / lech.groups.yahoo.com/ group/OpenCV . G. Bradski, and V. A. Kaehler, "Learning-Based Computer Pisarevsky, Vision with Intel's OPen Source Computer Vision LibrarY," lntel Technology Journal,Vol 9(1), MaY 19, 9005. www.intel.com/technology/ eOZ irl / 2OO5 / a*Ot / vol ume09issu p01 learning-vision/ -abstract.htm "ABrietIntroduction . R.E. to Schapire, Boosting," Joint Conference on MorganKauman, Artificial lntelligence, pp. 1401-1 406,1999. SanFrancisco, "Rapid . P. Object Viola andM.J. Jones, of Cascade usinga Boosted Detection 9001. CVPR, Features," Simple

in ag pplication , OpenCV's HaarTrainin T.h e e l i m i n a t i ns go m ef a l s ed e t e c t i o n s t h a t application U s i n g a p p s d i r e c t o r y . b y i d e n t i f i e d regions to skip are of this t h e s c o p e ( t h e i s b e y o n d C a n n y d e t e c t o r a n e d g e running are i n s t r u c tions t h e H o w e v e r , a r t i c l e i m a g e b e f o r e t h e d e t e c t o r ) o v e r edge i n O p e n C V ' sa p p s / h a a r t r a i n r n g / d o c s rrrnnino t h e f a c ed e t e c t O r . d rrectory A g a i n , t h e c h o i c eo f w h e t h e r o r f t r a d e o f f l a g i s a t h i s not to set between speed and detectingmore I a c e s S e t t i n gt h i s f l a g s p e e d s process N o w t h a t y o u ' v ef o u n d a f a c e ,Y o u i n g , b u t m a v c a u s ey o u t o m i s ss o m e m i g h t w a n t t o f o l l o w i t a r o u n d .N e x t , ou cando well with f a c e s .I n g e n e r a ly i t s e t , b u t i f y o u ' r e h a v r n gd i f f i c u l t y m o n t h , l ' l l s h o w y o u h o w t o u s e tf , OpenCV'sface tracking Camshi t h r sf l a g m a y f a c e s ,c l e a r i n g detectrng allow you to detect more reliably. method,to do lust that Be seerng m i n i m u m - n e i g h b o r s y o uI Setting the t o 0 s o y o u c a nv i e wt h e r a w threshold d e t e c t r o r ^w sr l l h e l p y o u b e t t e r g a u g e Pruning. t h e e f f e c to f u s i n gC a n n y Robin Hewitt is an independent soft' ware consultant working in the areas of The Haar Cascade computer vision and robotics. She has T h e r e a r e s e v e r a l f r o n t a l f a c e worked as a Computer Vision Algorithm i n O p e n C V .T h e Developer at Evolution Roboticsand is a detector cascades f o r y o u w i l l d e p e n do n y o u r member of 5O(3), a computer'vision b e s tc h o i c e s e t u p . l t ' s e a s y t o s w i t c h b e t w e e n research group at UC 9an Diego. She is W h y one of the original develoPers of t h e f i l en a m e . t h e m- l u s tc h a n g e SodaVision, an experimental facenot try each? recognition system at UC San Diego. I t ' s a l s o p o s s i b l et o c r e a t e Y o u r SodaVisionwas built with OPenCV. own, custom XN/L file using the

Up ... Coming

o2.2oo:' 52 srnvo