ee 0) The Geometry of Multiple Views Despite the wealth of inform ation contained in a photograph, the depth of a scene point along the corresponding projection ra is not directly accessible in a single image. With at least wo pic tures, depth can be measure: through triangulation. This is of course one of the reasons that most animals have at least two ¢ es and/or move their head when looking for friend or foe, as well as the motivation for equipy ing an autonomous robot with a stereo or motion analysis system. Before building such a syst m, we must understand how several views of the same scene con- strain its three-dimensional tructure as well as the corresponding camera configurations. This is the goal of this chapter. In articular, we elucidate the geometric and algebraic constraints that hold among two, three, or nore views of the same scene. In the familiar setting of binocular stereo vision, we show that |e first image of any point must le in the plane formed by its second image and the optical cente s of the two cameras. This epipolar constraint can be represented algebraically by a 3 x 3 ma rix, called the essential matrix when the intrinsic parameters of the ‘cameras are known and the, undamental matrix otherwise. Three pictures ofthe same line intro duce a different constraint— namely, that the intersection ofthe planes formed by their preimages be degenerate. Algebraicall this geometric relationship can be represented by a3 x 3 x 3 tr: ‘focal tensor. More images | troduce additional constraints, for example four projections of the ‘same point satisfy certain g tadrilinear relations whose coefficients are captured by the quadri- {focal tensor. Remarkably, tI ¢ equations satisfied by multiple pictures of the same scene feature ‘can be set up without any k owledge of the cameras or the scene they observe, and a number of ‘methods for estimating thei_ parameters directly from image data are presented in this chapter. Computer vision isn « the only scientific field concerned with the geometry of multiple views: The goal of photogra nmetry, already mentioned in chapter 3, is precisely to recover quan titative geometric informati m from multiple pictures. Applications of the epipolar and trifocal constraints to the classical shotogrammetry problem of transfer (ie. the prediction of the po- sition of a point in an ima e given its position in a number of reference pictures) are briefly 218216 ‘The Geor etry of Multiple Views Chap. 10 discussed in this chapter, along with some examples. Many nore applications in the domains of stereo and motion analysis are presented in latter chapters. 10.1 TWO VIEWS 10.1.1 Epipolar Geometry Consider the images p and p' of a point P observed by t vo cameras with optical centers O and 0’. These five points all belong to the epipolar plane «efined by the two intersecting rays OP and O'P (Figure 10.1). In particular, the point p’ lies on the line 1’ where this plane and the retina 11’ of the second camera intersect. The line is te epipolar line associated with the point p, and it passes through the point e' where the base: e joining the optical centers O and (O" intersects TT’. Likewise, the point p lies on the epipolar line ! associated with the point p’, and this line passes through the intersection ¢ ofthe baseline with the plane 1. ‘The points e and e’ are called the epipoles of the two cameras. The epipole eis the pro- jection of the optical center O of the first camera in the im ige observed by the second camera and vice versa. As noted before, if p and p’ are images of he same point, then p’ must lie on the epipolar line associated with p. This epipolar constrain plays a fundamental role in stereo vision and motion analysis. Let us assume, for example, that we know the intrinsic and extrinsic parameters of the two cameras of a stereo rig. As shown in chapter 11, the most d fficul part of stereo data analysis is, establishing correspondences between the two images (c., leciding which points in the second picture match the points in the frst one). The epipolar const: int greatly limits the search for these correspondences: Indeed, since we assime that the rg is cali rated, the coordinates of the point p ‘completely determine the ray joining O and p, and thus the ussociated epipolar plane QO’ p and epipolar line 1’, The search for matches can be restricted to his line instead of the whole image (igure 10.2), In two-frame motion asalysis, each camera 1 say be internally calibrated, but the rigid transformation separating the two camera coordinates ‘stems is unknown. In this case, the epipolar geometry obviously constrains the set of possible motions. The next sections explore several variants of this situation. Figure 10.1 Epipolar geometry: The point P, the c ical centers O and 0" of. the two cameras, and the two images p and p' of J all lie inthe same plane. Here, as in the other figures ofthis chapter, camer sare represented by their pinholes and a viral image plane located in fron ofthe pinhole. This is 0 simplify the drawings: The geometric and algebraic. rpuments presented in the rest of this chapter hold just és wel for physical imap: planes located behind the corresponding pinholes.27 Figure 10.2 Epipolar constraint: Given a calibrated st reo rg, the set of possi- ble matches fr the pont pis censrained tole on the a sociated epipolar line 10.1.2 The Calibrated Case Here we assume tha the intrinsic parameters of each camera known, so p= p Clearly, he epipolar constraint implies tat the thee veetrs Op, O77’, an | O' are coplanas. Equivalenty, ‘one of them must ie inthe plane spanned by the other two, or Op 100" x O77 — 0. ‘We can rewrite this coordinate-independent equation in te coordinate frame associated to the first camera as, plex (Rp). (10.1) where p = (u,v, 1)” and p’ = (w’, v’, 1)" denote the homoger 20us image coordinate vectors of and p’, tis the coordinate vector of the translation OO" separ ting the two coordinate systems, and 7 is the rotation matrix such that a free vector with coordi tates w" in the second coordinate system has coordinates Rw’ in the first ore. In this case, the tw » projection matrices are given in the coordinate system attached to the first camera by (Id 0) aid(R7 — RT), Equation (10.1) can finally be rewritten as prep’ (102) where € = {t.]R, and (a;-] denotes the skew-symmetric mati: such that (aJx = a x x is the ross-product of the vectors a andx. The matrix Eis called the essential matrix, and it was first introduced by Longuet-Higgins (1981). Its nine coefficients a e only defined up to scale, and they can be parameterized by the three degrees of freedom of te rotation matrix Rand the two degrees of freedom defining the direction of the translation vec t Note that £p' can be interpreted as the coordinate vect ¢ representing the epipolar line associated with the point p' in the first image: Indeed, an im ie line J can be defined by its equation au +bu-+c = 0, where (u, v) denote the coordinates «Fa point on the line, (a,b) isthe unit normal tothe line, and ~c i the (signed) distance between te origin and /. Alternatively, we ccan define the line equation in terms of the homogeneous coor: inate vector p = (1, ¥, 1)" of a Point on the line and the vector ! = (a, b,c)" by lp = 0, in whic hcase the constraint 246? is relaxed since the equation holds independently of any scale ch inge applied tol. In this context, Eq, (10.2) expresses the fact that the poin: p lies on the epipole “line associated with the vector218 The € sometry of Mutiple Views Chap. 19 Ep’. By symmetry, it is also clear that £ pis the coot inate vector representing the epipolad line associated with p in the second image. It is obvic us that essential matrices are sn since ¢is parallel to the coordinate vector e ofthe fist «sipole, so that Ee = R(t, Je Likewise, itis easy to show that e's inthe nullspace 0. As shown by Huang and Fauger (1989), essential matrices are in fact characterized by te fact that they are singular with equal nonzero singular values (sce Exercises) 10.1.3 Small Motions Let us now turn our attention to infinitesimal displaceme: ts, We consider a moving camera with \canslational velocity v and rotational velocity wand rew ite Eq, (10.2) for two frames by a small time interval 5. Let us denote by p = (d, 6,0 7 the velocity of the point p or moti field. Using the exponential representation of rotations (s. ¢ Exercises), i is possible to show | (to first order) t= ory, R=ld+ sw.) top. Substituting it Bq, (10.2) and neglecting all terms of orde two or greater in 8 yields: PT (Uraltwalp ~ (Px p) ¥ =0. (104): Equation (10.4) is simply the instantaneous form o ‘the Longuet-Higgins relation (10 which captures the epipolar geometry in the discrete case Note that in the case of pure trans tion, we have w = 0, thus (p x p)-¥ = 0. In other wo ds, the three vectors p = 2B, p, ¥ must be coplanar. If e denotes the infinitesimal epipol: ot focus of expansion (i. the poiag where the line passing through the optical center and pars lel to the velocity vector v pierces {mage plane), we obtain the well-kaown result thatthe n otion field points toward the focus ff expansion under pure translational motion (Figure 10.3) 10.1.4 The Uncalibrated Case ‘The Longuet-Higgins relation holds for internally calibi ted cameras, When the intrinsic rameters are unknown (uncalibrated cameras), we can wi ite p = Figure 10.3 Focus of expansion: Under pure tra slation, the motion field at every point in the image points toward the focus of xpansion.lt See. 10.1 Two Views 219 and K! are 3 x 3 calibration matrices and p and p’ are normnalized imag« coordinate vectors. The Longuet-Higgins relation holds for these vectors, and we obtain pF p'=0, 10.) where the matrix F = K-TEK’~, called the fundamental matrix, is no , in general, an essential matrix. It hes again rank two, and the eigenvector of F (resp. F7) corres ionding to its zero cigen- ‘value is as before the position e” (resp. e) of the epipole. Note that Fz’ (resp. FT p) represents the epipolar line corresponding to the point p’ (resp. p) in the frst (rest. second) image. ‘The rank two constraint means that the fundamental matrix only : Smits seven independent parameters. Several choices of parameterization are possible, but the my st natural one is in terms Pf the coordinate vectors e = (a, 8)” and e’ = (a’, B')" of the two epiy oles, and of the epipolar transformation that maps one set of epipolar lines onto the other one. \ fe examine in chapter 13, the properties of this transformation in the context of structure from my tion. For the time being, fet us just note (without proof) that it can be parameterized (up to s ale) by four numbers— a,b, c,d—and that the fundamental matrix can te written as > a ap — ba Fa=| - = ep+da (106) dp’ — ba! cf" aa! chp’ — dB’a + afa' + baa! 10.1.5 Weak Calibration ‘As mentioned eatlicr, the essential matrix is defined up to scale by ive independent parame- ters, It is therefore possible (atleast in principz) to calculate it by «iting Eq. (10.2) for five point correspondences. Likewise the fundamental marx is defined b; seven independent coef- Feients (the parameters a,b, c,d in Eq, (10.6) ar only defined up to ale) and can in principle be estimated from seven point correspondences. Methods for estimatir the essential and funda- mental matrices from a minimal numberof parameters indeed exist (s Notes), but they ae far too involved to be deseribed here. This section addresses the simpler soblem of estimating the tpipolar geometry from a redundant set of pointcorrespondences bet fen (wo images taken by ‘cameras with unknown intrnsie parameters—a process known as we < calibration ‘Note that Eq (10.5) is linear in the nine coefficients of the fund: mental matrix F: Fu Fin Fa) (u 2 (wv, D| Fa Fa Fa} fv') =0 (10.7) Fy Fu Pa) \1 ‘Since this equation is homogeneous in the coeficients of F, we can set Fs = 1 and use eight point correspondences p; +» pi (7 = 1,-. 8) 10 rewrite the core sponding instances of Eq (10.7) as an 8 x 8 system of nonhomogeneous near equations: fae) ivi you vi, or ao) (Fa v pu wavy 2 val vat ve th ve] | Fiz 1 tou uses uy vse oaey te a 0 || Fis 1 tua, wad, ty vail vary ve we OL Fr | dT usu use us vsity usu vs wy ve || Fan 1 Hl, wore 5 Veg vot V6 te U4) Fos 1 tue, yeh wy ry ray vy oh | Fat 1 tue, unt sandy UaUE Mey Ue) Fn 1 Using this system to estimate the fundamental matrix gives the eigh: point algorithm originally proposed by Longuet-Higgins (1981) in the case of calibrated camer s. I fails when the associ‘The: ieometry of Multiple Views Chap. 10 ated 8 8 matrix is singular. As shown in Faugeras (1‘ 93) and the exercises, this only happens when the eight points and two optical centers lie on a ‘ uadric surface, Fortunately, this is quite unlikely since a quadric surface is completely determin: d by nine points, which means that there is generally no quadric that passes through 10 arbitrary soints ‘When n > 8 correspondences are available, F ca 1 be estimated using linear least squares by minimizing Vel Fri (10.8) with respect to the coefficients of F under the constrai thatthe vector formed by these coeffi- cients has unit norm, Note that both the eight-point algorithm and its | ast-squares version ignore the rank two property of fundamental matrices.! To enforce this const aint, Luong etal. (1993, 1996) proposed touse the matrix F output by the eight-point algorithm: the basis for a two-step estimation pro- ‘cess: First, use linear least squares to find the epipoles ¢ and e’ that minimize |FTe|? and |Fe'?; second, substitute the coordinates of these points in Eq. 10.6): This yields a linear parameteriza- tion of the fundamental matrix by the coefficients of the epipolar transformation, which can now be estimated by minimizing Eq. (10.8) via linear least s wares. ‘The least-squares version of the eight-point algo: thm minimizes the mean-squared alge- braic distance associated with the epipolar constraint (i. the mean-squared value of e(p, p’ ' Fp’ calculated over all point correspondences). This error function admits a geometric inter- pretation: In particular, we have (P,P) = dp, Fp’) = 2 MC’, FT p), where d(p,.) denotes the (signec) Euclidean distance setween the point p and the line f, and Fp and Fp’ are the epipolar lines associated with y and p'. The scale factors A and 2! are simply the norms of the vectors formed by the first two components of Fp and FT p, and their dependence on the pair of data points observed may bia. the estimation process It is of course possible to climinate the scale fi tors and directly minimize the mean- squared geometric distance between the image points and the corresponding epipolar lines— thats, Lie. Fe) + Pw; FT) This is a nonlinear problem regardless of the para neterization chosen for the fundamental 1atrix, but the minimization canbe initialized withthe result of the eightpoint algorithm, This method was first proposed by Luong eral (1993), and it hs been shown to provide results vastly superior to those obtained using the eight-point methor. As an altemative, Hartley (1995) pro- posed to normalize the linear eight-point algorithm. TY s approach is based on the observation that the poor performance of the original technique isc xe, forthe most par, to poor numerical conditioning. This suggests translating and scaling the ¢ wa so they are centered at the origin and the average distance to the origin is ~/2 pixel. In pract ce, this normalization dramatically im- proves the conditioning ofthe Hinar least-squares esti ation process. Concretely, the algorithm is divided into four steps: Firs, transform the image cootlinates using appropriate translation and sealing operators T : pj —+ p, and 7” : pj -> Bj. Sec nd, use linea least squares to compute "The original algorithm proposed by Longuet- Higgins igor that exential matrices have rank two and two qual singular valves at wellTwo Views 221 the matrix ¥ minimizing Lol Fry. ‘Third, enforce the rank two constraint this can be done using te two-step method of Luong tal described earlier, but Hartley uses instead a technique suggest J by Tsai and Huang (1984) in the calibrated case, which constructs the singular value decompo ition F = USV" of F. Singular value decomposition is formally defined in chapter 12, Let us jst note here that S = diag. 5.1) is a diagonal 3 x 3 matrix with entries r > s > U,V are «-thogonal 3 x 3 matrices, and, as shown in chapter 12, the rank two matrix F minimizing the Fi abenius norm of F — F is simply F = Udiag(r.s.0)V". The last step of fe algorithm sets F = T™FT" as the final estimate of the fundamental matrix. Figure 10.4 shows weak-calibration experiments using a. input data aset of 37 point core spondences between two images ofa toy house. The data poi s ae shown in the figure a smal discs, and the recovered epipolar lines are shown as short line s-gments, Figure 10.4(8) shows the ‘output ofthe least-squares version ofthe plain eight-poin algorithm, and Figure 10.4) shows the results obtained using Hartley's variant ofthis method. As: xpecte, the results are much bet- teri the second case and, in fact, extrerrely close to those obt ined using the geometric distance criterion of Luong etal. (1993, 1996) @) ) [Teta Se [ Wa 195) ] Cag a, 1H [Acbin [233 pneu | ov2giner | O86 pt Figure 10.4 Weak-calibration experiment using 37 pc nt correspondences be: tween two images of a tay house, The figure shows the epipolar lines found by (a) the least squares version ofthe eight-point algorit m, and (b) the normal ized variant ofthis method proposed by Hartley (1995) Note, for example, the ‘much larger error in (a) for the feature point close to. he ottom of the mug. Quantitative comparisons are given inthe table, where te average distances be- tween the data points and corresponding epipolar lines re shown for both tech- niques as well asthe nonlinear algorithm of Luong era . (1993). Data courtesy of Boubakeur Boufama and Roger Mohr.222 ‘The Geometry of Multiple Views Chap. 10 102 THREE VIEWS Let us now go back to the calibrated case where p = as we study the geometric constraints associated with three views of the same scene. Consid r three perspective cameras observing the ‘same point P, whose images aredenoted by p1, p2,an | p3 (Figure 10.5). The optical centers O1, (03, and Oy of the cameras define a trifocal plane that ntersects their retinas along three trfocal lines t,t, and fs, Each one of these lines passes thrc agh the associated epipoles (e.g. the line ', associated with the second camera passes through the projections e:2 and ¢32 of the optical Centers ofthe two other cameras). Each pair of cameras defines an epipolar constre nt—that is, PREP: PIEsPs po i a) : where & denotes the essential matrix associated witt the image pais i «+ j. These three con straints are not independent ince we must have fj€; ex2 = efyEeis = elénen = 0 (10 see why, consider the epipoles es, and ¢32; they are the fir t and second images of the optical center 3 ofthe third camera and are therefore in epipolar ec respondence). ‘Any t0 of the equations in Eq. (10.9) are inde sendent. In particular, when the essential matrices are known, it i possible to predict the posi’ on of the point py from the positions of the two corresponding points p2 and py: Indeed, the fist and thd constraints in Eq (10.9) form Figure 105 Trinocular epipolar geometry. N te thatthe point P does not lie, in general, in the trifoal plane defined by the >ints 0, O3, and Os.Sec. 10.2 Three Views 223 ‘system of two linear equations in the two unknown coo dinates of pi. Geometrically, pris found as the intersection of the epipolar lines associated v ith p2 and ps (Figure 10.5). Thus, the trinocular epipolar geometry offess a solution to the p: oblem of transfer mentioned in the introduction 102.1 Trifocal Geometry ‘A second set of constraints can be ottained by considering three images of a line instead of a Point: The set of points that project onto an image line I ist plane L that contains the line and Pinhole. We can characterize tis plane as follows: If M c snotes a3 x 4 projection matrix, Point P in L projects onto the point pon! when zp = MP or UMP =o, (10.10) where P = (x,y,z, DT is the 4-vecter of homogeneous co mdinates of P and f= (a, b,c)" is the 3-vector of homogeneous coordintes of I. Equation (10 10) is, of course, the equation of the plane L that contains both the optical center O ofthe camer tand the line 1, and L = MT is the coordinate vector of this plane. ‘Two images I; and lz ofthe same Tine do not constrair the relative positon and orientation of the associated cameras since the corresponding planes ‘ and Lz always intersect (unless they are parallel, in which case they ean be thought of as: tersecting at infinity, more on this in chapter 13), Let us now consider thee images f, fa, and 13 ofthe same line {and denote by 1L1, Lz, and Ly the associated planes (Figure 10.6). The inte section of these planes forms a line instead of being reduced to a point in the generic case. Alge vaically, this means that the system uy u|p=0 ur of three equations in three unknowns 2, y, and z must be dey enerate, or, equivalently, the rank of the 3 4 matrix My flim EMy ‘must be 2, which in turn implies that the determinants of all ts 3 x 3 minors must be zero. These pa (Figh re 10.7), we obtain four indepen- dent constraints by rewriting Eq, (10.15) for independent pai of lines passing through pa and Ps (€Bf = [1,0,—uil” and ff = [0,1,—u]? fori = 2, ). These constraints ae tinea in the coordinates of the points py, p2, and p3. When the ten or is known, it can thus be used to predict the positon of, say, ps from the postions of pa anc py inthe other images, giving a second solution tothe transfer problem. 2 Figure 10.7 Given thee images px, pa, and py of the ame point P, and two axbitrary image lines ly and ly passing through p2 and ps, e ray passing through (0, and p, must intersect the line where the planes Lan | L projeting onto /; and ly meet in space.Th» Geometry of Multiple Views Chap. 10 10.2.3 The Uncalibrated Case We can sil derive trilinear constraints inthe image | ne coordinates when the intrinsic parame- ters ofthe three cameras are unknown, Since inthis ase p = Kp and the image line associated th the vector 1 is defined by !"p = 0, we imme iately obtain £ = X-Tl or, equivalently, xT, In particular, Eq. (10.11) holds when p, = p, ad; =i. In the general case, we have TK 0 £=|BGR, bkn|, BK SRE Kats and 4 i 0 Rank(£) = 2 <= Rank le(S | =Rank| 2A, Bb) =2, As bs, where Ai KR; 2,3. te thatthe projection mates associated with our three cameras are now Mi = (Ky 0),.Mz = (A2Ki_b2),and M3 = (AsKi bs). In particular, by and by can stil be interpreted as th» homogeneous image coordinates of the epipoles er2 and es, and the tilinear constraints of 54s. (10.14) and (10.15) still hold when, this time, Gj = bray —A 85, where Aj and A¥ (i = 1,2,3) denote the columns of tz and As. As before, we have IT Gil; = 0 for any par of matching epipolar lines la ands. 10.2.4 Estimation of the Trifocal Tensor ‘We now address the problem of estimating the trifo al tensor from point and line correspon ences established across triples of pictures. The eqt ations defining the tensor are linear in its coefficients and depend only on image measurements. As in the case of weak calibration, we can use linear methods to estimate these 26 parameters. E: ch triple of matching points provides four independent linear equations and every triple of mat ‘hing lines provides two additional linear constraints. Thus, the tensor coefficients can be comp ted from p points and! lines granted that 2p-+1 > 13. For example, 7 tiples of points or 13 tres of lines do the trick, as do 3 triples of Points and 7 triples of lines, and soon. As i the case 0| weak calibration, itis possible to improve the numerical stability of the tensor estimation proces. by normalizing the image coordinates 60 the datapoints are centered atthe origin with an avere ye distance from the origin of /2 pixel. ‘The methods outlined so far ignore that the 2€ parameters of the trifocal tensor are not independent, This should not come asa surprise: The -ssental matrix ony bas five independent coefficients (the associated rotation and translation par meters, the later being only defined up to scale) and the fundamental matrix only has seven. Lik wise, the parameters defining the trifocal tensor satisfy a number of constains, including the a“orementioned equations Gils = O (i = 1, 2,3) satisfied by any pair of matching epipolar line: fz and s. It is also easy to show thatthe tatrices Gj are singular—a property we come back in chapter 13. Faugeras and Mourrain (1995) showed thatthe coeficients ofthe trifocal ten: ar of an uncalibrated trinocular stereo rig, satisfy eight independent constrains, reducing the tota numberof independent parameters to 18. ‘The method described in Harley (1995) enforces the se constraints a posterior’ by recovering. |Sec. 10.3. More Views the epipolese12 and e13 (or equivalently the vectors f; and f in E ,(10.13]) from the linearly estimated trifocl tensor, then recovering ina liner fashion a set of t-nsorcoeficiemts that satisty the constraints. 10.3 MORE VIEWS ‘What about four views? In this section, we follow Faugeras and M vurrain (1995) and first note that clearing the denominators in the perspective projection Eq, (2.1 s) derived in chapter 2 yields uM} — M! (xp tp) P=. where M!, M?, and M? denote the three rows of the matrix WV. (Note that we depart here from our habit of denoting the rows of a projection matrix by m?.,s 12, and mz. This is to avoid possible confusions between the different rows of different matrice inthe rest of this section. It should be clear that AM" and mT denote the same row vector.) ‘Suppose now that we have four views, with associated pr jection matrices M, (j = 1,2,3.4), Writing Eq. (10.16) for each one 0° these yields (10.16) ‘uy M} — Mt iM — ME MM OP=0, where Qf sa ; (10.17) wyM} — MB aM} - MI toMy — Mi ‘This system of eight homogeneous equations in four unkno\ ns admits a nontrivial solu- tion, It follows that the rank of the corresponding 8 x 4 matrix Qs at most 3, or, equivalently, all its 4 4 minors must have zero determinants. Geometrically, «ach pair of equations in Eq. (10.17) represents the ray R; (i = 1, 2,3, 4) associated with the ima se point p,, and Q must have rank 3 for these rays to intersect ata point P (Figure 10.8). ‘The matrix Q has three kinds of 4 x 4 minors: 11, Those that involve two rows from one projection matrix and two rows from another one. ‘The equations associated with the six minors of this type incl de, for example,? ‘uyM} = Mi iM} ~ Mp pe ee _—m | =° 10.18) nM} = MY ‘These determinants yield bilinear constraints on the position o the associated image points Itis easy to show (see Exercises) that the corresponding equ itions reduce to the epipolar ‘constraints of Eq. (10.2) when we take My = (Id O)and A> =(R? —RP 0) General formulas can be obtained by using, for example (x), instead of (u,v) and playing around with indees and tetoral tation. We aban from hi wor exerise ere228 ‘he Geometry of Multiple Views Chap. 10 Figure 108 Four images py, po, Py, an ps of the same point P define this point asthe intersection of the correspondi g rays R (i = 1,2, 3,4). 2. The second type of mirors involves two rows from one projection matrix and one row from cach of two other matrices. There are 48 of tose, and the associated equations include, for example, uy My - My uM} - ME uM} - My be 3M} - MY (10.19) ‘These minors yield trilinear constraints on tt + corresponding image positions. I is easy to show (see Exercises) thatthe corresponding >quations reduce tothe trifocal constraints of Eq, (10.15) when we take M; = (Id 0). [r particular, they can be expressed in terms of ~ the matrices Gj (i = 1,2, 3). Note that this -ompletes the geometric interpretation of the trifocal constraints that express here thatthe ays associated with three images of the same point must intersect in space. 3. The last type of determinant involves one r w of each matrix. The equations associated with the 16 minors ofthis form include, for xample, z ‘uM - Mp M3 - ME Det) Md - ME uM ~MySec. 10.3 More Views 229 ‘These equations yield quadrilinear constraints on th position of the points p, (i = 1,2,3,4). Geometrically, each row of the matrix Q is associated with an image line or equivalently with a plane passing through the optical ce ter of the corresponding camera. ‘Thus each quadrilinearity expresses the fact that the fo r associated planes intersect in a point (instead of not intersecting a: all in the generic cas Let us focus on the quadrilinear equations. Developing determinants such as Eq. (10.20) with respect tothe image coordinates reveals immediately that he coefficients of the quadrilinear constraints can be written as My Mi eyu Det |e | 1021 ME 0.21) Mi where 6x1 = 1 and i, j,k, and | are indexed between 1 an 13 (see Exercises). These coefti- cients determine the quadrifocal tensor (Triggs, 1995). Like its trifocal cousin, this tensor can be interpreted ge »metrically using both points and lines. In particular, consider four pictures p; (i = 1,2, 3,4) of point P and four arbitrary image lines passing through these points, The four planes L; (? = 12,3, 4) formed by the preimages of the lines must intersect in P, which implies in turn thatthe - x 4 matrix ay a | Ma EM Ms must have rank 3 and, in particular, tat its determinant must t= zero, This obviously provides a {uadritnear constraint on the coefficients ofthe four lines l,i = 1,2,3,4). In addition, since each row L7 = 17 M; of Lisa linear combination ofthe rows of the associated matrix My, the coefficients ofthe quatrilinearties obtained by developing Det C) with respect othe coordinates ofthe lines, ae simply the coefficients ofthe quadrifocal ten or as defined by Eq, (10.21). Finally, note that since Det(C) is Hinar inthe coordinate of /, the vanishing of this dter- rminant can be writen as y -q(z,13 le) = 0, where q isa (il near) function of the coordinates of the lines (} (i = 2,3, 4). Since this relationship holds for ay Fine fy passing through py, it follows that p, c q(t. 44). Geometrcally, this means tha the ray passing through O; and 'P1 must also passthrough the intersection of the planes form d by the preimages of lf, and 4s (Figure 10.9). Algebraically, this means that, given the qua focal tensor and arbitrary lines Passing through three images of a point, we can predict the vosition ofthis point in a fourth image. This provides yet another method for transfer. "Note that the quadrifocal constrains are valid in both the calibrated and uncalibrated cases since we have made no assumption on the form of the matric + M;. The quadrifocal tensor is defined by 81 coefficients (or 80 upto scale), but it can be sh: wn that these coefficients salsty 51 independent consraints, reducing the total number of independent parameters to 29. It can also be shown that, although each quadruple of images of t> same point yields 16 indepen- 3. It would be interesting t further investigate the relationship between the trifocal tensor and the scale-constrant condi a, as well as its practical application to the estimation ofthe tifocal geometry. - Show that one ofthe singular values ofan essentic_matrx is 0 and the other two ae equal. (Huang and Faugeras, 1989 have shown thatthe converse is also true—that i, any 3 x 3 matrix with one singular value equal to 0 and the other two equal t: sch other isan essential matrix.) Hint: The singular values of € are the eigen alues of £€7. 2. Exponential representation of rotation matrices. 7 ¢ matrix associated withthe rotation whose axis 3 isthe unit vector a and whose angle is @ can be av wa to be equal to A=) #4 1 a, Use this representation to derive Bq, (103) . The infinitesimal epipolar constraint of Ea. (10.4) vas derived by assuming that the observed soene was static and the camera was moving. Show tat hen the cameras fxd and the scenes moving ‘ith trnlaional velocity» and rotational veloci yw, the epipolar constant canbe rewten at ” ((¥s lure Dp + (p x p)-¥ = 0. Note that this eq tation is now the sum of the two terms appearing in Bq, (104) instead ofthe dfference. nc If and decote the rotation max «1 antation vectors appearing inthe defiion ofthe essential matrix fora moving camera, show hat the objet displacement that yields the same ‘motion field for a static camera is given by the rota ion matrix R7 and the translation vector —R7t, . Show that when the 8 x 8 marx associated wit the eigh-pot algorithm is singular, the eght Points an te two optical centers lie ona quadee:arface (Pagers, 1993). int Use the fact that when a matix is sr pl, there exists some nontrivial linear combi- nation of ts columas that is equal to zero. Also ce advaniage ofthe fact thatthe races epee. senting the two projections inthe coordinalé syste 1 of the fit camera are in this case ld 0) and RRM, : ‘Show that thee ofthe deteinans ofthe 3 x 3 m nor of fo (EG = [BR tr) cantew tenas tx | HGR ERs ty, EEGs ‘Show that the fourth determinant can be writer: as «linear combination ofthese. Show that Eq. (10.18) reduces to Eq. (10-2) whens 4; = (Id 0) and M:=(R? — RTO, Show that Eq. (10.19) reduces to Ea. (10.15) when Mi = (ld 0). 2 Develop Eq, (10.20) with respect tothe image coor: intes, and verify that the coefficients can indeed ‘be weitten in the form of Eg (10.21). Use Eq, (10.28) to calcula the unknowns z;,2,. nd z} in terms of py, py» Ray and &, (= 2,3) ‘Show thatthe value off is directly related to the € polar constrain, and characterize the degree of the dependency of z} — 2} on the datapoints,Programming Assignments 10.10, Implement the eight-point algorithm for weak calibration fr n binocular point correspondences, 10.11. Implement the linear least-squares version of that algorith with and without Harley's precondi- tioning step. { 10.12. Implement an algorithm fo estimating the tifocal tensor fro n point corespondences. 10.13. Implement an algorithm for estimating the trifocal tensor fro a line correspondences.