Proceedings of the 2001 IEEE International Conference on Robotics & Automation Seoul, Korea May 21-26, 2001

Camera Calibration With Genetic Algorithms

Yongniian Zhang and Qiang Jit Department of Computer Science University of Nevada, Reno Department of Electrical, Computer, and Systems Engineering+ Rensselaer Polytechnic Institute qjieecse. rpi.edu
In this paper I, w e present a novel approach based o n genetic algorithms for performing camera c d i bration. C o n t r a i y t o the classical non-linear photogrammetric appr*oach the proposed technique can correctly find the near-optimal solution without the need of initid guesses (with only very loose parameter bounds) u n d ,with a minimum number of control poin.ts ('7 poirits). Results f r o m our extensive study usin,g both synthetic and real image data as well as perforirzaizce comparison with Tsai's procedure dem,onstrute t h e excellent performance of the proposed technique a t e r m s of convergence, accun racy, and robrrstness.

Abstract

1 Introduction
Camera calibration is an essential step in many machine vision and photogrammetric applications including robotics, 3D reconstruction, and mensuration. It addresses the issue of determining intrinsic and extrinsic cainera parameters using 2D image data arid the corresponding known 3D model data. Classical camera calibration techniques can be broadly classificcl into linear approaches [l,2 , 3 and 1 non-linear approaches [l,21. Linear methods have the advantage of coniputational efficiency but suffer from a lack of accuracy and robustness. Non-linear methods, on the other hand, offer a more accurate and robust solution but computationally intensive and require good initial estimates. To get around this problem, one common strategy in computer vision is to attack tlic camera calibration problem by using two steps [4, 51. The first step generates an approximate solution using a linear technique, while the second step improves the linear solution using a non-linear iterative procedure. The first step utilizing linear approaches is key to the success of two-step methods. Approximate solu'A longer version of this paper will appear in March issue of IEEE Transactioiis on System, Man, Cybernetics

tioiis provided by the linear techniques must be good enough for the subsequent non-linear techniques to correctly converge. Being susceptible to noise in image coordinates, existing linear techniques are, however, notorious for their lack of robustness and accuracy [6], [7]. The use of more points can help relieve this problem. However, fabrication of more control points often proves t o be difficult, expensive, and time-consuming. Given a small number (e.g., minimum) of control points, it is therefore questionable whether linear methods can consistently and robustly provide good enough initial guesses for the subsequent non-linear procedure t o correctly converge t o the optimal solution. Another problem is that almost all nonlinear techniques employed in the second step use variants of conventional optimization techniques like gradientdescent, conjugate gradient descent or the Newton method. They therefore all inherit well known problems plaguing these conventional optimization methods, namely, poor convergence and susceptibility to getting trapped in local extrema. If the starting point of the algorithm is not well chosen, the solution can diverge, or get trapped at a local minimum. To alleviate the problems with existing camera calibration techniques, we explore a n alternative paradigm based on genetic algorithms (GAS) t o conventional non-linear optimization methods. GAS were designed t o efficiently search a large, non-linear, poorly-understood spaces and have been widely applied in solving difficult search and optimization problems including camera calibration [8]. Results from our study are encouraging and promising. The proposed GA approach can quickly converge t o the correct solution without initial guesses and with the minimum number of points (7 points).

Background

In this section we provide short introductions t o genetic algorithms and perspective geometry used for camera calibration.

0-7803-6475-9/01/$10.000 2001 IEEE

21 77

2.1

Genetic Algoritlhms

be parameterized as

GAS are stochastic, parallel search algorithms based on the mechanics of natural selection and the process of evolution. The three basic GA operations are evaluation, selection, and recombination. Evaluation of each string which encodes a candidate solution is based on a fitness fuiiction that is problem dependent. Selection is done on the basis of relative fitness and it probabilistically culls solutions from the population that have relatively low fitness. Good individuals (with high fitness) are then assigned for further reproduction. Recombination, which consists of mutation and crossover, imitates sexual reproduction. Mutation insures against the permanent loss of genetic material during the selection process. Crossover is a structured yet stochastic operator that allows information exchange between candidate solutions.

R=

(;:

7'11

;;

7'12

~ 1 3

;:)

T=

( t, )

(2)

The T i J in matrix R can be expressed as the function of camera pan angle w , tilt angle 4 and swing angle. The collinearity of 3D world coordinate X and 2D image coordinate p can be written as

where sx and sy are scale factors(pixels/~nnz)due to spatial quantization. uo arid 210 are the coordinates of the principle point in pixels relative t o image frame. q is a vector of all camera parameters as defined in equation 4.

2.2

Perspective Geometry

Objective Function

Figure 1 shows a pinhole camera model that does not consider any type of lens distortions. Let X = be (x y , z ) ~ a 3D point in an object frame and U = (U w) the corresponding image point in the image frame. Let X, = ( p q s ) be the coordinates of ~ X in the camera frame and p = ( e r)T be the coordinates of U in the row-colunin frame as illustrated in figure 1. The image plane, which corresponds t o the image sensing array, is assumed t o be parallel to the (Xc,Yc)plane and at a distance f to the origin, where f denotes the focal length of the camera. The relationship between the camera frame C, and object frame COis given by

'The main task of camera calibration in 3D machine vision is t o obtain an optimal set of the interior camera parameters ( ( u g ,vug), s,. , sg , f ) T and exterior camera parameters ( w , 4, K , t,. t,, using known control points in the 2D image and their corresponding 3D points in the world coordinate system. Let q be an vector consisting of the unknown interior and exterior camera parameters, that is,
q = b o , VI, f , s x ,s,, w , 4 , 4 t x ,t,,

LIT

(4)

For notational convenience, vie rewrite q as q = ( q l , q 2 , . . . , q 1 1 ) ~ where 4 1 , q 2 and 4 1 1 correspond to , U O , wo and t , respectively in toheprevious notation used for q. Assume q is a solution of interior and exterior camera parameters and 'q C Q, then we have

Imagc Framc

Ci

Object Frame

I/irinciple Point
Figure 1. Perspective projection geomet,ry

where 4 : and q+ are the lower and upper bounds of % qi. The bounds on parameters can be obtained based on the knowledge of camera. Any reasonable interval which may covers possible parameter values may be chosen as the bound of paramleter q i . For example, we may have w , 4, K E [-T, f E [20,70] as in our T], test cases and so on. An optimal solution of q with M control points can be achieved by minimizing

where R is a 3 x 3 rotation matrix defining the camera orientation and T is a translation vector representing the camera position. R and T can further

i=l

where Xi = ( z i , y i , z i ) is the i t h 3D point, g and w are defined in equation 3

2178

A key issue tliat arises in this approach is the extremely large search space caused by the presence of uncertainties. This presents a serious challenge to conventional minimization procedures since there may be several local iiiininia , and the choice of starting point will determine which minimum the procedure converges to, or whether it will converge at all. If starting points are far away from the desired minimum, the traditional optimization techniques could converge erroneously.

qk with a probability 1 - p . It is reasonable t o expect that it is equally likely for a GA t o increase q k as to decrease q k . Then the next current point q;+' is given by

q;+,+'=

4; + I A ( t ,4;) + (1- l ) A ( t ,4 J q; if > q; qk i f q;+l < 4;

(8)

GA Operators and Representation

Designing an appropriate encoding and/or recombination method is often crucial t o the success of this algorithm. To improve GA's convergence, we proposed a n e w mutation operator that determines the amount and direction of perturbation in the search space. Mutation can be viewed as one dimensional or local search, while crossover performs multi-dimension, or more global, search.

where A(t,q;) and A(t,q1;)are the step sizes for the upper and lower bounds of q k respectively. I is an indicator function assuming the value of 0 and 1 depending on the outcome of a coin toss. The crucial issue is the amount of perturbation (step size) of point q k in the interval [q;, g z ] . TOO small perturbation may lead to sluggish convergence, while too large perturbation may cause the GA to erroneously converge or even oscillate. Since q k E [q1;,q;] E S holds, q k must be a fraction, say e, of the way between its lower bound q c and upper bound q t , i.e.

4.1

Representation

GA chromosome are usually encoded as bit strings and a long binary string is required in order to represent a large continuous range for each parameter. Instead, we encode the GA chromosome as a vector of real numbers. Each camera parameter q i , i = 1,..., 71 is initialized to a value within its respective bounds as defined in equation 5. The chromosome vector may be defined as

; E Assuming the successive point is q;+', where q" [ q i , q ; ] , we can utilize the golden section to determine the optimal step size of ' '4 ; as

where q: is an iiidividual from population N at t t h : is an individual from the new gengeneration. 9' eration after genetic selection; and qC indicates the ; parameter that is niodified during the evolutionary process.

where c is the golden fraction with value of (3 & ) / 2 ; and a and b represent, respectively, 1 - q k l 4 ; and 1qk - q i l . Since equation 10 linearly converges to its ideal value, to improve the GA's efficiency and skip away unimportant search region in the early stage of evolution processing, we then incorporate evolution time t in equation 10, i.e,

4.2

Mutation

Our mutation scheme comprises two steps: determining the search direction and simultaneously determining the step size in the selected search direction. In Equation 5, the search space Q should be a convex space S. The task of the GA is to determine an uii~ciiowiioptimal point ql E [ q i , q t ] in the convex space S which minimizes the global error function of equation G at that point. Assuniing that the probability of receiving a correct step size from the GA is p , whenever the current q k 5 qi the GA correctly increases q k with a probability p . It may also, however, incorrectly decrease

where r is a random variable distributed on the unit interval [0, I], a lies in [I,1.51 and T denotes the total number of iterations. The essence of our mutation scheme, termed golden section, lies in integrating equations 10, 11, 12 together and in each generation (or iteration) the scheme stochastically chose one of them to determine the position of the new current point. The random determination of step size allows discontinuous jumps in the parameter interval, and then golden

2179

section is used t o control the search direction. This ultimately makes the GA coiiverge more accurately t o a value arbitrarily close t o the optimal solution. Additionally, proposed mutat.ion scheme requires insignificant computational time.

5.1

Synthetic Images;

4.3

Crossover

Crossover produces new points in the search space. The initial population forms a basis of the convex space S and a new individual in current generation is generated by a linear combination of the parent individuals in the previous generation. Let qt and ql f be two individuals from population N at generation t. They satisfy
q, = {q3 E [q,,q,+]

We generated 200 independently perturbed sets of control points for each noise level so that an accurate ensemble avcrage of the results could be obtained. To ensure fair comparison, GA parameters were ideiitical in all test cases. We first investigated how image noise and control points affect the performance of our approach. For this study, the initial camera parameter bounds and their ground truth are given in Table 1.
Table 1. Camera parameter groundtruth and bounds
20, 40 110, 160 110, 160 200, 300

cS}, i

= 1. .., N j = 1,..,n (13)

144.0 144.0

where N denotes the population size, and n the nuniber of parameters (or chromosome length). Following equation 13, a new individual q, in generation t 1 can be expressed as a linear combination of two arbitrarily selected individuals from the previous generation t , that is,

170, 330
-38.0 35.0 1210.0 -1.90146 0.20916 0.15152
-SO, 50 -80, 50 900, 1400
-7r, 7r
-7r, 7r

where cy ranges within [O, I]. p is a bias factor that increases the contribution from the dominating individual with a better fitness at current stage. Assuming non-negative fitness function, p can be determined from the following equations.

where f ( * ) denotes the GAs fitness function defined in equation 6.

Experimental Results

This section describes experiments performed with synthetic and real images to evaluate the performance of our approach in terms of its accuracy and robustness under 1) varying amounts of image noise, 2 ) different numbers of control points, and 3) different ranges of parameter bounds. Furthermore, we describe results from a comparative performance study of our approach with that of Tsais calibration algorithm.

Results from this experiment show that sufficient accuracy can be achieved with the minimum number of control points without loss of accuracy and robustness. Compared with conventional calibration methods, this represents a practical advantage since creating many redundant control points usually is an expensive and time-consuming procedure. Second, this experiment demonstrates the algorithms considerable accuracy and robustness in the presence of different noise levels as seen from both camera errors and image pixel errors. The camera errors are also within acceptable margins given considerable image noises ( 3 pixels). We now present results from the experiment carried out to test the stability of the proposed method under varying initial bounds. Practically, camera scale factors (sz, and principle point ( u 0 , U O ) can sg) be restricted in a relative small ranges based on camera manufacture information and possible hardware errors. Accordingly, in this experiment we assume that bounds of s, sy,u 0 and vo can be maintained , in a reasonable range (which is often practiced in other existing calibration techniques) and then examine the change in performance by means of varying bounds of focal length and extrinsic parameters. We investigated 4 different cases using minimum control points (7 in this case). The initial bounds and corresponding estimated c.amera parameters under free-noise ( 0 = 0) are summarized in Table 2 (Ground truth see Table I).

21 80

Table 2. Estimated camera parameters under different parameter bounds ( o = 0)

Case 1
Estimated 26.29 138.40 244.06 192.00 -33.91 35.03 1223.05 -1.90156 0.206670 0.1525811 Estimated 27.2i

Case 2
20,40 120,150 230, 270 180, 200 -50, 50 -50, 50 1000, 1400

0 , 50

1 ;1 1
&

1100, 1400 -x,x -x,x

-?r, x

-x,7r -x,7r
-?r,

I
1
1

Case 3
20,50

Case 4 in,60

Estimated 27.87 132.37 245.17 191.82 -34.40 35.00 1240.43 -1.87552 0.~26407 0.173463 Estimated 25.42

180, 200 -110, 110 -110, 110

700,

14nn

194.91 -34.21 33.91 1207.32

-x,7r
-x,7r
-7r,

-1.851877 n.mm5 0. 195439

197.46 -37.26 33.01 1235.12 -1.820067 0.275590 0.225426

most popular camera calibration method for both computer vision and photogrammetry communities. We synthetically generate 108 non-coplanar control points to test Tsais calibration method and use the same methodology as described earlier in this section to produce the perturbed data sets. The experiments were performed over 16 different noise levels and the final results for each noise level is the average of results from 200 independently perturbed sets of control points. To see if equivalent results can he achieved by our proposed method with fewer calibration points, we use 108 and 8 of control points in our approach. The parameter ground truth and initial parameter bounds in this experiment are given in Table 3.

Table 3. Camera parameter ground truth and bounds

Ground T r u t h
70.0 43.4s

In four cases shown in Table 2, we gradually enlarge the bounds of focal length and the translation parameters, and keep all camera angles range from -nto Sn. Table 2 indicates that enlarging parameter bounds only causes marginal impact on the calibration accuracy. In all cases, the pixel errors and camera errors are within acceptable margin. Errors of some parameters such as t , become moderately large with an increase in their initial ranges, However, the pixel errors maintain almost the same level regardless of their bounds. This is because our GA can always find a correct search direction towards the global niiiiimuiri by gradually reproducing a fit parameter set, and errors caused by deflection of one parameter may offset by others. Particularly, if we can restrict the range of intrinsic parameters (except for focal length) in a reasonable range as practiced by other calibration techniques, the calibration would be more accurate as demonstrated in case 4, and this will be of riiore practical interest. More importantly, the results imply that the bounds of parameters can be any reasonable interval which may cover possible parameter values without the need of expert knowledge. This means that regardless of the parameters bounds, the G R algorithm can always converge to a solution very close to the optimal solution. This is significant and i s exactly what we set out to achieve, i.e., our method ca*navoid local rninima without the need of good iwitinl estimates.

43.48 258.0 204.0 -100.0 -85.0 2000.0 0.52:~m 0.017453 0.034907

Parameter Bound 40, 85 40, 60 40, 60 250, 270 190, 210 -150.0. 150.0 -150.0, 150.0

14on.0, ~ 6 n n . n -x.7r -x,7r -x,x

Figure 2 depicts the comparison results with different noise levels. Note that in Figure 2 , we ignored results froin Tsais method as noise level (0) is over 2 . 2 since parameter errors tend to skyrocket for Tsais method if D > 2.2. We can conclude from Figure 2 that when synthetic image data has no perturbations or very little perturbations, Tsais calibration technique is extremely accurate. The accuracy of the Tsais method, however, decreases dramatically after a short interval of noise level (0 5 0 5 0.4). For example, if the noise level cr is over 1.5, some parameters deteriorate severely and the pixel errors almost exponentially increase. We can therefore conclude that the accuracy potential of Tsais methods is very limited in noise situation. In contrast, our approach shows that camera errors remain approximately constant for various noise levels and pixel error increases linearly. This proves

the accuracy and robustness of our technique under noisy conditions . The method therefore has
higher capability of immunization against the image perturbation. Furthermore, increasing control points is almost not helpful for the algorithm accuracy and robustness in our approach as illustrated in Figure 2. Once again, it implies then, that with minimum control points (8 in this case), our approach is

5.2

Comparison with Tsais

To further study the performance of the proposed method, we con-qarecl our method with Tsais two step calibration technique [4], which perhaps is the

2181

able to achieve accuracy and robustness equivalent to or even better than that wi.th highly redundant control points.
181
, , , , ,

Conclusions

We presented a new approach to the problem of camera calibration based on genetic algorithms with novel genetic opcrators. Our study performed with both synthetic and real iinagcs demonstrates the excellent performance of our technique in terms of convergence, accuracy, and robustness. The comparison with Tsai's calibration technique shows that our approach has high potential in the various noise situation. Specifically, the proposed method enjoys several favorable properties:
0

0 . 4 0 8 1.2 1 6 2 2 . 4 2 . 8 GanSSlan Pixel NoiseiSlgmaI

It does not require initial guesses of camera parameters (with only very loose bounds) to converge correctly. It achieves the sufficient accuracy and robustness with the minimum number (7) of calibration points and for noisy images. It is tolerant to the large range of image perturbations.

References
[l] Y.I. Abdel-Aziz and H. M. Kara-ra, "Direct linear transformation into object space coordinates in close-range photogrammetry," in Proceedin.gs of Symp. Close-Range Photogrammetry, pp. 1-18, 1971.

0
0.03

0 . 4 0.8 I 2 1.6 2 2.4 2.8 Gaussian Pixel Nalse(Sigma!

[a]
I

5
x

0.025

D. C. Brown, "Close-range camera calibration," Photogrammetric Engeering and Remote Sensing, vol. 37> no. -, pp. 855-866, 1971.

0.02 0.015
0.01

5
2"
~

.-

2
=

0.005

[3] W. Faig, "Calibration of close-range photogrammetry systems: Mathematical formulation," Photogrammetric Engeering and Remote Sensing, vol. 41, no. -, pp. 1479-1486, 1975.
0

o
0 . 4 0.8 1 2 1.6 2 2 4 2.8 Gaussian Pixel NolseIS19maI

0 . 4 0.8 1.2 I 6 2 2 4 2.8 Gaussian Pixel ~ a i s e ( S i m a !

[4] R. Y. Tsai, "A versatile camera c-alibration technique for

Figure 2. Side-by-side comparison with Tsai's calibration algorithm. First ccllumn: estimated error of camera extrinsic parameters (T,, Ty, T,, R); second column: estimated errcc of camera intrinsic parameters ( f , sz, sy, U O ,WO); a.nd last figure on the second column: mean pixel error.

highaccuracy 3d machine vision metrology using offtheshelf tv cameras and lenses," I E E E Journal of Robotics a n d Automation, vol. RA-3, no. 4, pp. 323-344, 1987.

[5] 3 . Weng, P. Cohen, and M. Herniou, "Camera calibration with distortion models and accuracy evaluation," IEEE Transaction of Pattern Analysis and Machine Intelligence, vol. 10, no. 14, pp. 965-980, 1992.
[6] X. Wang and G . Xu, "Camera parameters estimation and evaluation in active vision system," Pattern Recognition, vol. 29, no. 3, pp. 439-447, 1996.

[7] It. M. Haralick, H. Joo, C. Lee, X. Zhang, V. Vaidya,

5.3

Experiments w i t h Real Images

We applied our method to real images of industrial parts and the accuracy is judged by visual inspection of the alignment between the image of a part and the re-projected outline of the part using estimated camera parameters. Results of visual inspection shows excellent alignment between the original images and the projected outlines, which further prove the performance of our proposed approach on real images.

and M. Kim, "Pose estimation from corresponding point data," IEEE D a n . on Systems, Man, and Cybernetics, vol. 19, no. 6, pp. 1426-1446, 1989. [Sj If. Y. Huang and F. H. Qi, "A genetic algorithm approach t o accurate calibration of camera,'' Infrared and millmeter waves, vol. 19, no. 1, pp. 1-6, 2000.

21 a2