Robot Tool Center Point Calibration

using Computer Vision

Master’s Thesis in Computer Vision
Linköping Department of Electrical Engineering
by

Johan Hallenberg

LiTH - ISY - EX - - 07/3943 - - SE

Department of Electrical Engineering

Linköpings University, SE-581 83, Linköping, Sweden

Linköping February 2007

Printed by:
UniTryck, Linköping, Sweden

Distributed by:
Linköpings University
Department of Electrical Engineering
SE-581 83, Sweden
Robot Tool Center Point Calibration
using Computer Vision
Master’s Thesis at Computer Vision Laboratory
University of Linköping
by
Johan Hallenberg
Reg nr: LITH - ISY - EX - - 07/3943 - - SE

Supervisors: Klas Nordberg

ISY, Linköping University

Ivan Lundberg
ABB Corporate Research Center, Sweden

Examiner: Klas Nordberg

ISY, Linköping University

Linköping 2007 - 02 - 05
Abstract

Robot Tool Center Point Calibration using Computer Vision

By: Johan Hallenberg
MSc, Linköping University, February 2007

Examiner: Klas Nordberg

Supervisors: Klas Nordberg and Ivan Lundberg

Today, tool center point calibration is mostly done by a manual procedure.

The method is very time consuming and the result may vary due to how skilled
the operators are.
This thesis proposes a new automated iterative method for tool center point
calibration of industrial robots, by making use of computer vision and image pro-
cessing techniques. The new method has several advantages over the manual cali-
bration method. Experimental verifications have shown that the proposed method
is much faster, still delivering a comparable or even better accuracy. The setup
of the proposed method is very easy, only one USB camera connected to a lap-
top computer is needed and no contact with the robot tool is necessary during the
calibration procedure.
The method can be split into three different parts. Initially, the transforma-
tion between the robot wrist and the tool is determined by solving a closed loop
of homogeneous transformations. Second an image segmentation procedure is
described for finding point correspondences on a rotation symmetric robot tool.
The image segmentation part is necessary for performing a measurement with six
degrees of freedom of the camera to tool transformation. The last part of the pro-
posed method is an iterative procedure which automates an ordinary four point
tool center point calibration algorithm. The iterative procedure ensures that the
accuracy of the tool center point calibration only depends on the accuracy of the
camera when registering a movement between two positions.

v
Acknowledgements

I have had the opportunity to write my Master’s Thesis at ABB Corporate Re-
search Center, Mechatronics department, in Västerås. This period has been a
great experience for me and I have really enjoyed working with the project.
I want to thank my supervisor Development Engineer Ivan Lundberg at ABB
Corporate Research Center for his great enthusiasm, all the conversations, and for
the extremely good supervision throughout the project.
I also want to thank my supervisor and examiner Associate Professor Klas
Nordberg at Linköping University, for all the great phone conversations, the good
tips and all the support and guidance throughout the project.
Special thanks also to PhD. Mats Andersson at the Center for Medical Image
Science and Visualization for his great interest in the project and helpfully ideas.

Linköping, February 2007

Johan Hallenberg

vii
Notation

Symbols
• f (x, y) denotes a 2D function or an image.
• x, X denote scalar values.
• x, X denote vectors or coordinates.
• X denotes a matrix.

Operators
• f (x, y) ∗ g(x, y) denotes the convolution between the image f (x, y) and the
image g(x, y).
• f (x, y) ? g(x, y) denotes the cross correlation between the image f (x, y)
and the image g(x, y).
• XT denotes the transpose of X.
• X† denotes the pseudo inverse of X.

Glossary
TCP Tool Center Point.
Jog Manually move the robot with joystick.
Base frame The robot’s base coordinate system.
Robtarget Cartesian target.
DOF Degrees of freedom.
Q.E.D ”Quod Erat Demonstrandum” a latin phrase meaning
”which was to be demonstrated”.

ix
Contents

1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Problem Specification/Objectives . . . . . . . . . . . . . . . . . . 2
1.3 Delimitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Robotics 3
2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Coordinate Systems . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.1 Tool Center Point, TCP . . . . . . . . . . . . . . . . . . . 4
2.2.2 Base frame . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.3 World frame . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.4 Wrist frame . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.5 Tool frame . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 TCP Calibration algorithm . . . . . . . . . . . . . . . . . . . . . 7

3 Theory 9
3.1 Image Processing Theory . . . . . . . . . . . . . . . . . . . . . . 9
3.1.1 Threshold . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.1.2 Morfological operations . . . . . . . . . . . . . . . . . . 9
3.1.3 Structure Element . . . . . . . . . . . . . . . . . . . . . . 9
3.1.3.1 Erosion . . . . . . . . . . . . . . . . . . . . . . 10
3.1.3.2 Dilation . . . . . . . . . . . . . . . . . . . . . 10
3.1.3.3 Opening . . . . . . . . . . . . . . . . . . . . . 11
3.1.3.4 Closing . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Computer Vision Theory . . . . . . . . . . . . . . . . . . . . . . 12
3.2.1 Camera System . . . . . . . . . . . . . . . . . . . . . . . 12
3.2.2 Pinhole Camera Model . . . . . . . . . . . . . . . . . . . 13
3.2.3 Camera Calibration . . . . . . . . . . . . . . . . . . . . . 14
3.3 Transformation Theory . . . . . . . . . . . . . . . . . . . . . . . 16
3.3.1 Homogeneous Transformation . . . . . . . . . . . . . . . 16

xi
xii CONTENTS

3.3.1.1 3D translation . . . . . . . . . . . . . . . . . . 16
3.3.1.2 3D Rotation . . . . . . . . . . . . . . . . . . . 16
3.3.1.3 Homogeneous Transformation Matrix . . . . . 16
3.3.2 Screw Axis Rotation representation . . . . . . . . . . . . 17
3.3.3 Rodrigues´s formula . . . . . . . . . . . . . . . . . . . . 18

4 Determination of Tool Center Point 21

4.1 Overview of the system . . . . . . . . . . . . . . . . . . . . . . . 21
4.1.1 Equipment . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2 Determine T2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.3 Finding X1 and X2 transformation matrixes . . . . . . . . . . . . 24
4.4 Importance of measure A and B with 6 DOF . . . . . . . . . . . 29
4.5 Iterative Method for increasing the accuracy . . . . . . . . . . . . 31
4.6 Alternative method for rotation symmetric tools . . . . . . . . . . 33

5 Image Segmentation of Robot Tool 35

5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5.1.1 Open Computer Vision Library (OpenCV) . . . . . . . . . 35
5.2 Creation of binary mask image . . . . . . . . . . . . . . . . . . . 35
5.2.1 Image registration . . . . . . . . . . . . . . . . . . . . . 36
5.2.2 Background subtraction . . . . . . . . . . . . . . . . . . 38
5.2.2.1 The Hough filling method (HFM) . . . . . . . . 39
5.3 Edge Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.3.1 Canny’s Edge Detection . . . . . . . . . . . . . . . . . . 40
5.4 Contour Retrieving . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.4.1 Freeman Chain Code Contour representation . . . . . . . 41
5.4.2 Polygon Representation . . . . . . . . . . . . . . . . . . 42
5.4.3 Freeman Methods . . . . . . . . . . . . . . . . . . . . . . 42
5.4.4 Active Contour (The Snake algorithm) . . . . . . . . . . . 42
5.5 Finding the contour matching the tool shape . . . . . . . . . . . . 44
5.5.1 Convex Hull and its defects . . . . . . . . . . . . . . . . 44
5.5.2 Polygonal Approximation of Contours . . . . . . . . . . . 45
5.5.2.1 Douglas-Peucker Approximation Method . . . . 46
5.6 Geometric constraints . . . . . . . . . . . . . . . . . . . . . . . . 46
5.7 Corner Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.7.1 Harris corner detector . . . . . . . . . . . . . . . . . . . 47
5.7.1.1 Sub pixel Accuracy . . . . . . . . . . . . . . . 48
5.7.2 Fast radial . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.7.3 Orientation Tensor . . . . . . . . . . . . . . . . . . . . . 50
CONTENTS xiii

6 Results 55
6.1 Repetition Accuracy of the Camera . . . . . . . . . . . . . . . . . 55
6.1.1 Averaging to achieve higher accuracy . . . . . . . . . . . 56
6.1.2 How does the number of points affect the repetition accuracy 57
6.2 Camera versus Robot measurements . . . . . . . . . . . . . . . . 59
6.2.1 Rotation measurements . . . . . . . . . . . . . . . . . . . 59
6.2.2 Translation measurements . . . . . . . . . . . . . . . . . 61
6.2.2.1 Averaging to achieve higher accuracy . . . . . . 64
6.3 Accuracy of the TCP Calibration method . . . . . . . . . . . . . 67
6.3.1 Averaging to obtain a higher accuracy . . . . . . . . . . . 69
6.4 Repetition Accuracy of the TCP Calibration method . . . . . . . . 69
6.4.0.1 How does the number of points affect the repe-
tition accuracy of the TCP calibration method . 72

7 Discussion 73
7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.1.1 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.1.2 Major Problems/known difficulties . . . . . . . . . . . . . 74
7.1.2.1 Finding distinct points . . . . . . . . . . . . . . 74
7.1.2.2 Light reflections in the tool . . . . . . . . . . . 74
7.1.3 Fulfillment of objectives . . . . . . . . . . . . . . . . . . 75
7.2 Future Development . . . . . . . . . . . . . . . . . . . . . . . . 75
7.2.1 Rotation symmetric tools . . . . . . . . . . . . . . . . . . 75
7.2.2 Light reflections in the tool . . . . . . . . . . . . . . . . . 76
7.2.3 Image Segmentation controlled by CAD drawings . . . . 76
7.2.4 Neural network based methods . . . . . . . . . . . . . . . 76
7.2.5 Online correction of the TCP calibration . . . . . . . . . . 76
7.2.6 Other image processing libraries . . . . . . . . . . . . . . 77

A Appendix 79
A.1 Finding intrinsic parameters . . . . . . . . . . . . . . . . . . . . 79

Bibliography 87
Chapter 1

Introduction

This chapter begins with a short background, then the problem is specified and the
objectives and delimitations are determined. Finally the disposition of the thesis
is presented.

1.1 Background
All robots delivered by ABB are calibrated in the production. When the calibra-
tion procedure is done the robot is calibrated up to the last axis and is thereby able
to calculate the positions of all axes during a movement. When a tool is mounted
on the last axis (the tool flange) the robot needs to know the actual position of the
active point of the tool, the tool center point, which for instance can be the muzzle
of a spot welding tool. For this reason a tool center point calibration has to be
performed every time the tool is changed.
Today the TCP calibration is done manually by moving the robot, letting the
TCP brush against a fixed point in the environment. The fixed point is typically
the tip of a nail. The tool center point needs to brush against the tip of the nail
with great precision and from at least four different angles. Then the coordinate
of the tool center point can be calculated by the robot in relation to the robot’s tool
flange coordinate system. The method is time consuming and the result may vary
due to how skilled the robot operator is. The resulting accuracy of todays method
is approximately ±1 mm.
Products for automating the TCP calibration exist, for instance BullsEye for
calibration of arc welding tools and TCP-Beam for calibration of spot welding
tools. The disadvantage with these methods are that they are specialized to one
single type of robot tools.

1
2 CHAPTER 1. INTRODUCTION

1.2 Problem Specification/Objectives

The purpose of this thesis is to investigate whether or not a computer vision
method can be used to automate the tool center point calibration process. If it
is possible to find a method then the accuracy of the method is also of great in-
terest. The objective is to find a method with an accuracy of at least ±1 mm,
that automates the tool center point calibration procedure. For development of the
method C#, C/C++ and Matlab are allowed to be used, but the final demonstration
software shall be programmed in C# and C/C++.

1.3 Delimitation
To deliver a method with the ability to calibrate all kinds of robot tools during a
5 months Master’s thesis is of course impossible. Instead the method is of impor-
tance, and any kind of tool is permitted to be used to demonstrate the method.

1.4 Thesis Outline

Chapter 1 Chapter 1 describes the background to the project.
The problem specification, delimitations and the
thesis outline are also presented.
Chapter 2 Chapter 2 gives a short introduction to robots and
the chapter concludes with a description of a four
point tool center point calibration algorithm.
Chapter 3 In chapter 3 some theory needed to understand the
rest of the thesis are presented. The Chapter is
divided in image processing theory, computer vision
theory and transformation theory.
Chapter 4 Chapter 4 describes the proposed method in detail and the
configuration of the devices is presented.
Chapter 5 Chapter 5 describes in detail how the image segmentation
of the robot tool was done.
Chapter 6 Chapter 6 presents the results of several tests performed
to investigate the accuracy of the proposed method.
Chapter 7 Chapter 7 presents the conclusions and difficulties of
the project. The chapter concludes with a discussion of
future development.
Appendix A Appendix A presents some results from the camera calibration,
performed with the Camera Calibration Toolbox in Matlab.
Chapter 2

Robotics

This chapter will give an overview of robots and their coordinate systems. The
chapter concludes with a description of a four point tool center point calibration
algorithm used in the robots today.

2.1 Overview
Traditionally a robot consists of a mechanical arm and a computer controlling its
position and movement. In this project only serial kinematics manipulators will be
concerned. The serial kinematics manipulators consist of several axes connected
by joints see figure 2.1.

Figure 2.1: The ABB Robot IRB 6600 (Courtesy of ABB).

3
4 CHAPTER 2. ROBOTICS

Usually the mechanical arm (the manipulator) is divided into an arm, a wrist
and an end-manipulator (the tool). All the manipulators used during this project
have got six degrees of freedom, making it possible to position the tool anywhere
in the robot workspace with a given orientation. Internally the robot end effector
orientation is represented by quaternions.
The ABB robots can be manually controlled by using the flexpendant. The
flexpendant is a hand held controller connected to the computer controlling the
robot. A robot program telling the robot how to move can be written in the
program language RAPID. The RAPID programs can be written directly in the
flexpendant but it is also possible to develop the RAPID program on a PC and
then transfer the program (by a TCP-IP connection or a USB connection) to the
computer controlling the robot.

2.2 Coordinate Systems

2.2.1 Tool Center Point, TCP

When a robot movement is programmed by specifying a path of robtargets for the

robot to follow, then all the robot movements and robot positions are relative to
the Tool Center Point (TCP). Normally the tool center point is defined to be the
active point of the tool e.g. the muzzle of a spot welding tool or in the center of a
gripper.
Several tool center points can be defined e.g. one for each tool, but only one
tool center point can be active at a given time. When the robot is programmed to
move along a given path, it is the tool center point that will follow the actual path.
It is possible to define other coordinate systems and program the robot to move
according to these coordinate systems. The tool center point is then expressed in
relation to the coordinate system used in the program [1].

2.2.2 Base frame

The robot base coordinate system is located on the robot base. The origin of the
base frame is located at the intersection of axis 1 and the robot’s mounting surface.
The x axis is pointing forward, the y axis points in the robot’s left side direction
and the z axis coincides with the rotational axis of axis 1. Figure 2.2 illustrates
the base frame [1].
2.2. COORDINATE SYSTEMS 5

Figure 2.2: The base coordinate system (Courtesy of ABB).

2.2.3 World frame

If several robots are working in the same area, a world frame can be set up and the
robots base coordinate systems can be expressed in relation to the world coordi-
nate system. It is then possible to make RAPID programs telling the robot to move
to a certain position in the world frame. If one of the robots is mounted up side
down, the definition of a world frame will of course simplify the programming of
the robot. Please, see figure 2.3
[1]

Figure 2.3: The world coordinate system (Courtesy of ABB).

2.2.4 Wrist frame

The wrist frame is fixed to the tool flange/mounting flange (the surface where the
tool is to be mounted). The origin of the wrist frame is positioned in the center
6 CHAPTER 2. ROBOTICS

Figure 2.4: The wrist coordinate system (Courtesy of ABB).

of the tool flange and the z axis coincides with axis six of the robot. There is a
calibration mark at the tool flange, see figure 2.5. The x-axis of the wrist frame
is always pointing in the opposite direction of the calibration mark and the y-axis
is achieved by constructing an orthogonal coordinate system axis to the x and z
axes.

[1]

2.2.5 Tool frame

To be able to define a tool center point, a tool frame is needed. The tool frame is a
coordinate system on the tool, and the origin of the tool frame coincides with the
tool center point. The tool frame can also be used to obtain information about the
direction of the robot movement.

[1]

Figure 2.5: The tool coordinate system (Courtesy of ABB).

2.3. TCP CALIBRATION ALGORITHM 7

2.3 TCP Calibration algorithm

Today, the TCP calibration is done by jogging the robot making the TCP brush
against a fixed point in the close surrounding to the robot. The fixed point is for
instance the tip of a nail. By jogging the robot, making the TCP brush against the
tip of the nail from at least four different orientations, the coordinate of the TCP
in relation to the robot base frame is calculated.
Let T1 be the position of the tool flange in relation to the robot base frame,
e.g. the robot’s forward kinematics. Assume N different positions of the robot
are known, making the TCP brush against the fixed point in the environment.
Then
h
N different transformations
iT
T1i are also known, where i ∈ [1...N ]. Let
T CPx T CPy T CPz be the translation from the origin of the tool flange
coordinate system to the tool center point.
Let Q be the homogeneous transformation from the tool flange to the tool center
point:
1 0 0 T CPx
 
 0 1 0 T CP 
y 
Q=

 0 0 1 T CPz 


0 0 0 1
It is obvious that equation 2.1 is satisfied, due to the fact that the tool center
point is at the same coordinate independent of which of the N robot positions that
is examined.
T1i Q = T1j Q (2.1)
Where i, j ∈ [1...N ] and i 6= j.

Denote:
 
a11 a12 a13 a14
 
Ra ta 
 
 a a22 a23 a24  
21  
T1i =  =
   (2.2)
 a31 a32 a33 a34
 

 0 0 0 1 

a41 a42 a43 a44


b11 b12 b13 b14

   
 b21 b22 b23 b24   Rb tb 
T1j =  = (2.3)
   
b31 b32 b33 b34

   
b41 b42 b43 b44 0 0 0 1
Examine one row of equation 2.1 gives
a11 T CPx + a12 T CPy + a13 T CPz + a14 = b11 T CPx + b12 T CPy + b13 T CPz + b14

⇔
8 CHAPTER 2. ROBOTICS

(a11 − b11 )T CPx + (a12 − b12 )T CPy + (a13 − b13 )T CPz = −(a14 − b14 ) (2.4)
For every equation 2.1 a system of three equations like equation 2.4 is retrieved.
The system of three equations can be rewritten as
 
h i T CPx h i
Ra − Rb  T CPy  = − ta − tb
 
(2.5)
T CPz

By using all distinct combinations i, j of the N robot positions, a system of

h iT
equations 2.5 is retrieved. T CPx T CPy T CPz is then easily calculated
as the linear least square solution of the system of equations.
As the rotation part of the transformation from the tool flange coordinate sys-
tem to the tool coordinate system is of no importance for determine the tool center
point, all information needed for determine the coordinate of the tool center point
is hereby achieved.
Chapter 3

Theory

This chapter will describe theories necessary for understanding the rest of the the-
sis. The chapter is divided in image processing theories, computer vision theories
and transformation theories.

3.1 Image Processing Theory

3.1.1 Threshold
The threshold operation is a grey level/one color channel image processing method
resulting in a binary image. Let the image be f (x, y), the resulting binary image
b(x, y) and the threshold value T then
(
1 if f (x, y) ≥ T
b(x, y) =
0 if f (x, y) < T

3.1.2 Morfological operations

Morfological operations are methods working on binary images. During this sec-
tion the following terminology will be used.
f (x, y) is the binary image.
r(x, y) is the resulting binary image.
a(x, y) is a structure element.
See [2] for more information.

3.1.3 Structure Element

A structure element is a binary image a(x, y) that defines the erosion or the di-
lation operations. The structure element is used as a convolution kernel in the

9
10 CHAPTER 3. THEORY

dilation operation and a cross correlation kernel in the erosion operation.

3.1.3.1 Erosion
The erosion operation is accomplished by setting all object pixels within a certain
distance from a background pixel to 0. In practice a structure element is used. The
origin of the structure element is then translated to every object pixel. If all of the
structure element is accommodated in the object at a certain position the pixel is
set to 1 in the resulting image, otherwise 0.

r(x, y) = a(x, y) f (x, y)

Where is the erosion operator.

The erosion operation can also be seen as a cross correlation between the struc-
ture element a(x, y) and the image f (x, y).
(
1 if a(x, y) ? f (x, y) = A
r(x, y) =
0 if a(x, y) ? f (x, y) 6= A

Where A is the number of pixels in the structure element.

Notice, a(x, y) f (x, y) 6= f (x, y) a(x, y) due to the fact that correlation
does not fulfill the commutative law

Figure 3.1: Original image before morfological operation.

Figure 3.2: Erosion operation applied to figure 3.1. A 8 × 8 structure element was
used.

3.1.3.2 Dilation
The dilation operation is the opposite of the erosion operation. It is accomplished
by setting all background pixels within a certain distance from an object point to
3.1. IMAGE PROCESSING THEORY 11

1. In practice a structure element is used. The origin of the structure element is

then translated to every object pixel. In every position a pixel wise OR operation
is carried out.

r(x, y) = a(x, y) ⊕ f (x, y)

Where ⊕ is the erosion operator.

The dilation operation can also be seen as a convolution between the structure
element a(x, y) and the image f (x, y).
(
1 if a(x, y) ∗ f (x, y) ≥ 1
r(x, y) =
0 else

Figure 3.3: Dilation operation applied to figure 3.1. A 4 × 4 structure element was
used.

3.1.3.3 Opening
The opening operation consists of one erosion operation followed by one dilation
operation, where the distance is the same in both of the operations. When im-
plemented with a structure element the same structure element is used in the two
operations.
The opening operation will split two objects which border on each other.

3.1.3.4 Closing
The closing operation consists of one dilation operation followed by one erosion
operation, where the distance is the same in both of the operations. When im-
plemented with a structure element the same structure element is used in the two
operations.
The closing operation will merge two objects which border on each other.
12 CHAPTER 3. THEORY

3.2 Computer Vision Theory

3.2.1 Camera System
The lens inside the camera refracts all rays of light from a certain object point to
one single point in the image plane. If the lens is thin implying the distortion can
be neglected, the lens law is valid.

1 1 1
+ = (3.1)
α β f

Where α is the distance between the lens and the object, β is the distance between
the lens and the image plane and f is the focal length. Figure 3.4 illustrates the
lens law.

Figure 3.4: Illustration of the lens law.

By the lens law it is obvious that an object at the distance α from the lens
will be reproduced with complete sharpness on the image plane. If the distance
between the object and the lens differs from α, the reproduction on the image
plane will be more or less blurred. How blurred the image will be, is determined
by the depth of field s. !
f
s = 2λ (3.2)
D
Where D is the diameter of the aperture and λ is the wavelength of the incoming
ray of light. The depth of field can also be defined as the interval where the
3.2. COMPUTER VISION THEORY 13

1 1
resolution is greater than 2d
where the resolution d
is defined as following

1 1D
= (3.3)
d λf

[3]

3.2.2 Pinhole Camera Model

One common way of modeling a camera is to use the pinhole camera model. The
model performs well as long as the lens is thin and no wide-angle lens is used.
In practise the image plane is located behind the lens, but to simplify calculations
and relations between the coordinate systems, the image plane can be put in front
of the lens. Figure 3.5 illustrates the pinhole camera model with the image plane
located in front of the lens.

Figure 3.5: Illustration of the pinhole camera model, image plane in front of the
lens to simplify calculations (Courtesy of Maria Magnusson Seger).

Equation 3.4 shows the relation between the coordinate systems.

X
 
   
un U h i Y 
W  vn  =  V  = R t  (3.4)
     
Z

 
1 W
1
h iT h iT
Where U V W are the camera coordinates, un vn are the ideal nor-
h iT
malized image coordinates and X Y Z are the world coordinates.
14 CHAPTER 3. THEORY

h i
Matrix R t is the extrinsic parameters and describes the translation and
the rotation between the coorinate systems, e.g. how the camera is rotateted and
translated in relation to the origin of the world coordinate system.
The real image plane usually differs from the ideal normalized image plane.
Equation 3.5 describes the relation between the two planes.
   
u un

 v  = A  vn 
  
(3.5)
1 1

Where A defines the intrinsic parameters.

 
α γ u0
A =  0 β v0 
 

0 0 1
" #
u0
Where α and β are scaling factors for the u and v axes. are the image
v0
coordinates of the intersection between the image plane and the optical axis. γ
describes the skewing between the u and v axes.
Observe, the relation in equation 3.5 implies equation 3.4 can be rewritten as

X
 
 
u h i Y 
W  v  = kA R t  (3.6)
   
 Z 

1
1
W
Where k is an arbitrary constant. Let s = k
implies equation 3.6 can be rewritten
as
X
 
 
u h i  Y 
s v  = A R t  (3.7)
  
Z
 
 
1
1

[3]

3.2.3 Camera Calibration

Equation 3.7 describes how a point in the environment maps to the image plane
up to a scale factor s.
Let: h i
C=A R t (3.8)
3.2. COMPUTER VISION THEORY 15

h iT h iT
If N corresponding points in the image ui vi and the world Xi Yi Zi
are found and i ∈ [1 . . . N ] then C can be determined to up a scale factor.
 
C11 C12 C13 C14
C =  C21 C22 C23 C24  (3.9)
 

C31 C32 C33 C34

By setting C34 = 1 in equation 3.9 the scale factor is determined.

Let:
h iT
c= C11 C12 C13 C14 C21 C22 C23 C24 C31 C32 C33

To determine c (determine the intrinsic and extrinsic parameters) a system of

equations given by the corresponding points can be used.

Dc = f (3.10)

0 0 −u1 X1 −u1 Y1 −u1 Z1

 
X1 Y1 Z1 1 0 0

 0 0 0 0 X1 Y1 Z1 1 −v1 X1 −v1 Y1 −v1 Z1 

D= .. .. .. .. .. .. .. .. .. .. .. 
. . . . . . . . . . .
 
 
0 0 0 0 XN YN ZN 1 −vN XN −vN YN −vN ZN
 
u1

 v1 

f = .. 
.
 
 
vN

c is then given by
 −1
c = DT D D T f = D† f (3.11)

Note, at least six corresponding points are needed to determine the intrinsic
and extrinsic parameters.

[3]
16 CHAPTER 3. THEORY

3.3 Transformation Theory

3.3.1 Homogeneous Transformation
3.3.1.1 3D translation
h iT 0
h iT
Let point P = x y z be translated to a point P = x0 y 0 z 0 by the
h iT 0
translation vector t = tx ty tz . Then P can be expressed as
0
P =P+t (3.12)

3.3.1.2 3D Rotation
h iT 0
Let point P = x y z be rotated θ◦ around the Z axis to a point P =
h iT 0
x0 y 0 z 0 . Point P can be expressed according to [4] as

x0 = x cos θ − y sin θ (3.13)

x0 = x sin θ + y cos θ (3.14)
z0 = z (3.15)

Let:  
cos θ − sin θ 0
Rz =  sin θ − cos θ 0 
 

0 0 1
0
Then P can be written as
0
P = Rz P (3.16)

3.3.1.3 Homogeneous Transformation Matrix

Translation and multiplicative terms for a three dimensional transformation can
be combined to a single matrix. Expand the three dimensional coordinates P and
P 0 in section 3.3.1.1 and section 3.3.1.2 to four element column vectors as
h iT
P
e
= xh yh zh h
f0 h iT
P = x0h yh0 zh0 h

Where h is the nonzero homogeneous parameter.

x = xhh , y = yhh , z = zhh

x0 y0 z0
x0 = hh , y 0 = hh , z 0 = hh
3.3. TRANSFORMATION THEORY 17

For a geometric transformation the homogeneous parameter h can be set to any

nonzero value. Suitably h is set to 1 implying e = eh where e ∈ {x,y,z}.
The translation in equation 3.12 can be represented by a homogeneous matrix
M as
f0
P = MP e
(3.17)
1 0 0 tx
 
 0 1 0 ty 
M=
 
0 0 1 tz

 
0 0 0 1
The rotation in equation 3.16 can be represented by the homogeneous matrix
M if
cos θ − sin θ 0 0
 
 sin θ cos θ 0 0 
M=
 
 0 0 1 0 


0 0 0 1
A complete transformation of a rigid body e.g. a transformation between two
different coordinate systems can be expressed as a homogeneous transformation
matrix T.
r11 r12 r13 tx
 
 r
21 r22 r23 ty 

T=
 r31 r32 r33 tz 
 

0 0 0 1

3.3.2 Screw Axis Rotation representation

 
a11 a12 a13
A rotation matrix R =  a21 a22 a23  is completely defined by the axis of
 

a31 a32 a33

h iT
rotation nx ny nz and the rotation angle θ as

a11 = (n2x − 1)(1 − cos θ) + 1 (3.18)

a12 = nx ny (1 − cos θ) − nz sin θ (3.19)
a13 = nx nz (1 − cos θ) + ny sin θ (3.20)
a21 = ny nx (1 − cos θ) + nz sin θ (3.21)
a22 = (n2y − 1)(1 − cos θ) + 1 (3.22)
a23 = ny nz (1 − cos θ) − nx sin θ (3.23)
a31 = nz nx (1 − cos θ) − ny sin θ (3.24)
a32 = nz ny (1 − cos θ) + nx sin θ (3.25)
a33 = (n2x − 1)(1 − cos θ) + 1 (3.26)
18 CHAPTER 3. THEORY

Equation 3.26 is called the screw axis representation of the orientation. [5]
h iT
Given a rotation matrix R, the angle of rotation θ and the rotation axis nx ny nz
can be obtained by the following equations.
a11 + a22 + a22 − 1
θ = cos−1 (3.27)
2
a32 − a23
nx = (3.28)
2 sin θ
a13 − a31
ny = (3.29)
2 sin θ
a21 − a12
nz = (3.30)
2 sin θ
This representation makes it possible to compare two different measuring units
which uses different coordinate systems to measure the rotation. In this project
the method was used to ensure the camera measured the same rotation angle as
the robot, when the robot was told to perform a rotation.

3.3.3 Rodrigues´s formula

One way of finding the rotation matrix R in section 3.3.2 is to make use of the
Rodrigues’s formula for a spherical displacement of a rigid body.
Let point P1 rotate around the rotation axis n, resulting in a new position P2
of the point. See figure 3.6. The Rodrigues’s formula is then defined according to
equation 3.31

Figure 3.6: Definition for Rodrigues’s formula.

r2 = r1 cos θ + n × r1 sin θ + n(rT

1 n)(1 − cos θ) (3.31)
3.3. TRANSFORMATION THEORY 19

According to [5] equation 3.31 can be rewritten as equation 3.32

r 2 = 1 R2 r 1 (3.32)

Where 1 R2 is the rotation matrix fulfilling the equations 3.18 - 3.26

Chapter 4

Determination of Tool Center Point

This chapter describes how the problem of finding the tool center point was solved.
The chapter starts by describing the proposed method for determination of the
camera to tool, tool to robot wrist and camera to robot base coordinate system
transformations respectively. The chapter also describes the proposed iterative
method for finding the tool center point. The iterative method is used to remove
uncertainties due to camera calibration errors and robot calibration errors.

4.1 Overview of the system

Figure 4.1: Overview of the system.

Figure 4.1 shows the arrangement of the system. A computer is connected to
a camera by a USB cable and to the robot by a TCP-IP cable.

21
22 CHAPTER 4. DETERMINATION OF TOOL CENTER POINT

4.1.1 Equipment
The robot used during the project was an IRB 1400 ABB robot controlled by an
IRC5 computer. A laptop computer with an Intel PentiumM 1600Mhz processor
and 1Gb RAM was connected to the IRC5 computer by a TCP-IP cable. A pro-
gram was then developed in C# and C/C++ making it possible to control the robot
movements directly from the laptop. The laptop computer was also used for im-
age segmentation of the tool. The images were retrieved at 50 frames per second
by an uEye (UI221x-M V2.10) CCD grayscale USB camera with a resolution of
640 × 480 pixels.
Except for the equipment, four different transformations T1 , T2 , X1 and X2
are displayed in figure 4.1.
By defining a tool coordinate system TF (tool frame) with its origin at the tool
center point, the problem of finding TCP actually becomes equivalent by finding
the transformation X1 from the tool frame TF to the TFF (Tool Flange Frame)
coordinate system fixed at the last axis of the robot manipulator.
To make it possible to determine transformation X1 , a closed loop of homo-
geneous transformation matrixes can be written as an equation.

T2 X1 = X2 T1 (4.1)
Transformation matrix T1 in Equation 4.1 is the robot’s forward kinematics.
Consequently T1 is known, making the problem of finding X1 become equivalent
by determine X2 and T2

4.2 Determine T2
By calibrating the camera using the method proposed by Zhengyou Zhang [6]
the intrinsic parameters of the camera were determined and the lens distortion
coefficients were retrieved, see Appendix A.
h iT h iT
Let m = u v denote a 2D image point and M = X Y Z de-
note a 3D point. The augmented representations of these vectors are achieved
h iT
by adding 1 as the last element of the vector. mf = u v 1 and Mf
=
h iT
X Y Z 1 . By modeling the camera as a perfect pinhole camera, the re-
lation between a 3D point M and its projected image point m is fulfilling equation
4.2
smf = ATM f
(4.2)
Where A is the intrinsic parameters, s is a scale factor and T is the extrinsic
parameters. h i
T = r1 r2 r3 t
4.2. DETERMINE T2 23

By setting up a plane in the world, the homography between the plane and its im-
age can be determined. Let the plane fulfilling the constraint of Z = 0. Equation
4.2 is then rewritten as
X
 
   
u h i Y  h i X
 v  = A r1 r2 r3 t 
s  = A r1 r2 t  Y 
    
 0 
1 1
1
h i h i
Let the homography H = h1 h2 h3 = A r1 r2 t . According to [6]
this implies the extrinsic parameters can be determined as
r1 = λA−1 h1
r2 = λA−1 h2
r3 = r1 × r1
t = λA−1 h3
Where λ = || A−11 h || = || A−11 h ||
1 2

By letting the plane be expressed in the tool coordinate system TF, finding trans-
formation
h T2 in figurei4.1 becomes equivalent of finding the extrinsic parameters
T = r1 r2 r3 t . From 4.2 it is obvious that T is totally determined by the
intrinsic parameters A and the homograpghy H. The camera calibration toolbox
for Matlab was used to determine the intrinsic parameters, A, for the camera. See
Appendix A. This toolbox uses theh same technique i as mentioned in this section
and in [6]. The homography H = h1 h2 h3 can be obtained according to
[6] by minimizing
  T   
T T
mi − 1  h1 Mi  (σ 2 I)−1 mi − 1  h1 Mi 
P
i T T i T T
h3 Mi h2 M i h3 Mi h2 M i
Where hj is the j:th row of H, I is the identity matrix and σ is the standard
deviation of the Gaussian noise that is assumed to affect mi . The problem can be
solved as a non-linear least square problem.
  2
T
 h1 Mi 
 
P  1

minH i mi − T T
 h3 Mi h2 M i



Due to the fact that a plane is by definition defined of at least three points, at
least three corresponding points mi and Mi belonging to the plane needs to be
determined. An image segmentation method for finding three distinct points in
the plane lying in the tool coordinate system TF, is described in chapter 5. The
tool used for the segmentation was a calibration tool with an awl looking shape.
The TCP of the tool was the tip of the awl, and the TCP was one of the points
determined by the method.
24 CHAPTER 4. DETERMINATION OF TOOL CENTER POINT

4.3 Finding X1 and X2 transformation matrixes

When the transformations T1 and T2 were found only X2 remained to be de-
termined. Jintao and Daokui proposed a method in [7] where the transformation
X2 was determined by moving the robot to a fixed position e.g. the tip of a nail
with known relation to a calibration pattern. Zhuang et al. [8] proposed a method
that solved X1 and X2 simultaneously by applying quaternion algebra to derive
a linear solution. Fadi Donaika and Radu Horaud [9] proposed one closed form
method and one method based on non-linear constrained minimization for solving
the equation 4.1 of homogeneous matrixes.
The method implemented was first described by Roger Y. Tsai and Reimar K.
Lenz [10]. By moving the robot to a second position finding T1 and T2 at the
new location of the robot, a system of two equations 4.3 and 4.4 can be achieved,
please see figure 4.2

T21 X1 = X2 T11 (4.3)

T22 X1 = X2 T12 (4.4)

By multiply the inverse of equation 4.4 by equation 4.3 equation 4.6 will be
obtained.

(T22 X1 )−1 T21 X1 = (X2 T12 )−1 X2 T11 (4.5)

⇔
X−1 −1
1 T22 T21 X1= T−1 −1
12 X2 X2 T11
⇔
T−1 −1
22 21 1 = X1 T12 T11
T X
⇔

AX1 = X1 B (4.6)

Where A = T−1 −1
22 T21 and B = T12 T11 are known. Observe, A and B are the
transformations from position 1 to position 2 of the tool flange frame and the tool
frame respectively. See figure 4.2
The transformation matrixes A,B,X1 are all homogeneous matrixes. A ho-
mogeneous matrix consists of one rotation matrix R and one translation vector
t.
4.3. FINDING X1 AND X2 TRANSFORMATION MATRIXES 25

Figure 4.2: Transformations AX1 = X1 B. RB is the robot base frame, CF is the

camera frame, TF is the tool frame and TFF is the tool flange frame. A and B are
the transformations from position 1 to position 2 of the tool flange frame and the
tool frame respectively.

!
RX1 tX1
X1 =
0 0 0 1
!
RA tA
A=
0 0 0 1
!
RB tB
B=
0 0 0 1

The rotation matrix R can be written as

T
 
n21 + (1 − n21 ) cos θ n1 n2 (1 − cos θ) − n3 sin θ n1 n3 (1 − cos θ) + n2 sin θ 
n22 + (1 − n22 ) cos θ

R =  n1 n2 (1 − cos θ) + n3 sin θ
 n2 n3 (1 − cos θ) − n1 sin θ 

n1 n3 (1 − cos θ) + n2 sin θ n2 n3 (1 − cos θ) + n1 sin θ n23 + (1 − n23 ) cos θ
h i (4.7)
Where n1 n2 n3 is the rotation axis and θ is the rotation angle. Obvi-
h i
ously it is possible to specify R in equation 4.7 by specifying n1 n2 n3 and
θ.
26 CHAPTER 4. DETERMINATION OF TOOL CENTER POINT

By using a modified
h version of Rodrigues
i formula, see section 3.3.3, a func-
tion Pr depending on n1 n2 n3 and θ can be defined as

θh i
Pr = 2 sin n1 n2 n3 , 0≤θ≤π (4.8)
2

Let:
PA , PB , PX1 be the rotation axes defined according to equation 4.8
for RA , RB , RX1 respectively.

0 1
PX1 = q (4.9)
4 − |PX1 |2

For a vector v = [vx vy vz ] let

 
0 −vz vy
Skew(v) =  vz

0 −vx 
 (4.10)
−vy vx 0

0
By setting up a system of linear equations according to equation 4.11, PX1
can be solved by using linear least square techniques.

0
Skew(PB + PA )PX1 = PA − PB (4.11)

Proof:
0
PA − PB ⊥ PX1 (4.12)

PA − PB ⊥ PA − PB (4.13)

By equation 4.12 and 4.13 follows

0
PA − PB = s(PA + PB ) × PX1 (4.14)

Where s is a constant
0
PA − PB and (PA + PB ) × PX1 has got the same length implying s = 1.
4.3. FINDING X1 AND X2 TRANSFORMATION MATRIXES 27

0
Let α be the angle between PX1 and (PA + PB ).
0 0
|(PA + PB ) × PX1 | = |PA + PB ||PX1 | sin α =

, ,
= Use equation 4.9 =

1
|PA + PB |2 sin 2θ (4 − 4 sin2 2θ )− 2 sin α =

|PA + PB | tan 2θ sin α =

|PA − PB |

This implies equation 4.14 can be written as

0
PA − PB = (PA + PB ) × PX1 (4.15)

By equation 4.15 and the relation a × b = Skew(a)b follows

0
Skew(PB + PA )PX1 = PA − PB

Q.E.D

Skew(PB + PA ) is always singular and therefor at least two pairs of positions

are needed to create the system of equation 4.11 e.g. three different positions for
0
the robot are needed. When PX1 is determined PX1 can be retrieved by equation
4.16. The rotation RX1 is then determined by equation 4.17
0
2PX1
PX1 = q 0
(4.16)
1 + |PX1 |2

|PX1 |2 1 T
q
RX1 = (1 − )I + (PX1 PX1 + 4 − |PX1 |2 Skew(PX1 )) (4.17)
2 2
Let:

1 0 0 |
 
 0 1 0 tX1 
TX1 =  (4.18)
 
0 0 1 |

 
1 0 0 1
28 CHAPTER 4. DETERMINATION OF TOOL CENTER POINT

1 0 0 |
 
 0 1 0 tA 
TA =  (4.19)
 
0 0 1 | 


1 0 0 1
1 0 0 |
 
 0 1 0 tB 
TB =  
(4.20)
0 0 1 |
 
 
1 0 0 1

To find the translation tX1 of the homogeneous transformation X1 , a linear

system of equations 4.21 can be solved by a linear least square technic.

(RB − I)TX1 = RX1 TA − TB (4.21)

Proof:
AX1 = X1 B
⇔
(TB + RB − I)(TX1 + RX1 − I) = (TX1 + RX1 − I)(TA + RA − I)
⇔
T B T X 1 + T B RX1 − T B + RB T X1 + RB RX1 − RB − T X1 − RX1 + I =
−RX1 − TA + RX1 TA + RX1 RA + TX1 + TA − I
⇔
T B + T X1 − I + T B + RX1 − I − T B + RB T X1 + RB RX1 − RB − T X1 − RX1 + I =
−RX1 + RX1 TA + RX1 RA + TX1 − I
⇔
TB − I + RB TX1 + RB RX1 − RB = −RX1 + RX1 TA + RX1 RA + TX1 − I
⇔
RB TX1 + RB RX1 − RB − TX1 =−RX1 + RX1 TA + RX1 RA − TB
⇔
, ,
Remove all terms not having a translation

⇔
RB T X1 − T X 1 = RX1 T A − T B
⇔
(RB − I)TX1 = RX1 TA − TB

Q.E.D.

At least three different positions of the robot are needed to calculate the equa-
tions 4.11 and 4.21 by a linear least square technique. In the implementation, four
4.4. IMPORTANCE OF MEASURE A AND B WITH 6 DOF 29

different positions were used. The result varied much due to how the positions
were chosen. The best estimation of X1 was achieved when the positions were
fulfilling the constraint of completely exciting all degrees of freedom. In practise
this constraint is equivalent of ensuring the three first positions differ in both a lin-
ear movement along orthogonally axes x, y, z and rotations around each axis. The
h iT h iT
fourth position was chosen to be a movement in x y z = 1 1 1
and a rotation around the same axis.

4.4 Importance of measure A and B with 6 DOF

Actually, if only five degrees of freedom can be measured in T1 or T2 , no unique

solution X1 exists to the equation AX1 = X1 B. This situation occurs for in-
stance if the tool has an axis of symmetry.

Proof:
First four lemmas have to be defined.

Lemma1 :
Matrixes A and B are similar if a matrix X exists such as
B = X−1 AX
⇔
AX = XB
If A and B are similar then A and B has got the same
trace, determinant and eigenvalues.[11]

Lemma2 :
The eigenvalues of a rotation matrix R are
λ1 = 1
λ2 = cos θ + i sin θ
λ3 = cos θ − i sin θ
Where θ is the angle of rotation specified by equation 4.7
[12]
30 CHAPTER 4. DETERMINATION OF TOOL CENTER POINT

Lemma3 :
A rotation matrix R fulfills according to [13] the orthogonality conditions
R−1 = RT
and
RT R = I

Lemma4 :
For a matrix R fulfilling the ortogonality conditions,
the trace of R is the sum of the eigenvalues of R according to [12].
P
T r(R) = i λi

Lemma 2 and 3 in Lemma 4 gives

T r(R) = 1 + cos θ + i sin θ + cos θ − i sin θ

⇔
!
T r(R) − 1
θ = arccos (4.22)
2
If the robot is told to perform a rotation θ around the axis of symmetry, the robot
will measure a rotation θB = θ. However the camera will not be able to distinguish
any difference between the orientation before and after the rotation around the axis
of symmetry. This implies the camera will measure a rotation θA = 0.
It is then obvious by equation 4.22 that in general T r(A) 6= T r(B) implying
according to Lemma1, there is no solution X1 to equation AX1 = X1 B.

Q.E.D

Observe, once transformation X1 is determined, the transformation X2 be-

tween the camera coordinate system and the robot base frame can be obtained as
X2 = T2 X1 T−1 1 .
The calibration tool described in chapter 5 had an axis of symmetry making
it impossible to find a solution X1 to equation AX1 = X1 B. Instead, the non
symmetric chess board in figure 4.3 was used as a robot tool. The TCP was defined
as the upper left corner of the chess pattern.
Even though it is impossible to find a solution X1 to equation AX1 = X1 B
if the robot tool has an axis of symmetry, it is still possible to determine X1 by
using another approach, see section 4.6.
4.5. ITERATIVE METHOD FOR INCREASING THE ACCURACY 31

Figure 4.3: Chessboard pattern.

4.5 Iterative Method for increasing the accuracy

Although the transformation X1 is known and the coordinate of the tool center
point is found, the accuracy of the method is under influence of errors in the
camera calibration and errors in the robot calibration. To ensure a high accuracy
an iterative method had to be used. Denote the tool center point coordinate found
by the method as TCPguess and the correct coordinate of TCP as TCPcorrect .
Assume the robot reorient around TCPguess . If TCPguess = TCPcorrect
the TCPcorrect would stay at the same point TCPguess during the reorientation,
while TCPcorrect would move away from TCPguess if TCPguess 6= TCPcorrect .
This phenomena can be used to achieve a great accuracy of the tool center point
calibration.
Denote the error vector between TCPguess and TCPcorrect after a reorienta-
tion around TCPguess as . By measure the coordinate of TCPcorrect after the
reorientation around TCPguess ,  can be retrieved as

 = TCPcorrect − TCPguess (4.23)

Of course the new measurement of TCPcorrect is only a guess, retrieved in the

same way as TCPguess and with the same accuracy. This implies e.g.

 = TCPguess2 − TCPguess (4.24)

The robot can then be told to perform a linear movement of −. Ideally this would
32 CHAPTER 4. DETERMINATION OF TOOL CENTER POINT

imply TCPguess = TCPcorrect but in most cases TCPcorrect 6= TCPguess2 im-

plying TCPguess 6= TCPcorrect . Instead TCPcorrect is measured again by the
camera and a new  is calculated. This procedure is done iteratively until || < β,
where β is the desired accuracy. Observe, to retrieve a faster convergence of the
iterative algorithm the robot can be told to move −α, where α is a constant < 1.
When the desired accuracy β is retrieved, two positions of the tool are known
where TCPcorrect = TCPguess . The orientation of the tool differs between the
two positions, because only linear movements are used when moving the robot
−.
By performing reorientations around TCPguess in six different directions and
using the iterative procedure to linearly move TCPcorrect back to TCPguess , six
different positions are obtained. All of the six robot positions ensure TCPcorrect
is at the same certain point TCPguess , but with six different orientations.
Let T1i , i ∈ [1...6] be the homogeneous transformations from the robot base
frame coordinate system to the tool flange coordinate system e.g.
 the robot’s for-
T CPx
ward kinematics for the six different positions. Let  T CPy  be the translation
 

T CPz
from the origin of the tool flange coordinate system to the tool center point.
Let:

1 0 0 T CPx
 
 0 1 0 T CPy 
Q=
 
0 0 1 T CPz 


0 0 0 1

Due to the fact that the tool center point is at the same coordinate independent
of which of the six robot positions that is examined, the algorithm described in
section 2.3 on page 7 can be used to determine the true position of the tool center
point. By using the iterative procedure and calculating the tool center point as a
least square solution described in section 2.3, the accuracy of the TCP calibra-
tion method becomes independent of the camera calibration errors and the robot
calibration errors. Instead, only the accuracy of the camera when registering a
movement between two different positions, affects the final accuracy of the tool
center point calibration achieved by the iterative method. This can be assured by
considering that the camera is an external observer during the iterative procedure,
and the method will iterate until the camera measures (with the desired accuracy)
the true tool center point is at the same location as the initial guess point.
4.6. ALTERNATIVE METHOD FOR ROTATION SYMMETRIC TOOLS 33

4.6 Alternative method for rotation symmetric tools

If the transformation between the camera and the robot’s base frame, X2 , is
known, the iterative method for increasing the accuracy will work even if the
tool is symmetric.
During the iterative method, the robot performs a reorientation and the cam-
era only needs to measure the translation error between the positions before and
after the reorientation. This assures the transformation, T2 between the camera
frame and the tool frame only needs to be determined with five degrees of freedom
during the iterative procedure.
Although, the direction of the movement - must be expressed in relation to the
robot’s base frame. Therefore the rotation part of the transformation, X2 between
the camera frame and the robot base frame needs to be determined. The rotation
part, RX2 , of the homogeneous transformation X2 can be obtained according to
[4] by moving the robot a certain distance in the X,Y and Z directions respectively
in the robot’shbase frame. iT h iT h iT
Let u = ux uy uz , v = vx vy vz and w = wx wy wz
be the vectors measured by the camera when the robot moves a specified distance
in the X, Y and Z directions respectively. According to [4] the rotation part RX2 ,
of the homogeneous transformation X2 is then obtained as
 
ux vx wx
RX2 =  uy vy wy 

 (4.25)
uz vz wz

The translational scaling factor between the two coordinate systems is also possi-
ble to determine, because the distances the robot moves in the X, Y and Z direc-
tions are known and the distances of the same movements measured by the camera
are also retrieved. This assures it is possible to find the translation error  between
TCPguess and the true coordinate of the tool center point TCPcorrect , even if the
robot tool is rotation symmetric. It is thereby possible to perform a tool center
point calibration, even if the robot tool is rotation symmetric.
Chapter 5

Image Segmentation of Robot Tool

This chapter will describe the methods evaluated during this project for image
segmentation of the robot tool.

5.1 Overview
Since one of the goals of this thesis was to deliver a software program made in
C/C++ and C# , all of the image processing had to be written in either of these
program languages. The HMI part of the program was written in C# and the image
processing functions were written in C/C++.
To keep the production costs as low as possible only open source libraries were
allowed to be used during this project. Due to this constraint the image processing
functions in Open Computer Vision Library 1.0 (OpenCV 1.0) were mainly used
for image segmentation.

5.1.1 Open Computer Vision Library (OpenCV)

OpenCV 1.0 is an open source project written in C consisting of a large collec-
tion of image processing algorithms. The library is also compatible with Intel’s
IPL image package and has got the ability to utilize Intel Integrated Performance
Primitives for better performance. [14] [15]

5.2 Creation of binary mask image

To retrieve a rough approximation of the image region where the object is located
numerous approaches can be used. In this section two different methods will be
evaluated.

35
36 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

5.2.1 Image registration

The
" idea
# of image registration is to find the transfer field v(x) : R2 → R2 , x =
x
making a reference image I2 (x, y) fit as good as possible in a target image
y
I1 (x, y) e.g. finding v(x) minimizing  in equation 5.1

2 = ||I2 (x + v(x)) − I1 (x)|| (5.1)

The method outlined in this section is described in more detail in [16]. By

letting the reference image be a picture of the desired tool and the target image be
the image achieved by the camera, this method should iteratively be able to find
the location of the tool in the target image. One constraint of the target image have
to be fulfilled.

ConstraintI :
The image can locally be described as a sloping plane.
I(x, t) = I(x, t) − ∇I T (x, t)v(x) + ∆t
∆t = I(x,"t + 1) − I(x, t) #
∇x I(x, t + 1)
∇I(x, t) =
∇y I(x, t + 1)

ConstraintI can be rewritten as

I(x, t) = I(x, t) − ∇I T (x, t)v(x) + ∆t

I(x, t) = I(x, t) − ∇I T (x, t)v(x) + (I(x, t + 1) − I(x, t))

I(x, t) = −∇I T (x, t)v(x) + I(x, t + 1)

∇I T (x, t)v(x) = I(x, t + 1) − I(x, t) (5.2)

5.2. CREATION OF BINARY MASK IMAGE 37

Let:
I1 = Ix,t+1

I2 = Ix,t

v(x) = B(x)p
" #
1 0 x y 0 0
B(x) =
0 1 0 0 x y
 
p1
 . 
p =  .. 


p6
Then 5.2 can be written as

∇I2T B(x)p = I1 − I2 (5.3)

Let:
∇I2 (x1 )T B(x1 )
 

A= .. 
 . 

∇I2 (xn )T B(xn )
 
I1 (x1 ) − I2 (x1 )

b= .. 

. 

I1 (xn ) − I2 (xn )
Where n is the number of pixels in I2 . Equation 5.3 is then rewritten to equation
5.4.
Ap = b (5.4)
This implies p and v(x) are determined as

p = (AT A)−1 AT b = A† b (5.5)

v(x) = B(x)p (5.6)

The new location I2new of the reference image is then obtained by interpolating
I2 from v(x). The method is then iterated with I2new as I2 until |v(x)| < β, where
β is the desired accuracy.
The method described in this section was implemented and evaluated in Mat-
lab. Unfortunately the method was not reliable. The resulting position of the
38 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

reference image I2 was heavily affected by the light condition. Although, when
using the method on synthetic binary test images the method performed very well.
Of course it would be possible to threshold the image I1 retrieving a binary
image Ibinary , and then apply the image registration method on the binary image.
Although, finding the threshold value that separates the background completely
from the tool is a hard problem to solve.
Of course there might exist image registration methods, better suited for solv-
ing the problem. A phase based image registration method would perhaps give
a better result than the method described here. However, due to the fact that it
is possible to move the robot to different positions, the background subtraction
method was instead evaluated.

5.2.2 Background subtraction

If several image frames can be acquired and the object is moving between each
frame the background subtraction method can be used to retrieve a smaller search
area for the object.
The background subtraction is done by subtracting two images from each other
and perform a threshold. If the scene is completely static except for the moving
object, all of the background will be eliminated resulting in a binary mask image
where background pixels are set to 0 and object pixels are set to 1.
Let
• f1 (x, y) be the image acquired at the start position.
• f2 (x, y) be the image acquired at the final position.
• b(x, y) be the binary result image.
• T be threshold value.
(
1 , kf1 (x, y) − f2 (x, y)k ≥ T
b(x, y) =
0 , kf1 (x, y) − f2 (x, y)k < T

The resulting binary mask image will of course have its pixels set to one in
both the start location and the final location for the object. To distinguish between
the object’s true location and the last location, the object can be moved to a third
position and a new image f3 can be retrieved.
By applying the background subtraction method to f1 − f3 and f2 − f3 sep-
arately and applying a pixel wise logical AN D operator to the resulting binary
images, a new binary mask image only giving the last location of the object will
be retrieved.
In practise the resulting binary mask image was not perfect, several holes oc-
curred in the object region see figure 5.1.
5.2. CREATION OF BINARY MASK IMAGE 39

Figure 5.1: Binary mask image from background subtraction.

To fill the holes in the mask image, morphological operations were applied to
the binary image. By gradually apply dilate and erode operators (cvDilate and
cvErode in OpenCV) on the mask image the holes were filled.
However, due to the quadrangular structure element used by the morpholog-
ical operators the tip structure of the object mask image was deteriorated. Later
on during the segmentation process the damaged tip structure showed to be an
obvious drawback.
Instead of using the morphological operators, a new method referred to as the
Hough Filling Method (HFM) was invented.

5.2.2.1 The Hough filling method (HFM)

By applying the Progressive Probabilistic Hough Transform (PPHT) described in
[17] to the binary mask image in figure 5.1, all probabilistic lines were found. If
the permitted gap constraint was set to a suitable value and the minimum distance
between pixels which were to be considered as a line was set to zero, all holes were
perfectly filled when the lines found by PPHT were drawn on the binary mask
image. The HFM method will not increase the mask image boundary as long as
the curvature of the boundary remains low. The PPHT algorithm is implemented
in the OpenCV function cvHoughLines2 [18].
By applying the background subtraction method and filling gaps and holes
with the Hough filling method, almost perfect binary mask images were achieved
for all three locations of the tool, see figure 5.2. After the binary mask was used
only a small search area was left in the image.

Figure 5.2: Binary mask image from background subtraction after applying the
HFM.
40 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

5.3 Edge Detection

An edge can be treated as a locally odd function in an image.

f (−x, −y) = −f (x, y)

And such a function will always have a gradient. The easiest way of finding the
gradient is probably to convolve the image with the sobel operators.

   
1 0 −1 −1 2 −1
Gx =  2 0 −2  , Gy =  0 0 0 
   

1 0 −1 1 2 1
q
The edge image may then be retrieved as fedge = (f ∗ Gx )2 + (f ∗ Gy )2

Unfortunately this method won’t, according to [2], work properly in noise af-
fected environments. Irregular light conditions will also decrease the functionality
of this method.

5.3.1 Canny’s Edge Detection

1986 J.Canny presented a more robust technique in [19] for edge detection. In
contrast to the previous method, Canny´s edge detection algorithm makes use of
the second derivates, which are zero when the first derivates reach their maxima.
Let n = √ 2 ∇f (x,y)2 be the unit vector in the gradient direction. Where fi (x, y)
fx (x,y)+fy (x,y)
is the image f (x, y) derived in the i direction.
The edge image fcanny can be found by deriving the absolute value of the gradient
image in the direction of the gradient.
2 2
fcanny = n · ∇k∇f (x, y)k = fx (x,y)fxx (x,y)+2fx (x,y)f y (x,y)fxy (x,y)+fy (x,y) fyy (x,y)
fx2 (x,y)+fy2 (x,y)
To be able to find the contour of the tool, the Canny edge detection algorithm
[19] was used on the masked image
(
f (x, y) , if b(x, y) = 1
f (x, y)masked = (5.7)
0 , if b(x, y) = 0

Where b(x, y) is the resulting image given from the background subtraction and
HFM methods. To ensure that all of the tool was left in the masked image fmasked ,
the dilation operator was applied to b(x, y) before calculating fmasked according
to equation 5.7.
5.4. CONTOUR RETRIEVING 41

After applying the Canny edge detection algorithm on the masked image fmasked ,
the binary edge image bedge was obtained. The Canny algorithm is implemented
in the OpenCV library function cvCanny [18].
The threshold values in the cvCanny function had to be determined according
to the current environment. If the image contained a lot of noise, the threshold had
to be set to a high value to ensure the elimination of the noise. Unfortunately the
high threshold sometimes resulted in discontinuities of the tool edges in the edge
image bedge . To solve the problem of discontinuities of the tool edges the Hough
Filling Method (see section 5.2.2.1 on page 39) was used with an excellent result.
Another disadvantage was the presence of a light reflection in the metallic surface
of the tool. The contour of the reflection had the same shape as the actual tool,
resulting in a possible false hit later in the segmentation procedure. To avoid the
influence of light reflections the tool was painted in a non-reflecting colour.

5.4 Contour Retrieving

To retrieve the contours from the binary edge image, bedge , two different meth-
ods were evaluated. This section will illuminate the two methods, but first two
different contour representations will be described.
Observe, it would be possible to retrieve an approximation of the tool contour
from the binary mask image, see figure 5.2. In practise the result became more
accurate by retrieving the tool contour from the binary edge image bedge given by
the Canny edge detection algorithm.

5.4.1 Freeman Chain Code Contour representation

The freeman chain code representation is a compact representation of contours.
By denoting the neighbours of the current pixel with different digits, see figure
5.3, the contour can be stored as a succession of digits. Each digit then contains
all necessary information for finding the next pixel in the contour.[18]
Please, look at the example in figure 5.4

Figure 5.3: The Freeman representation.

42 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

Figure 5.4: An example of the Freeman Chain Code.

5.4.2 Polygon Representation

The polygonal representation of a contour is according to [18] a more suitable
representation for contour manipulations. In the polygonal representation the se-
quence of points are stored as vertices e.g. the pixel coordinates are stored.

5.4.3 Freeman Methods

The four scan line methods described in [20] are implemented in the OpenCV
function cvFindContours [18]. The scan line methods scan the image line by line
until it finds an edge. When an edge is found the method starts a border following
procedure until the current border is retrieved in the Freeman representation.
The first method described in [20] only finds the outer most contours, while the
other methods discover contours on several levels. The methods were evaluated
on the binary edge image with satisfactory results.
The methods with the ability to find contours on different levels had a disad-
vantage because unnecessary contours were found, increasing the risk of mixing
the true object with a light reflection of the same shape.

5.4.4 Active Contour (The Snake algorithm)

Active contours or Snakes are model-based methods for segmentation. The active
contour is a spline function v(s) with several nodes.
v(s) = (x(s), y(s)) (5.8)
Where x(s), y(s) are the x, y coordinates along the contour. s ∈ [0, 1]
Each node has got an internal energy and the algorithm is trying to minimize
the total energy, E, of the spline function.
Z 1
E= Esnake (v(s))ds (5.9)
0
5.4. CONTOUR RETRIEVING 43

The spline function is affected by internal forces Eint (v(s)), external forces
Eimg (v(s)) and Econ (v(s)).

Esnake = Eint (v(s)) + Eimg (v(s)) + Econ (v(s))

The internal forces affecting the active contour are divided in tension forces
and rigidity forces. The tension forces imply spring behaviour of the spline func-
tion, while the rigidity forces make the spline function resist bending.
 dv(s) 2  d2 v(s) 2
   
Eint (v(s)) = α(s) 

 + β(s) 
  
 ds2 

 ds 

Where α(s) specifies the elasticity and β(s) specifies the rigidity of the spline
function.
The external forces consist of image forces Eimg and user specified constraint
forces Econ . The image force is an image, where each pixel value defines a force.
The constraint forces can be used to guarantee that the active contour is not getting
stuck at a local minima. The constraint forces might be set by a higher level
process.[21]
One can compare the snake algorithm to a rubber band expanded to its maxi-
mum when the algorithm is initialized. The rubber band then iteratively contract
until the energy function in each node has reached equilibrium.
By using the binary edge image as the external force and defining the initial
contour to be the result of applying the Freeman scan line method to the binary
mask image, the active contour will successively contract until it perfectly en-
closes the object, see figure 5.5

Figure 5.5: The snake contour at initial state to the left and at the final state to the
right. The contour is randomly color coded from the start point to the end point of
the contour.

In fact the mask image and the initial active contour have to be larger than
the object itself to be able to enclose the object. If there are structures or patterns
44 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

behind the object, the active contour might get stuck due to forces in the edge
image belonging to these structures see figure 5.6. This drawback together with
the fact that the algorithm is iterative and therefore time consuming made the
Freeman method better suited for retrieving the contour.

Figure 5.6: The snake contour could get stuck at the background pattern. The
contour is randomly color coded from the start point to end point of the contour.

The snake algorithm is implemented in the OpenCV function cvSnakeImage.

[18]

5.5 Finding the contour matching the tool shape

When all contours were retrieved by the Freeman method, a process of logics had
to be applied for actually finding a contour or a part of a contour matching the
wanted tool object. Two different methods were implemented and evaluated.

5.5.1 Convex Hull and its defects

A widely used methodology for finding objects with a shape reminding of a hand
or a finger is to analyse the convex hull of the object. The awl shape of the tool
object does remind much of a finger shape. In the OpenCV library a function
named cvConvexHull2 exists for finding the convex hull of a contour. There is
also a function for finding the deepest defects of the convex hull e.g. the points
most far away from every line segment of the convex hull. The two functions were
used to retrieve all points outlining the convex hull and the points constituting the
deepest defects of the hull. All five points defining the tool were found by these
functions. By finding the two deepest defect points, two of the five desired points
could be determined (the two points closest to the robot’s wrist), see figure 5.7. To
determine the three remaining points of the tool, lines were drawn between these
points and their nearest neighbours, resulting in four different lines. The four lines
5.5. FINDING THE CONTOUR MATCHING THE TOOL SHAPE 45

were then compared two by two to find the two lines being most parallel to each
other. See figure 5.8. By this method four out of five of the desired points were
easily determined. Finding the last point (TCP) of interest, was then becoming
easy while it was the only point lying on the complex hull fulfilling the constraint
of being a nearest neighbour of two of the points already found. The method
was fast and performed well as long as the robot only made reorientations around
axis five during the background subtraction method. When more complex robot
movements were used, a huge number of defect points in the convex hull were
found resulting in that the method became time consuming. Instead the polygonal
approximation method was evaluated.

Figure 5.7: The green lines is the convex hull found by the method, and the red
dots are the points defining the convex hull and the defect points.

Figure 5.8: The four lines are illustrated in this figure. The green lines are the two
lines being most parallel and found by the logics. The red lines are not parallel
and will therefore be neglected.

5.5.2 Polygonal Approximation of Contours

To further compress the retrieved contour several methods can be used such as Run
Length Encoding compression and polygonal approximation. In this implemen-
tation the Douglas-Peucker polygonal approximation was used (cvApproxPoly in
46 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

the OpenCV library). The reason of using a polygon approximation method was
due to the possibility to set the approximation accuracy to a level where the tool
object only consisted of five points see figure 5.9

Figure 5.9: Five points polygonal approximation of the tool.

5.5.2.1 Douglas-Peucker Approximation Method

The method starts in the two points p1 and p2 on the contour having the largest
internal distance. The algorithm loops through all points on the contour to retrieve
the largest distance from the line p1 p2 . If the maximum distance is lower than the
desired accuracy threshold, the process is finished. If not the point p3 at the longest
distance from the line p1 p2 is added to the resulting contour. The line p1 p2 is then
split into the two line segments p1 p3 and p3 p2 . The same methodology is then
applied recursively until the desired accuracy constraint is fulfilled.[18]

5.6 Geometric constraints

When the five points were found some logics were applied to make sure the five
points fulfilled the geometry of the tool shape. The angles α1 , α2 and α3 , between
the four different lines L1 , L2 , L3 and L4 of the shape, illustrated in figure 5.10,
were determined and the area A of the tool was calculated. The longest sides L1
and L2 of the tool had to fulfill the constraint of being parallel and the length of
the long sides L1 and L2 had to be at least twice the length of the short sides L3
and L4 . If all these constraints were fulfilled, the five points were classified as part
of the tool shape. To ensure only one tool was to be found, the constraints were set
to be very tough. If the method did not find a tool when the toughest constraints
were used, the constraints were gradually slackened.

Figure 5.10: Definition of geometric constraints of the tool.

5.7. CORNER DETECTION 47

5.7 Corner Detection

When the tool contour was found by the polygonal approximation algorithm, five
points defining the object were obtained. To use the camera as a measurement
unit, at least three points had to be found. Unfortunately only three of the five
points defining the contour of the tool could be found with high accuracy. Due to
occlusion the two points closest to the robot wrist were hard to detect. Instead the
tool center point (the tip of the awl) and the two points where the cylinder part of
the awl starts were used. See figure 5.11

Figure 5.11: Calibration tool with the three points found.

All three points had one thing in common, their locations were at corners. By
matching the three points given by polygonal approximation algorithm with the
closest points of high certainty being a corner, the three points in the figure 5.11
was found.
To find the three points of high probability being corners in the neighbourhood
of the points given by the polygonal approximation, three different methods were
evaluated.

5.7.1 Harris corner detector

The idea of Harris corner detection algorithm described in [22] is to use a function

w(x, y)(I(x + u, y + v) − I(x, y))2

X
E(x, y) = (5.10)

where w(x, y) is a window function, typically a Gaussian function, and I(x +

u, y + v) is a shifted version of the intensity image I(x, y). At a corner pixel the
difference between I(x + u, y + v) and I(x, y) would be high resulting in a high
value of E(x, y).
For a small shift (the u and v are small) a bilinear approximation can be used.
!" #
Ix2 Ix Iy u
E(x, y) ∼
h iX
= u v w(x, y) (5.11)
x,y Ix Iy Iy2 v
48 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

Where Ix is the x-derivate and Iy is the y-derivate of the image I.

!
X Ix2 Ix Iy
Let M = w(x, y) (5.12)
x,y Ix Iy Iy2

Equation 5.11 and equation 5.12 give

" #
u
E(x,y) ∼
h i
= u v M (5.13)
v

The eigenvalues λ1 and λ2 of M then include the information of the directions

of the fastest and the slowest changes. As mentioned earlier in this section a corner
has fast intensity changes in all direction, resulting in high λ1 and λ2 values.

λ1 ≈ λ2 = high values Corner

λ1 < λ 2 Edge
λ1 > λ 2 Edge
λ1 ≈ λ2 = small values Flat
Table 5.1: Interpretation of the eigenvalues.

The Harris corner detection algorithm makes use of the information stored in
the eigenvalues of M.

R = det(M) − k(trace(M)) = λ1 λ2 − k(λ1 + λ2 ), where k is a parameter.

(5.14)

R |R| Result
> 0 Large Corner
< 0 Large Edge
− Small Flat
Table 5.2: Interpretation of R.

The Harris corner detection algorithm was implemented in Matlab. Figure

5.13 shows the R image when the algorithm was applied to figure 5.12. By thresh-
olding the R image, only points with very high certainty for being a corner were
left.
The Harris corner detection algorithm is also implemented in the OpenCV
function cvGoodFeaturesToTrack.
5.7. CORNER DETECTION 49

Figure 5.12: Image of robot tool used for Harris corner detection.

Figure 5.13: The R image in Harris corner detection algorithm.

5.7.1.1 Sub pixel Accuracy

By using the positions of the corners found by the Harris detector as input to
the OpenCV function cvFindCornerSubPix, the positions of the corners could be
retrieved with sub pixel accuracy.
Let i be a minimization function, pi is a point in the neighborhood of q, ∇Ii
is the gradient in the point pi , q is the corner with sub pixel accuracy, see figure
5.14.
i = ∇IiT • (q − pi ) (5.15)

Figure 5.14: Accurate sub pixel corner detection. The blue arrows are the image
gradients and the green vectors are the vectors from q to pi .
50 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

Observe, all vectors qpi are orthogonal to ∇Ii . To retrieve the corner point q
with sub pixel accuracy, i in equation 5.15 was minimized for several points pi
resulting in a system of equations giving q.

∇Ii ∇IiT )−1 ( ∇Ii ∇IiT p1 )

X X
q=( (5.16)
i i

Please, refer to [18] for more information about sub pixel accuracy.

5.7.2 Fast radial

G. Loy and A. Zelinsky proposed a method in [23] based on Radial Symmetry
for finding points of interest. A Matlab implementation by the authors was found.
See the result image in figure 5.15 when the algorithm was applied to figure 5.12.

Figure 5.15: Result of the Fast Radial algorithm.

The fast radial algorithm resulted in more points of interest in relation to the
Harris corner detector. In this particular case the algorithm found many false hits.
Due to this effect, the fast radial algorithm was rejected.

5.7.3 Orientation Tensor

To be able to find the three points of the tool with sub pixel accuracy the orien-
tations of the tool sides can be used. The orientation of every pixel at the tool’s
contour was found in double angle representation [24]. To find the double angle
representation the orientation tensor was determined.
Several ways of determine the orientation tensor have been proposed. In [24]
G. Granlund and H. Knutsson propose a method based on quadrature filter re-
sponses. C. Mota, I. Stuke, and E. Barth describe a method in [25] for creating
the orientation tensor as an outer product of derivatives. G Farnebäck proposed
another method for retrieving the orientation tensor in [26] based on polynomial
expansion.
5.7. CORNER DETECTION 51

The method based on outer products of derivatives was implemented in Mat-

lab.
! !
2
T11 T12 Ixx Ix Iy
T= = 2 (5.17)
T21 T22 Ix Iy Iyy
Ixx ,Ixx and Ix Iy were retrieved by 2D convolutions of image I(x, y) and de-
rived 2D Gaussian functions.

Let:
−X 2
g(x) = e 2σ2
−Y 2
g(y) = e 2σ2
∂g(x) −X −X 2
= 2 e 2σ2
∂x σ
∂g(y) −Y −Y 2
= 2 e 2σ2
∂y σ
∂g(x)
h1 (x, y) = ∗ g(y)
∂x
∂g(y)
h2 (x, y) = ∗ g(x)
∂y
Figure 5.16 shows h1 (x, y) and h2 (x, y). By performing a 2D convolution of
h1 (x, y) and image I(x, y) the image Ix was retrieved. Observe, a faster way to
implement the function is to perform a sequence of 1D convolutions on image
I(x, y).
∂g(y)
Ix = I(x, y) ∗ g(x) ∗
∂y
The double angle representation of orientation can be retrieved from the ori-
entation tensor T.
Let v be the double angle orientation vector.
! ! !
v1 T11 − T22 Ix2 − Iy2
v= = =
v2 2T12 2Ix Iy

To ensure v is the double angle representation let Ix = cos(φ) and Iy = sin(φ)

! ! !
Ix2 − Iy2 cos(φ)2 − sin(φ)2 cos(2φ)
v= = =
2Ix Iy 2 cos(φ) sin(φ) sin(2φ)

The advantage of using a double angle representation of orientation is the pos-

sibility to avoid branch cuts in the representation. If only the angle between the
52 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

Figure 5.16: Gaussian derived in x and y directions respectively.

x-axis to an image line is used as orientation information, no average between

neighboring points can be done. For instance a line with an angle close to 0 rad
and a line with an angle close to π rad will have very different orientations. When
using the double angle representation these two lines will instead have orientations
close to each other. This implies the possibility of counting an average of orien-
tations when using the double angle representation. By color coding the double
angle and letting the intensity be |v| an orientation image can be retrieved, please
refer to [24]. Figure 5.18 shows the orientation image of 5.17

Figure 5.17: Test image for double angle representation.

5.7. CORNER DETECTION 53

Figure 5.18: Orientation image of figure 5.17

By finding the orientation tensor to the image achieved by the camera and
calculating the orientation image, figure 5.19 was retrieved. For every pixel a line
was drawn in the direction specified by the orientation image. If all lines were
accumulated the three points could be found by applying a high threshold.
Observe, by calculating the histogram of the orientation image it is possible to
tell if the tool was found in the image. Figure 5.20 shows the histogram. Three
peaks are observed, the highest peak is the orientation of the two longest sides
of the tool, while the two lower peaks are the orientations of the two shortest
sides of the tool. The relations between these three orientations are specific for
the tool. Of course the tool has to be segmented before calculating the histogram,
otherwise the tool specific orientations would be impossible to determine among
all the orientation information in the image. The histogram can also be used to
find the orientation of the tool (the highest peek in the histogram), after the tool
has been segmented.

Figure 5.19: Double angle representation of the robot tool.

54 CHAPTER 5. IMAGE SEGMENTATION OF ROBOT TOOL

Figure 5.20: Histogram of orientations.

Chapter 6

Results

In this chapter the results of different tests will be presented. The tests were
performed to determine the accuracy of the method.

6.1 Repetition Accuracy of the Camera

Figure 6.1 shows the configuration of the camera in relation to the robot tool flange
coordinate system. The optical axis of the camera is directed in -X direction,
where X is the X axis of the robot base coordinate system.

Figure 6.1: Configuration during the test.

To be able to determine the repetition accuracy of the camera the start position
h iT
of the TCP found by the method was stored in a vector POS1 = x1 y1 z1 .
The robot was then randomly moved to a second position. The new position of
h iT
TCP was measured by the method several times and stored as POSi = xi yi zi .

55
56 CHAPTER 6. RESULTS

Figure 6.2: Movment mi plotted for all i.

Where i is the estimation number. The tool used during this test was the chess
board in figure 4.3 and 24 points of the chess board pattern were used to determine
the transformationq between the camera and the tool (chess board). For all i the
movement mi = (x1 − xi )2 + (y1 − yi )2 + (z1 − zi )2 was calculated.
Figure 6.2 shows the result of mi .
By the graph it is possible to determine the difference between the maximum
and minimum interval to approximately 0.03 mm, resulting in that the camera is
not able to measure distances shorter than 0.03 mm.

6.1.1 Averaging to achieve higher accuracy

By letting every estimation mj be the average of several mi see equation 6.1, a
test was performed to investigate whether a better accuracy could be obtained.

P10
i=1 mi
mj = (6.1)
10

During the test every mj was the average of ten different mi . The result is illus-
trated in figure 6.3. The difference between the maximum and minimum interval
was still determined to 0.03 mm. It is then obvious that averaging the result does
not increase the repetition accuracy of the camera when 24 points are used for
measuring the transformation between the camera and the tool (chess board).
6.1. REPETITION ACCURACY OF THE CAMERA 57

Figure 6.3: Movment mj plotted for all j.

6.1.2 How does the number of points affect the repetition ac-
curacy
To investigate how the amount of points in the chess board pattern affects the
repetition accuracy of the camera, the same test (not with averaging) was done
again. Although, this time only the three most upper left points of the chess board
pattern were used to measure the transformation between the camera and the tool
(chess board).
Observe, three points are minimum for finding the transformation between the
camera and the tool. This is due to the fact that a plane is needed to be observed
by the camera to determine the homography, see section 4.2. A plane is by its
definition defined of at least three points.
The result of the new test is illustrated in figure 6.4 The difference between
the maximum and minimum interval was determined to approximately 0.58 mm.
The greater difference between the maximum and minimum interval indicates the
number of points used during the test affects the repetition accuracy. Although,
still using only three points the camera repetition accuracy is better than 1 mm.
58 CHAPTER 6. RESULTS

Figure 6.4: Movment mi plotted for all i when only 3 points of the chess board
were observed.
6.2. CAMERA VERSUS ROBOT MEASUREMENTS 59

6.2 Camera versus Robot measurements

Both the camera and the robot measurements are affected of errors during the
camera calibration and the robot calibration respectively. To investigate the effect
of these errors different tests were performed. The configuration of the devices is
illustrated in figure 6.1. All twenty four points of the chess board were observed
during these tests.

6.2.1 Rotation measurements

The robot was told to perform a rotation of θ = 20◦ around a random axis and the
forward kinematics transformation matrix T1 and the measured transformation
matrix between the camera and the tool T2 were saved before and after the rota-
tion, see figure 4.1 at page 21. The transformation A from the start position to the
final position measured by the camera is then A = T−1 2af ter T2bef ore and the trans-
formation B from the start position to the final position measured by the robot is
then B = T−1 1af ter T1bef ore , see figure 4.2 at page 25. Due to the fact that A and B
are homogeneous matrixes it is obvious that the rotation part of the transforma-
tions A and B can be obtained as the upper left 3 × 3 sub matrixes of A and B
respectively.
Denote these sub matrixes RA and RB respectively. The angle of rotation
θrobot measured by the robot and the angle of rotation θcamera measured by the
camera were then obtained by using the screw axis rotation representation de-
scribed in section 3.3.2 on page 17 and equation 3.27 on page 18.
The difference γ between θrobot and θcamera was then calculated accorded to
the following equation.
γ = |θrobot | − |θcamera | (6.2)
The test was done several times and the resulting γ:s were plotted in figure
6.5. The difference between the maximum and minimum interval was determined
to 0.06◦ and the maximum deviation was 0.04◦
60 CHAPTER 6. RESULTS

Figure 6.5: Difference of rotation measured by the robot versus the camera.
6.2. CAMERA VERSUS ROBOT MEASUREMENTS 61

6.2.2 Translation measurements

The robot was told to perform linear movements of 10 mm along the three orthog-
onal axes of the robot base coordinate system. The positions of the robot measured
by the robot and the camera respectively were stored after every movement.
h iT
Denote the positions of the robot measured by the robot as robposi = rxi ryi rzi
and the positions of the robot measured by the camera and transformed to the
h iT
robot’s base frame by applying X2 as camposi = cxi cyi czi where i ∈
[1,N ] and N is the number of movements.
Let:

robdiff i = robposi+1 − robposi

camdiff i = camposi+1 − campos i 
dif fxi
diff i = robdiff i − camdiff i =  dif fyi 
 

dif fzi
Figure 6.6 illustrates |diff | during movements in the X direction of the robot
base coordinate system. Figure 6.7 and Figure 6.8 illustrates |diff | during move-
ments in Y direction and Z direction of the robot base coordinate system respec-
tively. Table 6.1 shows the resulting differences between maximum and minimum
intervals and the maximum deviations. Observe, the accuracy is higher in the Y
and Z directions in relation to the X direction.

Figure 6.6: Difference of translation in X direction measured by the robot versus

the camera.
62 CHAPTER 6. RESULTS

Figure 6.7: Difference of translation in Y direction measured by the robot versus

the camera.

Figure 6.8: Difference of translation in Z direction measured by the robot versus

the camera.
6.2. CAMERA VERSUS ROBOT MEASUREMENTS 63

Movement in direction Difference between maximum and Maximum deviation

minimum interval [mm] [mm]

X direction 2.72 1.45

Y direction 1.49 1.16

Z direction 1.66 1.05

Table 6.1: Table showing the differences between maximum and minimum inter-
vals and the maximum deviations.
64 CHAPTER 6. RESULTS

6.2.2.1 Averaging to achieve higher accuracy

The same procedure was redone, but this time every robot position, camposj ,
measured by the camera was the average of ten measurements.
P10
i=1 camposi
camposj =
10
The results are illustrated in figure 6.9, figure 6.10 and figure 6.11. Table 6.2
shows the resulting differences between maximum and minimum intervals and
the maximum deviations.
Note, the accuracy has increased in all directions in relation to the test when
no averaging was used. Although, the accuracy of movements in the X direction
is still lower than the accuracy in Y or Z directions.
Observe, the accuracy of these tests are approximately ten times lower than
for the camera repetition accuracy. This shows the importance of using the itera-
tive method for increasing the accuracy of the TCP calibration method, since the
iterative method is independent of the robot calibration error, see section 4.5 on
page 31.

Figure 6.9: Difference of translation in X direction measured by the robot versus

the camera, averaging has been done.
6.2. CAMERA VERSUS ROBOT MEASUREMENTS 65

Figure 6.10: Difference of translation in Y direction measured by the robot versus

the camera, averaging has been done.

Figure 6.11: Difference of translation in Z direction measured by the robot versus

the camera, averaging has been done.
66 CHAPTER 6. RESULTS

Movement in direction Difference between maximum and Maximum deviation

minimum interval [mm] [mm]

X direction 0.56 0.33

Y direction 0.26 0.49

Z direction 0.21 0.22

Table 6.2: Table showing the differences between maximum and minimum inter-
vals and the maximum deviations. Averaging has been done.
6.3. ACCURACY OF THE TCP CALIBRATION METHOD 67

6.3 Accuracy of the TCP Calibration method

The accuracy of the proposed method is limited to the least distance between two
points which the camera is able to register, see section 4.5. Due to the fact that the
repetition accuracy of the camera is 0.03 mm, see section 6.1, there is no need of
trying to register a shorter distance.
A test was performed to find the shortest distance between two points which
the camera is able to register. Denote the initial position of the tool center point,
determined by the camera and transformed to the robot base coordinate system
as p1 . This position was calculated as the mean value of 100 different measure-
ments. The robot was then told to move a distance d and the new position, p2 , was
measured by the camera and transformed to the robot base coordinate system.
The test was performed until the smallest distance d between p1 and p2 was
found, still making the camera able to register a movement. To ensure the camera
registered a movement, the distance, dcamera , measured by the camera had to be
greater than the repetition accuracy of the camera (of course dcamera was also
expressed in the robot base coordinate system).
Figure 6.12, figure 6.13 and figure 6.14 illustrate the distance dcamera when
the robot was told to move a distance d = 0.10 mm in the X, Y and Z direc-
tions respectively. By these figures it is possible to tell that the accuracy in the
X direction limits the total accuracy of the method. When the robot was told to
move a distance shorter than 0.1 mm some of the measurements, dcamera , became
lower than the camera repetition accuracy, resulting in that the system was not
able to determine a significant movement. Therefore the accuracy of the proposed
method is assumed to be 0.1 mm.

Figure 6.12: Distance dcamera registered by the camera when a movement of 0.1
mm in X-direction was performed by the robot. The dashed line is the camera
repetition accuracy.
68 CHAPTER 6. RESULTS

Figure 6.13: Distance dcamera registered by the camera when a movement of 0.1
mm in Y-direction was performed by the robot.

Figure 6.14: Distance dcamera registered by the camera when a movement of 0.1
mm in Z-direction was performed by the robot.
6.4. REPETITION ACCURACY OF THE TCP CALIBRATION METHOD 69

6.3.1 Averaging to obtain a higher accuracy

The same tests were performed once again, but this time both positions of the
tool, p1 and p2 , were determined as an average of 100 measurements. Table
6.3 shows the results of the distance dcamera measured by the camera, when a
movement d = 0.06 mm was performed by the robot. If the distance d was
chosen to d < 0.06 mm no significant movement was registered by the camera.
The result of this test assures an increased accuracy of the method, if the average
of several measurements are used.

d dcamera

The robot moves 0.06 mm in X direction 0.04 mm

The robot moves 0.06 mm in Y direction 0.04 mm

The robot moves 0.06 mm in Z direction 0.04 mm

Table 6.3: Significant movements registered by the camera.

6.4 Repetition Accuracy of the TCP Calibration method

To test the repetition accuracy of the TCP calibration method presented in this the-
sis, the method was executed several times for the same tool. The tool used was
the chess board pattern and all 24 points of the chess pattern were observed. The
desired accuracy of the iterative method was set to 1 mm. Denote  the translation

T CPx
from the tool flange to the TCP found by the method as TCP =   T CPy . The


T CPz
result of T CPx , T CPy , T CPz for the different runs of the method are illustrated
in figure 6.15, figure 6.16 and figure 6.17 respectively. Table 6.4 shows the dif-
ferences between the maximum and minimum interval for each of the results. By
observing the information in table 6.4, it is obvious the desired accuracy of 1 mm
was obtained.
70 CHAPTER 6. RESULTS

Figure 6.15: Repetition accuracy of T CPx measured by the proposed TCP cali-
bration method.

Figure 6.16: Repetition accuracy of T CPy measured by the proposed TCP cali-
bration method.
6.4. REPETITION ACCURACY OF THE TCP CALIBRATION METHOD 71

Figure 6.17: Repetition accuracy of T CPz measured by the proposed TCP cali-
bration method.

Element of TCP Difference between maximum and minimum interval [mm]

T CPx 0.97

T CPy 0.52

T CPz 0.48
q
T CPx2 + T CPy2 + T CPz2 0.66

Table 6.4: Difference of max and min interval of |TCP| and for the elements of
TCP respectively.
72 CHAPTER 6. RESULTS

6.4.0.1 How does the number of points affect the repetition accuracy of the
TCP calibration method
To investigate how the number of points affect the repetition accuracy of the pro-
posed TCP calibration method, the same procedure was executed but only the
three most upper left points of the chess board pattern were observed. The method
managed to give a satisfying result most of the times, but the method became un-
reliable and very time consuming. Sometimes the robot diverged from the correct
TCP point during the iterative phase. This implies more than three points have to
be observed to make the method reliable.
Although, the reason why the robot sometimes diverged during the iterative
procedure might be that the wrong solution to the three points was found. If
three points with almost the same distances to each other are observed, it might
be difficult for the algorithm to distinguish between the points. This results in
three possible solutions, but only one of the solutions is correct. If the mirrored
solutions are concerned, six different solutions are possible to find. The mirrored
solutions were removed during the test by only permitting solutions with Z > 0,
where Z is the camera coordinate axis coinciding with the optical axis.
Chapter 7

Discussion

This chapter presents the conclusion of this Master’s thesis. Problems and diffi-
culties are described and the chapter concludes with a discussion of future devel-
opment.

7.1 Conclusion
This thesis has proposed a new method for tool center point calibration. The
method can be split into three different parts.
1. Calculation of a good guess of the Tool Center Point, by solving a closed
loop of transformations resulting in a homogeneous equation AX1 = X1 B.

2. Image Segmentation of the tool.

3. Iterative method for increasing the accuracy.

The method proposed is completely automated, resulting in no variations in
accuracy, due to how skilled the robot operator is. The method is also very easy to
configure, only a USB camera connected to a laptop computer with the software
developed during this project is needed. No fixed devices with known relations
to the robot’s base frame are needed, implying the calibration set up used in this
method is completely portable.

7.1.1 Accuracy
Tests have been performed with regard to investigate the accuracy of the pro-
posed method. The tests showed the need of the iterative method to achieve a
satisfying accuracy. Due to the fact that only accuracy of the camera when reg-
istering a movement between two different positions limits the accuracy of the

73
74 CHAPTER 7. DISCUSSION

iterative method, an accuracy of approximately 0.1 mm is possible to achieve by

this method if a camera of the same model is used, see section 6.3. By that remark,
this method is able to perform a tool center point calibration with equivalent ac-
curacy or even better in relation to the manual four point TCP calibration method.

7.1.2 Major Problems/known difficulties

During the project two main problems have occurred.

1. Finding distinct points

2. Light reflections in the tool

7.1.2.1 Finding distinct points

To be able to determine the transformation between the camera and the tool, at
least three distinct points on the tool have to be observed by the camera. In this
thesis results have been presented, showing that it is possible to perform a cal-
ibration by only observing three points. However, to make the method reliable
and fast, more points are needed. If the tool has a very simple geometry it might
be difficult to find several points and assure it is the same points found in every
frame.
This problem can be solved by for instance painting the tool with a certain
pattern. Another possibility would be to press a matrix pattern on the tool. The
matrix pattern could then also store information about what kind of tool it is.
The best idea though, would probably be to ensure that the tool has some kind
of structure e.g. not a simple geometric shape.

7.1.2.2 Light reflections in the tool

A major problem during the development of the image segmentation part of the
method, was the influence of light reflections in the tool. This is a well known
problem within the field of image segmentation. Several methods exist to make
image segmentation robust against reflections, but no suitable functions for solv-
ing this problem exist in the Open Computer Vision library. Instead the problem
was solved by painting the tool in a non reflecting color.
Before using the method in an industrial environment, this problem must be
solved.
7.2. FUTURE DEVELOPMENT 75

7.1.3 Fulfillment of objectives

The objectives of the thesis is to investigate whether or not it is possible to create
an automated method for tool center point calibration, by making good use of
computer vision and image processing technologies. To consider the method as
useful; an accuracy of ±1 mm had to be obtained.
In this thesis a method has been proposed and a software has been delivered
with the ability to successfully perform tool center point calibrations. The acu-
racy of the proposed method has been analyzed, showing the method’s ability of
performing TCP calibrations with an accuracy of approximately 0.1 mm, see the
discussion in section 6.3. Therefore all objectives are considered to be fulfilled.

7.2 Future Development

In this section some future development will be presented.

1. Rotation symmetric tools

2. Light reflections in the tool

3. Image Segmentation controlled by CAD drawings

4. Neural network based methods

5. Online correction of the TCP calibration

6. Other image processing libraries

7.2.1 Rotation symmetric tools

One problem of the proposed method is to calculate a good initial guess of the
tool center point, if the robot tool has an axis of symmetry. The symmetry makes
it impossible to measure the transformation between the camera and the tool by 6
degrees of freedom. A mathematical proof, showing it is impossible to measure
a good guess of TCP by using this method if the tool is rotation symmetric, is
published in the thesis. Although, the iterative part of the method still works for
rotation symmetric tools, if the transformation between the camera and the robot
coordinate system is known and an initial guess of the tool center point is obtained
by another procedure.
The alternative method for finding the transformation between the camera
frame and the robot’s base frame presented in section 4.6 should be implemented
and evaluated.
76 CHAPTER 7. DISCUSSION

Observe, other methods for solving this problem exist. For instance the rota-
tion symmetry can be removed by painting the tool with a non symmetric pattern.
Attaching a blinking light emitting diode (LED) to the tool, easy to find by image
segmentation, can also remove the symmetric geometry.
By fasten a calibration pattern, for instance a chess board pattern, at the tool
it is possible to determine the transformation between the camera frame and the
robot’s base frame. The iterative method then works even though the tool is rota-
tion symmetric. Instead of fasten a calibration pattern at the tool, a matrix pattern
can be pressed on the tool. Pressing a pattern on the tool will also solve the prob-
lem of finding distinct points, see section 7.1.2.1.

7.2.2 Light reflections in the tool

Before using the method in an industrial environment, the problem of handling
light reflections in the tool, see section 7.1.2.2, must be solved.

7.2.3 Image Segmentation controlled by CAD drawings

To make the method general it needs the ability of calibrating all kinds of tools.
Therefore the next step is to control the image segmentation part of the method by
data from for instance a CAD drawing of the actual tool.
Developing a software with the ability to read 3D drawings of tools and an-
alyze the drawings to find distinct points or lines to search for during the image
segmentation would ensure that the method would be able to handle different kind
of tools. It would be even better if only one general 3D model for each kind of
tool would be enough to control the image segmentation method. Then the user
would only need to choose what kind of tool he/she wants to calibrate.

7.2.4 Neural network based methods

When the method is able to handle different kinds of tools, it would be possible to
make the system able to distinguish between different tools by itself e.g. classify-
ing tools. This can be done by making use of a neural network approach such as
principle component analysis (PCA).
As soon as a robot tool is located in front of the camera, the system would
then be able to classify the tool and start the calibration procedure.

7.2.5 Online correction of the TCP calibration

There are several events which can cause misplacement of the tool center point
during a robot program. The method proposed in this thesis could easily be re-
7.2. FUTURE DEVELOPMENT 77

configured for correction of the current TCP calibration. By placing the camera
at a fixed position and once and for all determine the relation between the camera
and the robot base frame, only the iterative part of the method is needed to deter-
mine the correct placement of the tool center point. The initial guess of the TCP
is then of course chosen to be the current tool center point.

7.2.6 Other image processing libraries

The image processing library (Open Computer Vision Library) used during this
project, does not include suitable functions for handling light reflections in metal-
lic tools. Therefore it could be of great interest to try other image processing
libraries, for instance the library developed by Cognex or the library developed by
Halcon.
Appendix A

Appendix

This appendix will present the results of the camera calibration done during this
project.

A.1 Finding intrinsic parameters

The camera calibration toolbox for Matlab by Jean-Yves Bouguet was used to
perform the camera calibration. Fifty two images of a chess board pattern were
achieved by the camera. The chess board was rotated and translated between
every frame. Figure A.1 illustrates the positions of the chessboard. The camera
calibration toolbox was then used to determine the intrinsic parameters presented
in table A.1. Figure A.2 illustrates the reprojection errors in pixels after the camera
calibration was done.
Also the distortion coefficients were calculated and the result can be seen in
table A.1. Figure A.3 and figure A.4 illustrates the radial and the tangential lens
distortion respectively. By the figure it is possible to tell that the radial distortion
is greater than the tangential distortion. Figure A.5 illustrates the complete lens
distortion.

79
80 APPENDIX A. APPENDIX

Calibration results (with uncertainties):

" # " #
615.17386 3.32053
Focal Length: fc = ±
615.69958 3.41099
" # " #
337.27613 4.32134
Principal point: cc = ±
233.77556 4.42747

Skew: alpha c = [0.00000] ± [0.00000] =>

angle of pixel axes = 90.00000 ± 0.00000 degrees
   
−0.20033 0.00863
0.25346 0.02484
   
   
   
Distortion: kc =  0.00009 ± 0.00164 
   

 −0.00143 


 0.00166 

0.00000 0.00000
" #
0.06992
Pixel error: err =
0.07343

Note: The numerical errors are approximately three

times the standard deviations (for reference).

Table A.1: The camera calibration result given by camera calibration toolbox for
Matlab.
A.1. FINDING INTRINSIC PARAMETERS 81

Figure A.1: Illustration of the chess board positions.

82 APPENDIX A. APPENDIX

Figure A.2: Reprojection errors in pixels.

A.1. FINDING INTRINSIC PARAMETERS 83

Figure A.3: Radial lens distortion.

84 APPENDIX A. APPENDIX

Figure A.4: Tangential lens distortion.

A.1. FINDING INTRINSIC PARAMETERS 85

Figure A.5: Complete lens distortion.

Bibliography

[1] ABB, Rapid reference manual Industrial robot controller system. ABB.

[2] P.-E. Danielsson., “Bildanalys,” in Bildanalys, pp. 52–63, 2005.

[3] M. M. Seger, “Lite om kamerageometri.” September 2005.

[4] Donald Hearn, M.Pauline Baker, Computer Graphics with OpenGL. Pearson
Education, Inc. Upper Saddle River: Pearson Prentice Hall, third ed., 2004.

[5] Lung-Wen Tsai, Robot Analysis The Mechanics of Serial and Parallel Ma-
nipulators. 605 Third Avenue, New York, N.Y. 10158-0012: John Wiley &
Sons, INC., 1998.

[6] Z. Zhang, “A flexible new technique for camera calibration,” technical re-
port, Microsoft Research, http://research.microsoft.com/ zhang, 8 2002.

[7] Jintao Wang, Daokui Qu and Fang Xu, “A new hybrid calibration method
for extrinsic camera parameters and hand-eye transformation,” International
Conferance on Mechatronics and Automation Niagara Falls, Canada, July
2005.

[8] H. Zhuang, Z. Roth, and R.Sudhakar, “Simultaneous robot/world and

tool/flange calibration by solving homogeneous transformation of the form
AX=YB,” IEEE Transaction on Robotics and Automation, vol. 10, pp. 549–
554, August 1994.

[9] Fadi Dornaika and Radu Horaud, “Simultaneous robot-world and hand-eye
calibration,” IEEE Transaction on Robotics and Automation, vol. 14, August
1998.

[10] Roger Y. Tsai and Reimark K.Lenz, “A new technique for fully au-
tonomous and efficient 3d robotics hand/eye calibration,” IEEE Transaction
on Robotics and Automation, vol. 5, June 1989.

87
88 BIBLIOGRAPHY

[11] “Matrix reference manual: Matrix relations.”

Internet link. http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/relation.html#similarity.

[12] “3d rotations (wrf).” Internet link. http://www.ecse.rpi.edu/Homepages/wrf/Research/

Short Notes/rotation.html#find.

[13] “Rotation matrix.” Internet link. http://mathworld.wolfram.com/RotationMatrix.html.

[14] “Sourceforge OpenCV.” Internet link. http://sourceforge.net/projects/opencvlibrary/.

[15] “Intel OpenCV.” Internet link. http://www.intel.com/technology/computing/opencv/.

[16] Björn Svensson, Johanna Pettersson, Hans Knutsson, “Bildregistrering ge-

ometrisk anpassning av bilder.” Mars 2004.

[17] C. Galambosi J. Matastsand J. Kittlert, “Progressive probabilistic hough

transform for line detection,” Computer Vision and pattern Recognition,
vol. 1, pp. 733–737, June.

[18] Intel Corporation, http://surfnet.dl.sourceforge.net/sourceforge/opencvlibrary/

OpenCVReferenceManual.pdf, Open Source Computer Vision Library Ref-
erence Manual, first ed., 12 2000.

[19] J.Canny, “A computational approach to edge detection,” IEEE Transaction

on Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp. 679–698,
1986.

[20] K. A. S. Suzuki, “Topological structural analysis of digital binary images by

border following,” CVGIP, pp. 32–46, July 1985.

[21] F. Albregtsen, “Active contours and region based segmentation,” INF 269
Digital Image Analysis, p. 25, 10 2002. Lecture 6.

[22] C. Harris and M.J. Stephens, “A combined corner and edge detector,” Alvey
Vision Conference, pp. 147–152, 1988.

[23] G. Loy and A. Zelinsky, “Fast radial symmetry for detecting points of in-
terest,” IEEE Transaction on Pattern Analysis and Machine Intelligence,
vol. 25, pp. 959–973, August 2003.

[24] Gösta H. Granlund, Hans Knutsson, Signal Processing for Computer Vision.
P.O Box 17, 3300 AA Dordrecht The Netherlands: Kluwer Academic Pub-
lishers, 1995.
BIBLIOGRAPHY 89

[25] C. Mota, I. Stuke, and E. Barth, “Analytic solutions for multiple motions,”
International Conference on Image Processing, 2001. Institute for Signal
Processing, University of Lübeck Ratzeburger Allee 160, 23538 Lübeck,
Germany.

[26] G. Farnebäck, Polynomial Expansion for Orientation and Motion Estima-

tion. PhD thesis, Linköpings University, Department of Electrical Engineer-
ing Linköpings University, SE-581 83 Sweden, 2002.
På svenska

Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare –

under en längre tid från publiceringsdatum under förutsättning att inga extra-
ordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner,
skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för
ickekommersiell forskning och för undervisning. Överföring av upphovsrätten
vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av
dokumentet kräver upphovsmannens medgivande. För att garantera äktheten,
säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ
art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i
den omfattning som god sed kräver vid användning av dokumentet på ovan
beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan
form eller i sådant sammanhang som är kränkande för upphovsmannens litterära
eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press se
förlagets hemsida http://www.ep.liu.se/

In English

The publishers will keep this document online on the Internet - or its possible
replacement - for a considerable time from the date of publication barring
exceptional circumstances.
The online availability of the document implies a permanent permission for
anyone to read, to download, to print out single copies for your own use and to
use it unchanged for any non-commercial research and educational purpose.
Subsequent transfers of copyright cannot revoke this permission. All other uses
of the document are conditional on the consent of the copyright owner. The
publisher has taken technical and administrative measures to assure authenticity,
security and accessibility.
According to intellectual property law the author has the right to be
mentioned when his/her work is accessed as described above and to be protected
against infringement.
For additional information about the Linköping University Electronic Press
and its procedures for publication and for assurance of document integrity,
please refer to its WWW home page: http://www.ep.liu.se/

© [Johan Hallenberg]