See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/348430673

Simultaneous Calibration of Multicoordinates for a Dual-Robot System by

Solving the AXB = YCZ Problem

Article in IEEE Transactions on Robotics · January 2021

DOI: 10.1109/TRO.2020.3043688

CITATIONS READS
83 773

7 authors, including:

Gang Wang Dahu Zhu

Chongqing University Wuhan University of Technology
23 PUBLICATIONS 339 CITATIONS 85 PUBLICATIONS 2,476 CITATIONS

SEE PROFILE SEE PROFILE

He Xie
Hunan University
15 PUBLICATIONS 439 CITATIONS

SEE PROFILE

All content following this page was uploaded by Gang Wang on 18 January 2022.

The user has requested enhancement of the downloaded file.

1172 IEEE TRANSACTIONS ON ROBOTICS, VOL. 37, NO. 4, AUGUST 2021

Simultaneous Calibration of Multicoordinates

for a Dual-Robot System by Solving
the AXB = YCZ Problem
Gang Wang, Wen-long Li , Member, IEEE, Cheng Jiang , Da-hu Zhu , He Xie ,
Xing-jian Liu , Member, IEEE, and Han Ding, Member, IEEE

Abstract—Multirobot systems have shown great potential in

dealing with complicated tasks that are impossible for a single robot
to achieve. One essential problem encountered in cooperatively
working of the multirobot systems is the unknown initial trans-
formation relationships from hand to eye, base to base, and flange
to tool. In this article, the problem of multicoordinates calibration
for a dual-robot system is formulated to a matrix equation AXB =
YCZ. A novel approach for simultaneously solving the unknowns
in equation AXB = YCZ is proposed, which is composed of a closed
form method based on the Kronecker product and an iterative
method which converts the calculation of a nonlinear problem to
an optimization problem of a strictly convex function. The closed
form method is used to quickly obtain an initial estimation for the
iterative method to improve the efficiency and accuracy of iteration. Fig. 1. Problem formulation to calibrate the hand–eye, tool–flange, and robot–
In addition, a series of conditions on the solvability of the problem robot transformation matrices for a dual-robot system that can be formulated into
are proposed to guide the operators to select appropriate robot a matrix equation as AXB = Y CZ.
attitudes during the calibration process. To show the feasibility
and superiority of the proposed iterative method, two other cali-
bration methods are chosen to be compared to the proposed method collective behaviors, some complicated goals that are impossible
through simulation and practical experiments. The comparison for a single robot to achieve become feasible and attainable.
results verify the superiority of the proposed method in accuracy, Multirobot systems have various benefits compared to single
efficiency, and stability.
robot systems, including:
Index Terms—AXB = YCZ, calibration and identification, dual 1) Resolving task complexity.
arm manipulation, multirobot systems. 2) Increasing the performance.
3) Increasing reliability.
I. INTRODUCTION
4) Simplicity in design [1].
ULTIROBOT systems are a group of robots that are
M designed to perform some collective behaviors. By these
Multirobot systems have been applied to many industrial
manufacturing processes in aviation, automotive, energy, and
other fields to overcome some tricky tasks. For examples, Boe-
Manuscript received June 30, 2020; revised September 30, 2020; accepted
November 24, 2020. Date of publication January 11, 2021; date of current ing’s Fuselage Automated Upright Build (FAUB) multirobot
version August 5, 2021. This work was supported in part by National Natural system has automated approximately 60000 fasteners on the
Science Foundation of China under Grant 51535004 and Grant 52075203, in fuselage of the Boeing 777 and 777X [2], Tianjin University
part by Hubei Province Key Research and Development Plan under Grant
2020BAA025, and in part by Wuhan Applied Basic Research Project under Grant proposes a dual-arm collaborative mirror milling system to
2020010601012173. This article was recommended for publication by Associate machining large skin parts [3], Comau develops a multirobot
Editor G. Venture and Editor P. Dupont upon evaluation of the reviewers’ system named Openrobogate with a maximum of 18 robots for
comments. (Corresponding author: Wen-Long Li.)
Gang Wang, Wen-long Li, Cheng Jiang, He Xie, and Han Ding are with flexible welding and assembling of automobile body-in-white
the State Key Laboratory of Digital Manufacturing Equipment and Technol- [4], etc.
ogy, Huazhong University of Science and Technology, Wuhan 430074, China A multirobot system can be decomposed into several dual-
(e-mail: [email protected]; [email protected]; [email protected];
[email protected]; [email protected]). robot systems. As shown in Fig. 1, a typical form of dual-robot
Da-hu Zhu is with the Hubei Key Laboratory of Advanced Technology for system consists of two robots, a sensor (such as a camera or a
Automotive Components, Wuhan University of Technology, Wuhan 430070, scanner) fixed on the end flange of robot 1, an implementation
China (e-mail: [email protected]).
Xing-jian Liu is with the Department of Mechanical and Industrial Engineer- tool, and a calibration target attached to an operational tool
ing and the Robotics Institute, University of Toronto, Toronto M5S 2E8, Canada fixed on the end flange of robot 2. The prerequisite such that
(e-mail: [email protected]). the dual-robot system can work cooperatively is to calibrate the
Color versions of one or more figures available at https://doi.org/10.1109/
TRO.2020.3043688. transformation relationships from the end flange frame of robot
Digital Object Identifier 10.1109/TRO.2020.3043688 1 to the sensor frame, the base frame of robot 1 to the base frame
1552-3098 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: SIMULTANEOUS CALIBRATION OF MULTICOORDINATES FOR A DUAL-ROBOT SYSTEM 1173

of robot 2, and the end flange frame of robot 2 to the target frame system calibration utilizing a calibration model based on the
accurately. Product of Exponentials formula (POE formula) without nom-
inal kinematic parameters. The result of the calibration is not
only identifying the exact kinematic parameters of the robot
A. Related Work but also determining the pose of the robot base frame with
Robot hand-eye calibration is a classical single robot calibra- respect to a predetermined coordinate. Dan et al. [23] proposed
tion problem [5]–[9], which is to solve the pose transformation a kinematic modeling method and a base frame calibration
matrix from the frame of robot end flange to the sensor frame. method for a dual-machine-based drilling and riveting system.
The research on hand-eye calibration can be grouped into solving The experimental results show that the achievable positioning
two forms of homogeneous equations: AX = XB and AX = accuracy of dual machines is within 0.1 mm and 0.07°. The
Y B [10]–[12], where X and Y are unknown hand-to-eye and compensated base frame calibration accuracy is improved from
robot-to-world rigid transformation matrices, respectively. For 0.187 and 0.052° to 0.053 mm and 0.022°. Zhu et al. [24]
solving the problem of equation AX = XB, Li et al. [13] proposed a kinematic parameters and tool–flange transformation
introduced a hand-eye calibration method for a robot scanning matrix calibration method for a dual manipulator system based
system. A tool center point (TCP) calibration approach was on optical axis constraint. Another robot–robot and hand-eye
proposed to calibrate the pose from a 3-D scanner to a robot using calibration method based on virtual constraints are also proposed
a criterion sphere as the calibration object. Zhao [14] proposed a by Zhu et al. [25]. After calibration, the average errors of dual
hand-eye calibration algorithm using convex optimization, and it manipulator system are decreased more than 80%. Ruan et al.
can be solved in the form of a global linear optimization without [26] proposed a base–base calibration method for a coopera-
starting values. Andreff et al. [15] proposed a liner method that tive grinding robot system with binocular vision system. Wang
combines structure-from-motion with known robot motions, and et al. [27] proposed a contact-mode approach to calibrate the
this method can solve for both the hand-eye parameters and base–base transformation matrix of the cooperative manipula-
the unknown scale factor. Ren et al. [16] proposed a method tors based on the geometric constraints. The novel feature of this
to determine the pose between the 3-D scanner and the robot, approach is that only two calibration points are required. The
where the scanner is rigidly mounted on robot end-effector. above approaches can only calibrate one or two of X, Y , Z for
The experiment showed that the calibration accuracy reached a dual-robot system. However, X, Y , Z usually all need to be
up to 0.062 mm. Nguyen and Pham [17] provided a rigorous calibrated under actual conditions. Although these approaches
derivation of the covariance of the solution X in the problem of could be used to calibrate X, Y , Z step by step, the errors
solving AX = XB, when A and B are randomly perturbed would be transferred and accumulated between steps leading
matrices. The experiments showed that the proposed approach to an unreliable calibration result.
could predict the covariance of the hand-eye transformation To reduce the calibration errors in step-by-step calibration
with excellent precision. Equation AX = Y B is for solving methods, simultaneous calibration methods need to be studied.
another problem in which the relationship between the world To the best of our knowledge, only Yan et al. [28], Wu et al.
frame and the robot base frame is unknown in addition to the [29], [30], and Ma et al. [31] have proposed several approaches
unknown hand-to-eye relationship. Ernst et al. [18] presented a for solving X, Y , Z , simultaneously. Yan et al. proposed a
method for simultaneous tool-flange and robot-world calibration closed form Degradation-Kronecker (D-K) method and a purely
using a least-squares approach. The experiments showed that nonlinear (PN) method toward calibrating a serial–parallel ma-
this method achieved more accurate results and needed less cal- nipulator. This research is a first attempt to propose methods
ibration stations than classical methods. Ha et al. [19] presented to address the issue for the AXB = Y CZ problem. Wu et
a fast and numerically robust local optimization algorithm for al. presented a quaternion-based closed form method and an
the two-frame sensor calibration objective function and a two- iterative method for the simultaneous calibration of a coopera-
phase stochastic geometric optimization algorithm for finding a tive dual-robot system. For both Yan’s and Wu’s methods, one
stochastic global minimizer based on local optimum. Li et al. common problem is that the iteration progress is time consuming
[20] presented a probabilistic method to simultaneously solve because it needs to carry out the least squares of all samples and
for X and Y without a priori knowledge of the correspondence calculate the inverse matrices in each iteration. Furthermore,
between the streams of A and B. the accuracy of their calibration results need to be improved,
The calibration of a dual-robot systems mainly involves kine- too. Ma et al. proposed two probabilistic approaches and a
matic parameters calibration [21], and flange-sensor, base–base, hybrid approach that can solve the AXB = Y CZ problem
flange-tool transformation matrices calibrations. In addition to without a priori knowledge of the temporal correspondence of
kinematic parameters calibration, other calibrations of dual- the data. But the probabilistic methods deteriorate quickly as
robot systems can be concluded as the problem of solving the noise level rises up and can only be used in no noise or
AXB = Y CZ, where X, Y , Z represents unknown flange- small noise conditions. The hybrid approach combines Yan’s
sensor, base–base, and flange-tool transformation matrices to or Wu’s iterative approaches with the proposed probabilistic
be solved and A, B, C represent known matrices that can approaches to mitigate the influence of noise. But, meanwhile,
be obtained from the robot controllers and the sensor. Many the hybrid approach also introduces the additional computation
research works have been carried out on calibrating parts of cost of Yan’s or Wu’s approaches and makes the solving process
X, Y , Z. Yu et al. [22] proposed a method for multirobots more time consuming.

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
1174 IEEE TRANSACTIONS ON ROBOTICS, VOL. 37, NO. 4, AUGUST 2021

is an obvious observation from Fig. 1, where X, Y , Z represent

the unknown constant homogeneous transformation matrices
from {E 1 } to {S}, from {O 1 } to {O 2 } , and from {E 2 } to
{T }, respectively. Symbols A, B, C represent the given varying
homogeneous transformation matrices from {O 1 } to {E 1 } ,
{S} to {T }, and {O 2 } to {E 2 }, respectively. A homogeneous
transformation matrix T can be represented as
 
R T tT
T = (2)
0 1
where RT represents a 3 × 3 rotation matrix and tT represents
a 3×1 translation vector. Equation (1) can be decomposed into
rotation and translation parts
Fig. 2. Block diagram to show the organization of this article. RA RX RB = RY RC RZ (3)
RA RX tB + RA tX + tA = RY RC tZ + RY tC + tY . (4)
B. Contribution
In the calibration process, the two robots need to change
In this article, a novel approach for simultaneously solving their attitudes to obtain n sets of homogeneous transformation
the unknown matrices X, Y , Z of equation AXB = Y CZ matrices Ai , B i , C i (i = 1, 2, . . . , n). Then, (3) and (4) can be
are proposed, which is composed of a closed form method based extended to
on the Kronecker product and an iterative method, which con-
verts solving a nonlinear problem into solving an optimization R A i R X R Bi = R Y R C i R Z (5)
problem of a strictly convex function. The result of the closed RAi RX tBi + RAi tX + tAi = RY RCi tZ + RY tCi + tY
form method is not sufficiently accurate. So, it is used as the (6)
initial value of the iterative method to improve the convergence
efficiency and stability of iteration. The proposed approach have where i = 1, 2, . . . , n, and RAi , RBi , RCi , tAi , tBi , tCi repre-
two main advantages. sent rotation matrices and translation vectors of the ith group of
1) High Accuracy: The proposed approach can simultane- robot attitudes during the calibration process. Thus, solving (1)
ously solve the rotational and translational components of is equivalent to solving (5) and (6) for RX , RY , RZ , tX , tY , tZ .
X, Y , Z and avoid error transformation and accumulation in
traditional calibration approaches. In test experiments, the pro- B. Closed Form Method
posed method also exhibits superiority of calibration accuracy 1) Solution for Rotational Components: This section
to other simultaneous calibration methods under different noise presents a closed form method to solve (5) for the unknown
levels and initial estimations. rotational components RX , RY , RZ . A vec(•) operator can be
2) High Efficiency: The closed form method can be used applied on both sides of (5) to obtain
to quickly get a good initial value for the iterative method.
vec (RA RX RB ) = vec (RY RC RZ ) (7)
Moreover, the iterative method does not need to calculate the
gradient of all samples and the inverse matrices in each iteration, where vec(•) is an operator which reorders the entries of an a
so that the computational efficiency can be improved. by b matrix V to a column vector, i.e.,
The remainder is organized as shown in Fig. 2. In Section II,
vec (V ) = [v11 , v21 , . . . , vn1 , v12 , v22 , . . . , vab ]T . (8)
the calibration problem is modeled and formulated, then the
closed form method and the iterative method are presented. The following relationship holds:
In Section III, the experiments are carried out to evaluate the  
vec (RA RX RB ) = RTB ⊗ RA vec (RX ) (9)
performance of the proposed method. Section IV concludes this  
article. vec (RY RC RZ ) = RTZ ⊗ RY vec (RC ) (10)
where the symbol ⊗ represents the Kronecker product. It should
II. CALIBRATION METHOD MODELING AND SOLUTION be noted that there is a strong coupling between RY and RZ
A. Problem Formulation in (10). To avoid dealing with solving a complicated nonlinear
problem, it can be defined as
As shown in Fig. 1, {O 1 }, {E 1 } ,{S}, {O 2 } ,{E 2 }, and {T }
respectively represent the basic coordinate system of robot 1, the M AB = RTB ⊗ RA |9×9 ,
end-flange coordinate system of robot 1, the sensor coordinate mX = vec (RX ) |9×1 (11)
system, the basic coordinate system of robot 2, the end-flange ⎡ ⎤
coordinate system of robot 2, and the target coordinate system. vec(RC )T 0
The equation ⎢ .. ⎥
M =⎣C . ⎦
AXB = Y CZ (1) 0 vec(RC )T 9×81

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: SIMULTANEOUS CALIBRATION OF MULTICOORDINATES FOR A DUAL-ROBOT SYSTEM 1175

 
mY Z = vec RTZ ⊗ RY |81×1 (12) and the solution of (17) can be determined combining with the
orthogonal constraint of the rotation matrices in (18).
where •|a×b means that the matrix has a rows and bcolumns. Then, solving mXY Z is equivalent to solving the following
And (7) is equivalent to objective function:
T
[M AB , −M C ] · mTX , mTY Z = 0. (13) T
min f (mXY Z ) = mTXY Z M̃ ABC M̃ ABC mXY Z
Then, define √
s.t. mXY Z  = 2 3. (20)
M ABC = [M AB , −M C ] |9×90 , T
Note that M̃ ABC M̃ ABC is a real symmetric matrix. There-
T
mXY Z = mTX , mTY Z |90×1 (14) fore, a Rayleigh Quotient function [32] is built as
T
and (13) is equivalent to mTXY Z M̃ ABC M̃ ABC mXY Z f (mXY Z )
R (mXY Z ) = =
M ABC mXY Z = 0. (15)
mXY Z mXY Z
T
mXY Z 2
f (mXY Z )
Move the robots to a series of different attitudes. Then, n = . (21)
sets of homogeneous transformation matrices Ai , B i , C i (i = 12
1, 2, . . . , n) are obtained. Define According to the min–max theorem, the Rayleigh Quotient
function R(mXY Z ) satisfies
T
M̃ ABC = M TABC,1 , M TABC,2 , . . . , M TABC,n |9n×90
λmin ≤ R (mXY Z ) ≤ λmax (22)
(16)
and (15) can be extended to where λmin and λmax represent the minimum and the maximum
T
M̃ ABC mXY Z = 0. (17) eigenvalues of M̃ ABC M̃ ABC . Notice when ΔM̃ ABC = 0,
λmin = 0. From (22), there exists
Here, solving the nonlinear problem (5) has been converted T
to solving the linear problem (17). When RX , RY , RZ are min f (mXY Z ) = mTXY Z M̃ ABC M̃ ABC mXY Z = 12λmin
orthogonal matrices, there exists (23)
where mXY Z is the min solution of the function f . Then, substi-
mTXY Z mXY Z = mXY Z 2 = 3 + 3 · 3 = 12. (18) T
tute the singular value decomposition (SVD) of M̃ ABC M̃ ABC
Equation (18) is a necessary condition for the orthogonaliza- into (23) and obtain
tion of RX , RY , RZ and it is added to (17) as a constraint to mTXY Z V ABC ΣTABC U TABC U ABC ΣABC V TABC mXY Z
obtain
12
M̃ ABC mXY Z = √0 mTXY Z V ABC ΣTABC ΣABC V TABC mXY Z
(19) = = λmin (24)
st. mXY Z  = 2 3. 12
In practical cases, the noise of the sample data needs to be where ΣABC is a 9 n×90 diagonal matrix, and its diagonal ele-
considered. Therefore, M̃ ABC can be decomposed into T
ments are the eigenvalues of M̃ ABC M̃ ABC in descending or-
idea der, and the other elements are zero. U ABC is a 9n × 9n orthog-
M̃ ABC = M̃ ABC + ΔM̃ ABC
onal matrix. V ABC is a 90×90 orthogonal matrix, and each of
idea T
where M̃ ABC represents the ideal part of M̃ ABC without noise its columns is composed of the eigenvectors of M̃ ABC M̃ ABC
and ΔM̃ ABC represents the disturbance to M̃ ABC due to noise. corresponding to the arrangement of eigenvalues in ΣABC .
The solvability of (19) is analyzed in following two cases. The unit eigenvector corresponding to the minimum eigen-
T
value of M̃ ABC M̃ ABC can be defined as v min . Now, define
Case 1: ΔM̃ ABC = 0.
x = V TABC mXY Z . Then, (24) can be rewritten as
In this case, when n < 10 , M̃ ABC is not a full column rank ⎡ ⎤
λmax . . . 0
matrix, therefore (17) has infinite solutions. When both the con- ⎢ .. . . .. ⎥
ditions of n = 10 and rank(M̃ ABC ) = 90 are satisfied, (17) ⎢ . . . ⎥
⎢
T ⎢
⎥
has a unique zero solution. When both the conditions of n ≥ 11 2
ΣABC x = x ⎢ 0 . . . λmin ⎥
⎥ x = 12λmin . (25)
and rank(M̃ ABC ) = 90 are satisfied, (17) is an overdetermined ⎢ .. . . .. ⎥
⎣ . . . ⎦
equation, and it has a least squares solution in addition to the
0 ... 0
zero solution
√
idea Solve (25) and obtain x = [ 0 0 . . . 2 3 ]T . Then, mXY Z
Case 2: ΔM̃ ABC = 0 and M̃ ABC = M̃ ABC .
can be represented as
In this case, there always exists rank(M̃ ABC ) ≤ 89, so √
mXY Z = V ABC x = 2 3v min . (26)
that M̃ ABC is never a full column rank matrix, and the so- √
lution of (17) makes up the null space of M̃ ABC . When From (26), it can be noted that mXY Z is 2 3 times the
rank(M̃ ABC ) = 89, the null space of M̃ ABC is a 1-D space, unit eigenvector corresponding to the minimum eigenvalue of

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
1176 IEEE TRANSACTIONS ON ROBOTICS, VOL. 37, NO. 4, AUGUST 2021

T be obtained by orthogonalizing the rotation matrix with the im-

M̃ ABC M̃ ABC , therefore mXY Z can be solved directly from
T proved Gram–Schmidt algorithm proposed in the literature [33].
M̃ ABC M̃ ABC . According to (11)–(14), the relationships be- The closed form method is fast in calculation because it does
tween RX , RY , RZ and mXY Z are not require multiple iterative computations, but only needs one
RX = unvec (mXY Z |1:9 ) , calculation to get the solution. But there are two significant
 T disadvantages associated with the closed form method:
RZ ⊗ RTY = unvec (mXY Z |10:90 ) (27) 1) The closed form method simply treats the nonlinearly
constrained problem as an linearly unconstrained one, the
where unvec(•) is the opposite operator of vec(•), which con-
accuracy of the solution is reduced.
vert a column vector to a square matrix. Note that RX is easy
2) The rotational components and the translational compo-
to be obtained from (27), but RY and RZ are difficult to be
nents are calculated in sequence. That means the error of
calculated by the second equation of (27) because there is a
rotational components will be introduced into the calcu-
strong coupling between them. To calculate RY and RZ without
lation of the translational components.
solving the nonlinear problem of the second equation of (27),
Therefore, the closed form method is more suitable for quickly
(3) can be rewritten as
calculating an initial estimation for an iterative method to im-
RTA RY RC = RX RB RTZ prove its stability of convergence and efficiency of computation.
In the following section, an iterative method is introduced for
RC RZ RTB = RTY RA RX (28) simultaneously calculating the rotational and translational com-
where RY and RZ are equivalent to RX in (3), so that the ponents.
above process can also be applied to solving for RY and RZ ,
respectively. C. Iterative Method
2) Solution for Translational Components: Equation (4) can 1) Objective Function Establishment: In this section, an it-
be rewritten as erative method will be presented to solve (5) and (6) simultane-
Jt = b (29) ously. For n sets of dual-robot attitudes, solving for the rotational
and translational components is equivalent to solving for the
where J = [RA −I −RY RC ], t = [tTX tTY tTZ ]T , b = following objective function:
RY tC − tA − RA RX tB . For nsets of dual-robot attitudes,
there exists min g (RX , RY , RZ , tX , tY , tZ )
⎛ ⎞
J˜t = b̃ (30) RAi RX RBi − RY RCi RZ 2F
1 n
⎜ + R R t + R t + t ⎟
where J˜ = [ J 1 J 2 . . . J n ] , b̃ = [ b1 b2 . . . bn ] . And the
T T = ⎝ A i X Bi Ai X Ai ⎠
n i=1
translational component t can be solved by −RY RCi tZ − RY tCi − tY 2
 T −1 T
t = J˜ J˜ J˜ b̃. (31) s.t. RX RTX = I, RY RTY = I, RZ RTZ = I (33)
where RAi RX RBi −RY RCi RZ 2F (i=1, . . ., n) are
It is important to note that (5) is still satisfied when any two
residual error items of (5), and RAi RX tBi + RAi tX +
of RX , RY , RZ multiplying itself by −1, i.e.,
tAi − RY RCi tZ − RY tCi − tY 2 (i = 1, . . . , n) are residual
RAi (−RX ) RBi = (−RY ) RCi RZ error items of (6). The constraints mean that RX , RY , RZ are
all orthogonal matrices. But it is difficult to find the optimal
RAi RX RBi = (−RY ) RCi (−RZ )
solution directly under the constraints, therefore the Lagrangian
RAi (−RX ) RBi = RY RCi (−RZ ) . (32) relaxation method [34] is used to convert the minimization
problem with constraints (33) to that without constraints
But when one of the situations in (32) occurs, the translational
components will be far from true value, therefore (1) will be min g (RX , RY , RZ , tX , tY , tZ )
false. To avoid the situations in (32) occur, the residual error e ⎛ ⎞
can be defined as μ1 RAi RX RBi − RY RCi RZ 2F
⎜ + μ2 RAi RX tBi + RAi tX + tAi ⎟
1 ⎜ ⎟
n
e = AXB − Y CZF ⎜ − RY RCi tZ − RY tCi − tY 2 ⎟
= ⎜  2  2 ⎟
n i=1 ⎜ ⎟
where  • F represents the Frobenius’ norm of a matrix. ⎝ + μ3 RX RTX − I F + μ4 RY RTY − I F ⎠
 2
Change the signs of RX , RY , RZ to find the right combination + μ5 RZ RTZ − I F
of rotational components that can minimize the residual e. Then, (34)
the translational components according to the right rotational
components can be calculated by (31). where RX RTX − I2F , RY RTY − I2F , and RZ RTZ − I2F
Due to the inevitable errors of inputs Ai , B i , C i in the are penalty terms that force the rotational components
actual situation, the rotation matrices obtained by the closed RX , RY , RZ to be orthogonal matrices. Symbols
form method are not strictly orthonormal matrix. Therefore, μ1 , μ2 , μ3 , μ4 , μ5 are weight coefficients. The relaxation of
the orthogonalization of the rotation matrix is necessary. In this the orthogonal constraints can lead the computation faster and
article, strictly orthogonal rotation matrices RX , RY , RZ can stable, but it can also cause the final solution of RX , RY , RZ

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: SIMULTANEOUS CALIBRATION OF MULTICOORDINATES FOR A DUAL-ROBOT SYSTEM 1177

not satisfying the orthogonality strictly. Therefore, it is of (36) at V t . For SVRG, two important things need to be deter-
necessary to normalize the solution to make RX , RY , RZ mined in advance: the iteration step sizes ηt and the expression
strictly orthogonal. To simplify the expression, the residual of the gradient ∇V t g(V t ).
error items of (34) can be defined as The selection of step sizes ηt has great impact on the stability
⎧ and efficiency of the iteration process. In practice, a diminishing
⎪
⎪ E 1 = R A i R X R Bi − R Y R C i R Z
⎨ step size or a best-tuned fixed step size which is very difficult
e2 =RAi RX tBi + RAi tX + tAi − RY RCi tZ
. to be found is usually used. Neither of these two approaches are
⎪
⎪ −RY tCi − tY
⎩ efficient. To find appropriate ηt , an algorithm named SVRG-BB
E 3 = RX RTX − I, E 4 = RY RTY − I, E 5 =RZ RTZ − I
is proposed by Tan et al. [37]. It has been proved that SVRG-BB
(35)
converges linearly for strongly convex objective functions, and
Substituting (35) into (34), the objective function can be the performance of convergence rate and computational effi-
rewritten as ciency is comparable to and sometimes even better than SVRG
with best-tuned step sizes.
min g (RX , RY , RZ , tX , tY , tZ )
But in [37], the author only considers the case where the
1 
n
independent variable is a scalar or a vector. However, in this
= μ1 E 1 2F + μ2 e2 2 + μ3 E 3 2F article, the independent variable V t is a 3×12 matrix, so that the
n i=1
 expression of ηt is no longer applicable. To handle this problem,
+ μ4 E 4 2F + μ5 E 5 2F we modify the expression of ηt to

     
1
n
μ1 tr E T1 E 1 + μ2 eT2 e2 + μ3 tr E T3 E 3 ηt = V t − V t−1 2F (V t − V t−1 )T (∇V t g (V t )
= 
n i=1 +μ4 tr E T4 E 4 + μ5 tr E T5 E 5
− ∇V t−1 g (V t−1 ))F . (38)
1 
n
According to the matrix derivation rule, the gradient ∇V g(V )
= gi (RX , RY , RZ , tX , tY , tZ ) (36)
n i=1
of the objective function (36) is derived as (51) in Appendix A.
The algorithm flow of SVRG to solve (36) is described in Table V
where tr(•) represents the trace of a matrix. It is important to in Appendix B. The result of the closed form method is taken as
note that the Frobenius’ norm and the 2-norm are both strictly the initial value Ṽ 0 of iteration, and the other input parameters
convex functions of their respective arguments [35]. Hence, the include weight coefficients μ1 , μ2 , μ3 , μ4 , μ5 , update frequency
objective function (36), which is the sum of strictly convex m, the initial step size η0 , and maximum number of iterations o
functions, is obviously still a strictly convex function. Therefore, are given by the user. The optimal value V ∗ can be solved with
the nonlinear problem (1) is converted to a strictly convex the proposed iterative method.
optimization problem (36). The penalty terms can make the rotation matrices of the
The objective function (36) can achieve high calibration ac- result tend to be orthogonalized, but the rotation matrices will
curacy, because it considers both the residual error terms and not be strictly orthogonal due to the relaxation. Therefore, the
the penalty terms. The residual error terms can simultaneously same orthogonalization method used in Section II-B can also
optimize the rotational and translational components without be used to normalize the rotation matrices in the results of
error accumulation and amplification. And the penalty terms iterative method to obtain strictly orthogonal rotation matrices
can keep the rotation matrices orthogonal or nearly orthogonal. RX , RY , RZ .
2) Iterative Algorithms: For solving the strictly convex ob-
jective function (36), many convex optimization algorithms can
D. Solvability Analysis
be referenced, such as gradient descent algorithm (GD) and its
variants stochastic gradient descent algorithm (SGD), stochastic It is known that a homogeneous transformation matrix has six
variance reduced gradient algorithm (SVRG) [36], and so on. independent variables, corresponding to 3 DOF (degrees of free-
GD is a basic convex optimization algorithm that can be used dom) of rotation and 3 DOF of translation. For solving the prob-
to solve the strictly convex problem like (36). But it needs to lem AXB = Y CZ, there are three unknown homogeneous
calculate the gradients of all samples at each iteration, leading transformation matrices X, Y , Z that contain 18 independent
to high computational burden. To improve the calculation ef- variables need to be solved. But a single set of robot attitudes
ficiency, SVRG is derived from GD, which utilizes a variance can only provide six independent equations. Therefore, at least
reduction technique. It has been proved that SVRG has high three sets of robot attitudes are needed to solve 18 independent
computational efficiency and good iteration stability, and it will variables of the problem (1) in theory.
be used in this article. A typical iteration of GD for solving (36) It is important to be noted that even if the number of robot
is represented as attitudes is more than 3, it may still not be sufficient to uniquely
determine X, Y , Z. The necessary and sufficient condition of
V t+1 = V t − ηt ∇V t g (V t ) (37)
the unique solution in robot-exterior calibration problems has
where V t = [RXt , RYt , RZt , tXt , tYt , tZt ] represents the been proved in [29] and [38], which can be summed up as:
3×12 independent variable matrix at the tth iteration, ηt > 0 Condition of uniqueness: For a minimum of three sets of robot
represents the step size, and ∇V t g(V t ) represents the gradient attitudes, the necessary and sufficient conditions for a unique

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
1178 IEEE TRANSACTIONS ON ROBOTICS, VOL. 37, NO. 4, AUGUST 2021

solution of X, Y , Z is that the interstation rotation axes are not

collinear for different pairs of robot attitudes.
Let us take RA as an example to explain this condition.
For any three rotation matrices RAi , RAj , RAk chosen from
all robot attitudes {RA1 , RA2 , . . . , RAn }, the interstation rota-
tions between RAi and RAj , RAi and RAk can be respectively
defined as
 
Rot kAij , αij = RAi RTAj , Rot (kAik , αik ) = RAi RTAk
(39)
where Rot(k, α) represents a rotation transformation of angle
α around axis k. Symbols kAij and kAik represent interstation
rotation axes between RAi and RAj , RAi and RAk , respec-
tively. Symbols αij and αik represent the corresponding rotation Fig. 3. Schematic diagram shows the experiment setup consisting of two ABB
IRB 1600 10/1.45 robots, a sensor and a calibration target to solve the dual-robot
angles. system calibration problem.
When n ≥ 3, the necessary and sufficient condition for a
unique solution of X, Y , Z
  TABLE I
k A × k A  = 0 (40) SETTINGS OF THE TRANSFORMATION MATRICES FOR THE DUAL-ROBOT
ij jk
SYSTEM IN THE SIMULATION EXPERIMENTS
are not at the same time true for all RAi , RAj , RAk chosen
from {RA1 , RA2 , . . . , RAn }. Note that, if (40) is true for only
parts of the sample data, the problem still has a unique solution,
because the other sample data are enough to make the problem
overdetermined. But the equations Ai XB i = Y C i Z corre-
sponding to these sample data are linear correlation, which will
not contribute to the solution. In order to avoid invalid robot
attitudes, RAi , RAj , RAk should be checked for the following
conditions during planning of the robots’ calibration:
 
k A × k A 
ij jk

≥ τ {i, j, k |1 ≤ i, j, k ≤ n ∧ i, j, k ∈ N ∧ i = j = k } (41) 1, 1, 1, the update frequency m is set to 3, and the initial step
size η0 is set to 0.1. All experiments described in the following
where τ is a threshold value to avoid parallel rotation axis.
sections are written in MATLAB R2019b and run on a computer
Similarly, RB and RC should be satisfied:
  with an Intel i7-6700HQ CPU with 2.60 GHz and a 16 GB DDR4
k B × k B  RAM.
ij jk
1) Sample Data Generation: The sample data of A and C in
≥ τ {i, j, k |1 ≤ i, j, k ≤ n ∧ i, j, k ∈ N ∧ i = j = k } (42)
  the simulation experiments are generated by randomly changing
k C × k C  the robot attitudes within their respective workspaces. Invalid
ij jk
robot attitudes, where the calibration target on robot 2 is not
≥ τ {i, j, k |1 ≤ i, j, k ≤ n ∧ i, j, k ∈ N ∧ i = j = k } . within the visual range of sensor on robot 1, should be deleted.
(43) Then, the rotation matrices of all samples are substituted to
If any one of conditions (41)–(43) are not satisfied for conditions (41)–(43) to ensure that there is no linear correlation
RAi , RAj , RAk , the operator should delete one group of cali- between them. Then, n (n = 20, 40, 60, . . . , 300) sets of data
bration parameters in groups i, j, k, or adjust the robot attitudes A0i (i = 1, 2, . . . n) and C 0i (i = 1, 2, . . . n) can be obtained.
corresponding to i, j, k until all calibration parameters meet the Data B 0i (i = 1, 2, . . . n) can be calculated by applying (1). To
conditions (41)–(43). simulate the actual calibration process, uniform noise is added
to the ideal sample data, and obtain
III. EXPERIMENTAL RESULTS   
Ri = R0i Rot kr , rand 0θ̃
A. Simulation Experiments
       T
In the simulation experiments, we adopt two ABB IRB 1600 ti = t0i + rand −l˜l , rand −l˜l , rand −l˜l (44)
10/1.45 robots to construct a dual-robot system as shown in
Fig. 3. And the kinematics parameters of the ABB IRB 1600 where rand(low ∼ up) represents the operation of randomly
10/1.45 robot are given by the robot manufacturer. The settings selecting a value in the interval [low, up]. Symbol kr represents
of the transformation matrices for the dual-robot system are a randomly generated arbitrary unit vector. Symbols θ and l
shown in Table I. In the following experiments, the weight represent the upper bound of rotational noise and translational
coefficients μ1 , μ2 , μ3 , μ4 , μ5 are respectively set to 1, 10−5 , noise. The noise is divided into three levels named “low level,”

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: SIMULTANEOUS CALIBRATION OF MULTICOORDINATES FOR A DUAL-ROBOT SYSTEM 1179

TABLE II
NOISE LEVEL DIVISIONS

“medium level,” and “high level” according to the magnitudes

of boundary values as shown in Table II. After adding noise to
the initial sample data A0i , B 0i , and C 0i by applying (44), the
sample data Ai (i = 1, 2, . . . n), B i (i = 1, 2, . . . n), and C i (i =
1, 2, . . . n) for the simulation experiments are obtained.
2) Experiment on Different Number of Samples: The errors
of rotational and translational components in the simulation
experiments are defined as
   
 
eRs = R̄R̂ − I  , ets = t̄ − t̂
T
(45) Fig. 4. Average accuracy for the four methods under low-level noise by using
different numbers of samples.
where R̄ and R̂ represent the true value and calculated value of
the rotation matrices to be solved. Symbols t̄ and t̂ represent the
true value and the calculated value of the translation vectors to
be solved.
Four methods are used to conduct simulation experiments
at low, medium, and high noise level, respectively, and their
errors of rotational and translational components are compared
as the number of samples increases (repeat the experiments for
ten times at each number of sample data to get average errors).
The four methods are the proposed closed form method, the
step-by-step calibration method, the proposed iterative method,
and the calibration method proposed by Wu [29]. (Here, we do
not choose the simultaneous calibration methods proposed by
Yan [28] or Ma [31]. Because Ma’s method is aimed at handling
the simple data without a priori knowledge of the temporal
correspondence, which is different to the application scenarios in
this article. Moreover, Ma’s method is susceptible from noise,
which is not suitable for using at large noise scenarios. Yan’s
method is an earlier one, which has been compared with Wu’s
Fig. 5. Average accuracy for the four methods under medium-level noise by
method in [29]. It has shown that the accuracy of Yan’s method using different numbers of samples.
is worse than that of Wu’s method. Therefore, Wu’s method
is chosen for the comparative experiments, whose application
scenarios are closer to that of the method proposed in this article.) The experimental results are shown in Figs. 4 –6. The results
For the last two iteration methods, the number of iterations is show that the rotation and translation errors of the proposed
set to 100. iterative method are always the smallest among the four cali-
In the step-by-step calibration method, first, robot 2 is kept bration methods at low, medium, and high noise level, followed
fixed and the posture of robot 1 is changed to calculate the hand- by those of Wu’s method, and the errors of the step-by-step
eye matrix X using the AX = XB algorithm introduced in calibration method are the second largest. The proposed closed
[39] (notice that A and B in this equation have different meaning form method has the largest errors. Furthermore, the errors of
with the symbols mentioned above). Then, keep robot 1 fixed both rotational and translational components decrease with the
and change the posture of robot 2, n sets of data of B and C are increase of sample number for all four methods. The errors tend
generated. Y and Z are then calculated through the AX = Y B to be stable after the number of samples reaches 100. Therefore,
method provided by Li et al. [40]. taking more than 100 groups of sample data in the application
Wu proposed a simultaneous calibration method that could can reduce the impact of noise and avoid the overfitting problem
solve the rotational components of hand-eye, target-flange, and caused by insufficient samples.
robot–robot transformation matrices using an iterative method. The reason for the large errors of the proposed closed form
Then, the results of rotational components are put into (6) to method has been described in Section II-C. For the step-by-step
solve the translational components. calibration method, the errors of the rotational and translational

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
1180 IEEE TRANSACTIONS ON ROBOTICS, VOL. 37, NO. 4, AUGUST 2021

Fig. 7. Average accuracy for the proposed method and Wu’s method when
using different numbers of iterations.

Fig. 6. Average accuracy for the four methods under high-level noise by using
different numbers of samples.

components of Y and Z are larger than those of X. This

is due to the second step for solving the matrices Y and Z
introduced the errors of the first step for solving the hand-eye
matrix X. Errors are transmitted and accumulated in different
steps resulting in large final calibration errors. The errors of Wu’s
method are close but slightly larger than those of the proposed
Fig. 8. Average accuracy for the proposed method and Wu’s method by using
iterative method for RX , RY , RZ , tX , and tZ . However, different initial values.
for tY , the errors of Wu’s method are obviously larger than
those of the proposed iterative method. The reason is that the
TABLE III
proposed iterative method achieves simultaneous calculation COMPARISON OF TIME CONSUMPTION OF THREE METHODS
of translational and rotational components, thus avoiding the
problem of error transfer and accumulation when calculating
the translational components. For Wu’s method, the rotation
matrices are calculated at first. Then, the rotational components
are substituted into (6) to calculate the translational components.
That will cause the errors of the rotational components transform
into the translational components, resulting in large errors of the the optimal value stably. This is because Wu’s method takes the
translational components. Moreover, the norm of tY is much first-order approximation of Taylor expansion as the iterative
larger than that of tX and tZ , so the difference of the errors is coefficient, which introduces a truncation error. Besides, the
more obvious for tY . iterative coefficient in Wu’s method is calculated by the least
3) Experiment on Different Number of Iterations: In order square method with all the samples in each iteration, so it is
to compare the iteration errors and time consumption between susceptible to the noise of sample data.
the proposed iterative method and Wu’s method, 200 groups of The time consumption results of the closed form method and
sample data with medium-level noise are substituted into these two iterative methods are shown in Table III under the condition
two methods, respectively. Both methods use the same initial of 200 groups of samples and 100 iterations. Multithreading is
iteration value calculated by the proposed closed form method. not used to calculate the gradient in this experiment. The duration
The iterative experiment is repeated for ten times, and takes the of the proposed closed form method is only 2.11 s for calculating
average error at each iteration step. an initial value with 200 groups of samples. Besides, the run time
The experimental results are shown in Fig. 7. For both of the proposed iterative method is 12.69 s, and the run time of
rotational and translational components, the convergence rate Wu’s method is 126.67 s. Obviously, the proposed closed form
of the proposed iterative method is consistent with that of Wu’s method is the fastest because it does not need iteration. And the
method, both of them converge after about 30 iterations. But the iterative calculation efficiency of the proposed iterative method
errors of the proposed iterative method after convergence are is improved by 90% compared to that of Wu’s method. This is
smaller than those of Wu’s method. Moreover, after reaching because Wu’s method needs to carry out the least squares of all
the minimum value, Wu’s method fluctuates if the iteration samples and calculates the inverse matrices in each iteration,
continues, while the proposed iterative method can converge to which leads to the lower calculation efficiency. The proposed

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: SIMULTANEOUS CALIBRATION OF MULTICOORDINATES FOR A DUAL-ROBOT SYSTEM 1181

Fig. 10. Reachable spaces of the dual-robot system where the area inside the
Fig. 9. Practical experiments are carried out with an ABB IRB 1600 10/1.45 red dashed line is the working area of the dual-robot system.
robot, a KUKA KR16 R1610 robot, and a PowerScan Pro-5M 3D scanner. The
3-D scanner is mounted on the end-effector of the ABB robot, and a calibration
target board with maker points is fixed on the end-effector of the KUKA robot.
0.07 mm), a KUKA KR16 R1610 robot (six DOFs, pose re-
peatability 0.04 mm), and a PowerScan Pro-5M 3D scanner
(measurement accuracy ±0.02 mm, optimum measuring dis-
iterative method does not need to calculate the gradients of all tance 500 mm) are involved. The 3-D scanner is mounted on
samples in each iteration, but calculates the gradient by sampling the end-effector of the ABB IRB 1600 10/1.45 robot, and a
the samples. Meanwhile, the proposed iterative method does not calibration target board with 3 marker points is fixed on the
involve calculating the inverse matrices. Thus, the calculation end-effector of the KUKA KR16 R1610 robot. The 3-D scanner
efficiency of the proposed method is significantly improved. is used to collect the 3-D coordinates of the marker points in
4) Experiment on Different Initial Values: In order to com- the sensor coordinate system {S}. In the practical experiments,
pare the impact of the deviation of initial iteration values on the homogeneous transformation matrices from the robots’ end-
the proposed iterative method and Wu’s method, 200 groups of effectors to their bases (A and C) are read from their robot
samples with medium-level noise are generated, and the number controllers simultaneously. The homogeneous transformation
of iterations is set to 100. The initial iteration values of the matrices from the maker points to the 3-D scanner (B) is
rotational and translational components used in the experiment calculated and output from the measurement results of the 3-D
can be defined as scanner at the same time.
B. 1) Calibration Experiment: In order to generate the sam-
R0 = R̄Rot (kr , θ) , t0 = t̄ + lkt (46) ple data needed for the calibration of the dual robot system, the
where kr and kt represent randomly generated unit vectors. θ TCP of the two robots are established according to assembly
and l represent the amplitudes of rotational and translational relation at first. The TCP of robot 1 is set at the 3-D scanner’s
offsets. With the increasing of θ and l, the initial iteration values optimum measuring distance, and the TCP of robot 2 is set in
R0 and t0 will deviate further and further from the true value. the center of the calibration board. Then, the intersection area of
In this experiment, θincreases linearly from 0° to 2°, mean- the reachable spaces of both robots’ TCP is shown inside the red
while l increases linearly from 0 to 10 mm. In order to describe dashed line in Fig. 10, which can be considered as the working
the degree of deviation of the initial iteration value, a deviation area of the dual-robot system. 200 groups of robot postures
index d is defined which increases linearly with θ and l from 0 satisfying the conditions (41)–(43) are uniformly selected in
to 1. Fig. 8 shows the variation of the errors of the two methods the working area to ensure that the sample data can cover the
with the increase of d (repeat the experiment for 10 times for each working area of dual-robot system.
d to get the average errors). For the proposed iterative method, Note that, for any group of robot attitudes, the TCP points
both the errors of rotational and translational components almost of the two robots are kept overlapping. Then, the end poses
do not change with the increase of the initial value deviation. of both robots are adjusted so that the target board faces the
However, for Wu’s method, both the errors of rotational and 3-D scanner and is located around the 3-D scanner’s optimum
translational components increase with the increase of the initial measuring distance.
value deviation, and the error values also fluctuate greatly during For a point T p located in the frame {T }, there exists
this process. This experiment shows that the proposed iterative T
p = (Ai XB i )−1 Y C i Z T p.
method is relatively insensitive to the variation of initial value
compared with Wu’s method. So, the calibration errors at the ith group of robot attitudes
can be defined as
B. Practical Experiments E i = (Ai XB i )−1 Y C i Z − I. (47)
In addition to simulation experiments, practical experiments The errors of rotational and translational components at the
are also carried out to evaluate the proposed dual-robot cali- ith group of robot attitudes can be represented as
bration method. In following experiments, as shown in Fig. 9,  T 
 T 
an ABB IRB 1600 10/1.45 robot (six DOFs, pose repeatability eRpi = R̂Z RTCpi R̂Y RApi R̂X RBpi − I  ,
F

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
1182 IEEE TRANSACTIONS ON ROBOTICS, VOL. 37, NO. 4, AUGUST 2021

Fig. 11. Mean errors, error standard deviations, and paired-sampled t-test Fig. 12. Dual-robot cooperative measurement experiment is carried with a
results of the proposed calibration method and Wu’s method for both rotational standard ball mounted on the end of KUKA robot.
and translational components.



etpi = RApi R̂X tBpi + RApi t̂X + tApi − R̂Y RCpi t̂Z


− R̂Y tCpi − t̂Y  . (48)

Since the advantages of simultaneous calibration method over

step-by-step calibration method have been verified in [29], the Fig. 13. Selection of the postures of dual-robot system in the collaborative
method proposed in this article is no longer compared to the step- measurement experiment. (a) Selection of measurement regions. (b) Selection
of measurement views.
by-step calibration method, but only to Wu’s method. For both
methods, 150 sets of sample data are used to solve the unknown
matrices X, Y , Z, and the remaining 50 sets of sample data TABLE IV
DIAMETER ERRORS OF THE STANDARD BALL IN THE DUAL-ROBOT
are substituted into (48) to evaluate the calibration errors. The COOPERATIVE MEASUREMENT EXPERIMENT
mean errors and the error standard deviations of the calibration
errors are calculated, respectively. Paired-sampled t-test is used
to compare the performance of the proposed iterative method
with that of Wu’s method. The results for both rotational and
translational components are shown in Fig. 11. For the rotational
components, the mean error of the calibration results of the
proposed method is obviously smaller than that of Wu’s method.
p < 0.01 in the paired-sampled t-test, that means the calibration this experiment, the postures of the dual-robot system is gener-
accuracy of the proposed method is significantly better than that ated according to VDI/VDE 2634 Part 3 through the following
of Wu’s method at 99% confidence level. For the translational process:
components, the mean error of the calibration results of the 1) Selection of measurement regions. Five measuring regions
proposed method is smaller than that of Wu’s method. And p are selected from the working area of the dual-robot
< 0.05 in the paired-sampled t-test, which means the calibration system. The five regions are uniformly distributed in the
accuracy of the proposed method is better than that of Wu’s working area. For example, in the A-A section of the
method at 95% confidence level. working area in Fig. 10, the distribution of the five regions
2) Dual-Robot Cooperative Measurement Experiment: The is shown in Fig. 13(a).
calibration results of the proposed calibration method and Wu’s 2) Selection of measurement views. For any one of the five
method are respectively brought into the dual-robot system to regions, keep the standard ball in this region, and change
carry out a dual-robot cooperative measurement experiment. In the postures of two robots simultaneously to collect the
order to evaluate the accuracy of the cooperative measurement point cloud of the standard ball from five different views as
of the dual robot system while verifying the feasibility, the shown in Fig. 13(b). Note that the postures of both robots
measurement object is selected as a high precision standard use in validation should not be the same as the postures
single ball with known diameter ds = 50.8068 mm. The standard used in the calibration.
ball is fixed at the end of the KUKA robot as shown in Fig. 12. A measure point S pi in the frame {S} can be transformed
The dual-robot system can be seen as an optical multi-angle into the frame {E 2 } using the following equation:
measurement system in the cooperative measurement experi- E2
pi = C −1
i Y
−1
Ai X S p i . (49)
ment. VDI/VDE 2634 Part 3 is a guideline with high recognition
that is used as the operational guideline to evaluate the accuracy So that the measure points from different angles are pieced
of a multiple view system based on area scanning. Therefore, in together in the frame {E 2 }. The least-square method is used

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: SIMULTANEOUS CALIBRATION OF MULTICOORDINATES FOR A DUAL-ROBOT SYSTEM 1183

stage, the points measured from different angles are obviously

separated. Then, substitute the initial measurement points into
(49) and obtain the points in the frame {E 2 } , which has been
pieced into one sphere as shown in Fig. 14(b). Table IV shows
the diameter errors of the standard ball in the dual-robot coop-
erative measurement experiment. When the calibration result of
the proposed calibration method is applied, the mean error is
Fig. 14. Dual-robot cooperative measurement results of the standard ball. (a) 1.428 mm. And when the calibration result of Wu’s method is
Initial measurement data before coordinate transformation. (b) Measure data
after coordinate transformation. applied, the mean error reaches 2.303 mm. The result of this
experiment shows the superiority of the proposed calibration
method in accuracy.

IV. CONCLUSION
This article addressed the simultaneous hand–eye, base–base
and tool–flange calibration problem, which is essential for the
collaboration of dual-robot systems. This problem was formu-
lated as solving the matrix equation AXB = Y CZ. A closed
form method based on the Kronecker-product and an iterative
method, which converts a nonlinear problem to an optimization
problem of a strictly convex function were proposed. The result
of the closed form method was used as an initial estimation
for the iterative method to improve the speed and stability of
iterative convergence. The solvability of the calibration problem
was analyzed, and a series of conditions were presented to guide
the operators to select appropriate robot attitudes during the cali-
bration process. To show the better performance of the proposed
method compared to other traditional methods, a series of com-
prehensive simulation and practical experiments were carried
out, and the proposed method was proved to have superiority in
accuracy (at 99% confidence level for rotational components and
95% confidence level for translational components), efficiency
(increase by 90%), and stability. The method proposed in this
article was a general calibration method, which can be applied to
Fig. 15. Multirobot collaboration system. (a) Multirobot collaborative grind- various application scenarios of multirobot collaboration, such
ing of wind power blade. (b) Multirobot collaborative grinding of high-speed as multirobot collaborative grinding of wind power blade (length
railway body in white. >60 m) or high-speed railway body in white (length >22 m) as
shown in Fig. 15, multirobot collaborative assembly, and rivet-
to do spherical fitting and calculate the measured diameter dm ing of aircraft wall panel components, multirobot collaborative
of the standard ball. The above process is repeated five times welding of automotive body in white, etc. Because, in these
through moving the location of the ball to five different regions applications, the calibration of multirobot system is required.
in the overlapping region of the workspace of the two robots
[as shown in Fig. 13(a)], and the mean value of the diameter is
APPENDIX A
calculated. The diameter error for the kth (k = 1, . . . , 5) group
of data is defined as Gradient of the Objective Function (36)
edk = |ds − dmk | . (50) According to the matrix derivation rule, the gradient ∇V g(V )
of the objective function (36) is derived as (51).
Note that this error is caused by the accumulation of various
kinds of errors including the error of calibration algorithm, the
absolute positioning error of the robots (that can be reduced by APPENDIX B
some joint error calibration methods such as the methods in [41]
The Algorithm Flow of SVRG
and [42]), and the measurement error of the 3-D scanner and so
on. Substitute (38) into traditional SVRG algorithm to generate
The initial measurement points of the standard ball are shown the step size ηt . The algorithm flow of SVRG to solve (36) is
in Fig. 14(a), which are located in the frame {S}. At this described in Table V. Eq. (51) shown at the top of the next page.

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
1184 IEEE TRANSACTIONS ON ROBOTICS, VOL. 37, NO. 4, AUGUST 2021

 
1
n
∂g (V ) ∂g (V ) ∂g (V ) ∂g (V ) ∂g (V ) ∂g (V ) ∂g (V )
∇V g (V ) = = , , , , , = ∇V gi (V )
∂V ∂RX ∂RY ∂RZ ∂tX ∂tY ∂tZ n i=1
⎡    ⎤
μ1 RX RBi RTBi − RTAi RY RCi RZ RTBi + μ2 RXtBi tTBi + tX tTBi + RTAi tAi tTBi
⎢− RT R R t tT − RT R t tT − RT t tT + 2μ R RT R − R ⎥
⎢ 3 , ⎥
⎢ μ Ri R R RT iRT − R
A Y Ci Z B A i Y C i Bi A i Y  Bi X X X X
R R R T T
R ⎥
⎢ 1 
Y C i Z Z C A i X B i Z C  ⎥
⎢ i i
⎥
⎢ RY tCi tTCi + RY RCi tZ tTZ RCi − RAi RX tBi tTZ RCi − RAi RX tBi tTCi − RAi tX tTZ RCi
T T T
⎥
⎢ + μ ⎥
2⎢
n 2
−R t t T
− t t R
T T
− t t T
+ R t t R
T T
+ R R t t T
+ t t R
T T
+ t t T
⎥
= ⎢ A i X C i A i Z C i A i C i Y C i Z C i Y C i Z C i Y Z C i Y C i ⎥ .
n i=1 ⎢ + 2μ4 RY RTY RY − RY ,   
⎥
⎢ ⎥
⎢ μ1 RTCi RTY RY RCi RZ − RTCi RTY RAi RX RBi + 2μ5 RZ RTZ RZ − RZ , ⎥
⎢   ⎥
⎢μ2 tX + RX tBi + RTA tAi − RTA RY RCi tZ − RTA RY tCi − RTA tY , ⎥
⎢ i i i i ⎥
⎣μ2 (tY − RAi RX tBi − RAi tX − tAi + RY RCi tZ + RY tCi ) , ⎦
 T T 
μ2 RCi RY RY RCi tZ − RTCi RTY RAi RX tBi − RTCi RTY RAi tX − RTCi RTY tAi + RTCi RTY RY tCi + RTCi RTY tY 3×12
(51)

TABLE V [8] H. Xie, W.-L. Li, Z.-P. Yin, and H. Ding, “Variance-minimization iterative
ALGORITHM FLOW OF SVRG WITH A MODIFIED STEP SIZE matching method for free-form surfaces—Part I: Theory and method,”
IEEE Trans. Autom. Sci. Eng., vol. 16, no. 3, pp. 1181–1191, Jul. 2019.
[9] H. Xie, W.-L. Li, Z.-P. Yin, and H. Ding, “Variance-minimization iter-
ative matching method for free-form surfaces—Part II: Experiment and
analysis,” IEEE Trans. Autom. Sci. Eng., vol. 16, no. 3, pp. 1192–1204,
Jul. 2019.
[10] F. C. Park and B. J. Martin, “Robot sensor calibration: Solving AX =
XB on the euclidean group,” IEEE Trans. Robot. Autom., vol. 10, no. 5,
pp. 717–721, Oct. 1994.
[11] H. Zhuang, Z. S. Roth, and R. Sudhakar, “Simultaneous robot/world and
tool/flange calibration by solving homogeneous transformation equations
of the form AX = yB,” IEEE Trans. Robot. Autom., vol. 10, no. 4,
pp. 549–554, Aug. 1994.
[12] I. Ali, O. Suominen, A. Gotchev, and E. R. Moralesl, “Methods for
simultaneous robot-world-hand–eye calibration: A comparative study,”
Sensors, vol. 19, no. 1, 2019, Art. no. 2837.
[13] J. F. Li et al., “Calibration of a portable laser 3-D scanner used by a robot
and its use in measurement,” Opt. Eng., vol. 47, no. 1, pp. 1–8, 2008.
[14] Z. Zhao, “Hand-eye calibration using convex optimization,” in Proc. IEEE
Int. Conf. Robot. Autom., 2011, pp. 2947–2952.
[15] N. Andreff, “Robot hand-eye calibration using structure-from-motion,”
Int. J. Robot. Res., vol. 20, no. 3, pp. 228–248, 2001.
[16] Y. J. Ren, S. B. Yin, and J. G. Zhu, “Calibration technology in applica-
tion of robot-laser scanning system,” Opt. Eng., vol. 51, no. 11, 2012,
Art. no. 114204.
[17] H. Nguyen and Q. C. Pham, “On the covariance of x in AX = xB,” IEEE
Trans. Robot., vol. 34, no. 6, pp. 1651–1658, Dec. 2018.
[18] F. Ernst et al., “Non-orthogonal tool/flange and robot/world calibration,”
Int. J. Med. Robot. Comput. Assisted Surg., vol. 8, no. 4, pp. 407–420,
2012.
REFERENCES [19] J. Ha, D. Kang, and F. C. Park, “A stochastic global optimization algorithm
for the two-frame sensor calibration problem,” IEEE Trans. Ind. Electron.,
[1] A. Koubea and J. M. de Dios, Cooperative Robots and Sensor Networks, vol. 63, no. 4, pp. 2434–2446, Apr. 2016.
Ser. Studies in Computational Intelligence. Vienna, Austria: Springer [20] H. Y. Li, Q. L. Ma, T. M. Wang, and G. S. Chirikjian, “Simultaneous hand-
International Publishing, 2015, vol. 604, pp. 31–51. eye and robot-world calibration by solving the AX = XB problem without
[2] D. Gates, “Boeing’s struggle with 777 assembly robots adds to Everett correspondence,” IEEE Robot. Autom. Lett., vol. 1, no. 1, pp. 145–152,
production snarl the Seattle times,” Seattle Times, 2016. [Online]. Avail- Jan. 2015.
able: https://www.seattletimes.com/business/boeing-aerospace/boeings- [21] J. Hollerbach, W. Khalil, and M. Gautier, “Model identification,” in
struggle-with-777-assembly-robots-adds-to-everett-production-snarl Springer Handbook of Robotics. New York, NY, USA: Springer-Verlag,
[3] J. L. Xiao, S. L. Zhao, H. Guo, T. Huang, and B. Lin, “Research on the 2008.
collaborative machining method for dual-robot mirror milling,” Int. J. Adv. [22] Y. Qiao, Y. Chen, B. Chen, and J. M. Xie, “A novel calibration method for
Manuf. Technol., vol. 105, pp. 4071–4084, 2019. multi-robots system utilizing calibration model without nominal kinematic
[4] S. A. Webster, “Flexible manufacturing for a future with mixed materials,” parameters,” Precis. Eng.-J. Int. Soc. Precis. Eng., vol. 50, pp. 211–221,
Adv. Manuf. Org., vol. 154, pp. 89–96, 2015. 2017.
[5] D. H. Zhu et al., “Robotic grinding of complex components: A step towards [23] D. Zhao, Y. Bi, and Y. Ke, “Kinematic modeling and base frame calibration
efficient and intelligent machining—Challenges, solutions, and applica- of a dual-machine-based drilling and riveting system for aircraft panel
tions,” Robot. Comput.-Integr. Manuf., vol. 65, 2020, Art. no. 101908. assembly,” Int. J. Adv. Manuf. Technol., vol. 94, pp. 1873–1884, 2018.
[6] H. Xie, W. L. Li, D. H. Zhu, Z. P. Yin, and H. Ding, “A systematic model [24] Q. Zhu, X. Xie, C. Li, G. Xia, and Q. Liu, “Kinematic self-calibration
of machining error reduction in robotic grinding,” IEEE-ASME Trans. method for dual-manipulators based on optical axis constraint,” IEEE
Mechatronics, vol. 25, no. 6, pp. 2961–2972, Dec. 2020. Access, vol. 7, pp. 7768–7782, 2019.
[7] W. L. Li, H. Xie, G. Zhang, and S. J. Yan, “3-D shape matching of [25] Q. Zhu, X. Xie, and C. Li, “Dual manipulator system calibration based
a blade surface in robotic grinding applications,” IEEE/ASME Trans. on virtual constraints,” Bull. Pol. Acad. Sci.-Tech. Sci., vol. 67, no. 6,
Mechatronics, vol. 21, no. 5, pp. 2294–2306, Oct. 2016. pp. 1149–1159, 2019.

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: SIMULTANEOUS CALIBRATION OF MULTICOORDINATES FOR A DUAL-ROBOT SYSTEM 1185

[26] C. Ruan, X. Gu, Y. Li, G. Zhang, W. Wang, and Z. Hou, “Base frame cal- Cheng Jiang received the B.S. degree in mechani-
ibration for multi-robot cooperative grinding station by binocular vision,” cal design, manufacturing, and automation from the
in Proc. 2nd Int. Conf. Robot. Autom. Eng., 2017, pp. 115–120. Huazhong University of Science and Technology,
[27] J. Wang, W. Wang, C. Wu, S. Chen, J. Fu, and G. Lu, “A plane projection Wuhan, China, in 2016.
based method for base frame calibration of cooperative manipulators,” He is currently a Doctor Student with the Huazhong
IEEE Trans. Ind. Informat., vol. 15, no. 3, pp. 1688–1697, Mar. 2019. University of Science and Technology. His research
[28] S. J. Yan, S. K. Ong, and A. Y. Nee, “Registration of a hybrid robot using the interest include point cloud processing and robot
degradation-Kronecker method and a purely nonlinear method,” Robotica, machining.
vol. 34, no. 12, pp. 2729–2740, 2016.
[29] L. Wu, J. L. Wang, L. Qi, K. Y. Wu, H. L. Ren, and Q. H. Meng, “Simultane-
ous hand–eye, tool–flange, and robot–robot calibration for comanipulation
by solving the AXB = YCZ problem,” IEEE Trans. Robot., vol. 32, no. 2,
pp. 413–428, Apr. 2016.
[30] J. Wang, L. Wu, M.-Q.-H. Meng, and H. Ren, “Towards simultaneous
coordinate calibrations for cooperative multiple robots,” in Proc. IEEE/RSJ
Int. Conf. Intell. Robots Syst., 2014, pp. 410–415. Da-hu Zhu received the Ph.D. degree in mechanical
[31] Q. L. Ma, Z. Goh, S. Ruan, and G. S. Chirikjian, “Probabilistic approaches engineering from the Donghua University, Shanghai,
to the AXB = YCZ calibration problem in multi-robot systems,” Auton. China, in 2011.
Robot., vol. 42, no. 3, pp. 1497–1520, 2018. He conducted his postdoctoral research in HUST
[32] K. Neymeyr, “A geometric theory for preconditioned inverse iteration I: from 2011 to 2013. He is now an Associate Pro-
Extrema of the Rayleigh quotient,” Linear Alg. Appl., vol. 322, no. 1/3, fessor with the School of Automotive Engineering,
pp. 61–85, 2001. Wuhan University of Technology. His current re-
[33] H. Xie, C. T. Pang, W. L. Li, Y. H. Li, and Z. P. Yin, “Hand-eye calibration search interests include the NC machining and robotic
and its accuracy analysis in robotic grinding,” in Proc. IEEE Int. Conf. machining.
Autom. Sci. Eng., Aug. 2015, pp. 862–867.
[34] M. A. Bragin, P. B. Luh, J. H. Yan, N. P. Yu, X. Han, and G. A. Stern,
“Convergence of the surrogate Lagrangian relaxation method,” J. Optim.
Theory Appl., vol. 164, no. 1, pp. 173–201, 2015.
[35] J. Dattorro, Convex Optimization and Euclidean Distance Geometry. Palo
Alto, CA, USA: Meboo Publishing USA, 2010. He Xie received the B.S. degree in mechanical de-
[36] R. Johnson and T. Zhang, “Accelerating stochastic gradient descent using sign, manufacturing, and automation in 2013 from
predictive variance reduction,” in Proc. 26th Int. Conf. Neural Inf. Process. the Huazhong University of Science and Technology,
Syst., 2013, pp. 315–323. Wuhan, China, where he is currently working toward
[37] C. Tan, S. Ma, Y.-H. Dai, and Y. Qian, “Barzilai–Borwein step size for the Ph.D. degree in mechanical engineering.
stochastic gradient descent,” in Proc. 30th Annu. Conf. Neural Inf. Process. His current research interests include the au-
Syst., 2016, pp. 685–693. tomation of geometric measurement and geometric
[38] R. Y. Tsai and R. K. Lenz, “A new technique for fully autonomous and inspection.
efficient 3D robotics hand/eye calibration,” IEEE Trans. Robot. Autom.,
vol. 5, no. 3, pp. 345–358, Jun. 1989.
[39] R. Horaud and F. Dornaika, “Hand-eye calibration,” Int. J. Robot. Res.,
vol. 14, no. 3, pp. 195–210, 1995.
[40] A. Li, L. Wang, and D. Wu, “Simultaneous robot-world and hand-eye
calibration using dual-quaternions and kronecker product,” Int. J. Phys. Xing-jian Liu (Member, IEEE) received the B.E.
Sci., vol. 5, no. 10, pp. 1530–1536, 2010. degree in electronic information engineering and the
[41] R. He, Y. J. Zhao, S. N. Yang, and S. Z. Yang, “Kinematic-parameter Ph.D. degree in materials processing engineering
identification for serial-robot calibration based on POE formula,” IEEE from the Huazhong University of Science and Tech-
Trans. Robot., vol. 26, no. 3, pp. 411–422, Jun. 2010. nology, Wuhan, China, in 2013 and 2018, respec-
[42] W. L. Li, H. Xie, G. Zhang, and Z. P. Yin, “Hand-eye calibration in tively.
visually-guided robot grinding,” IEEE Trans. Cybern., vol. 46, no. 11, He is currently working as a Postdoctoral Fellow
pp. 2634–2642, Nov. 2016. with the Department of Mechanical and Industrial
Engineering and the Robotics Institute, University
Gang Wang received the B.S. degree in mechanical of Toronto, Toronto, Canada. His research interests
engineering from Chongqing University, Chongqing, include computer vision, 3-D sensing and metrology,
China, in 2015. He is currently working toward the and robotics.
doctoral degree with the Huazhong University of
Science and Technology, Wuhan, China.
His research interest include multirobot coopera-
tive measurement and machining.

Han Ding (Member, IEEE) received the Ph.D. degree

in mechatronic engineering from the Huazhong Uni-
versity of Science and Technology (HUST), Wuhan,
China, in 1989.
Wen-long Li (Member, IEEE) received the B.S. Supported by the Alexander von Humboldt Foun-
degree in mechanical engineering and automation dation, he was a Researcher with the University of
from Xi’an Jiaotong University, Xi’an, China, in Stuttgart, Germany, from 1993 to 1994. He has been
2004, the Ph.D. degree in mechatronic engineering a Professor with HUST, since 1997 and is currently
and the Postdoctoral research in control engineering the Director of the State Key Lab of Digital Man-
from the Huazhong University of Science and Tech- ufacturing Equipment and Technology. His research
nology (HUST), Wuhan, China, in 2010 and 2012, interests include robotics, multiaxis machining, and
respectively. control engineering.
He is currently a Professor with HUST. His cur- Dr. Ding was a “Cheung Kong" Chair Professor of Shanghai Jiao Tong
rent research interests include blade inspection and University from 2001 to 2006. He was elected a member of the Chinese Academy
robotic machining. of Sciences in 2013.

Authorized licensed use limited to: CHONGQING UNIVERSITY. Downloaded on January 18,2022 at 02:03:05 UTC from IEEE Xplore. Restrictions apply.
View publication stats