Learning Driven Coarse-to-Fine Articulated Robot Tracking

Christian Rauch1 , Vladimir Ivan1 , Timothy Hospedales1 , Jamie Shotton2 , Maurice Fallon3

Abstract— In this work we present an articulated tracking

approach for robotic manipulators, which relies only on visual
cues from colour and depth images to estimate the robot’s
state when interacting with or being occluded by its environ-
ment. We hypothesise that articulated model fitting approaches
can only achieve accurate tracking if subpixel-level accurate
correspondences between observed and estimated state can be
established. Previous work in this area has exclusively relied
on either discriminative depth information or colour edge cor-
respondences as tracking objective and required initialisation
from joint encoders. In this paper we propose a coarse-to-fine
articulated state estimator, which relies only on visual cues
from colour edges and learned depth keypoints, and which
is initialised from a robot state distribution predicted from
a depth image. We evaluate our approach on four RGB-D
sequences showing a KUKA LWR arm with a Schunk SDH2
hand interacting with its environment and demonstrate that this
combined keypoint and edge tracking objective can estimate the
palm position with an average error of 2.5cm without using any
joint encoder sensing. Fig. 1. Grasping in a cluttered environment. Top left: colour image,
top right: colourised depth image of the manipulator and objects in the
scene with the background removed. The combined tracking of keypoints
I. I NTRODUCTION (yellow/green, bottom left) and edges (blue, bottom right) enables precise
tracking (red) during a grasping task even when parts of the manipulator
Traditional robot manipulation requires a precisely mod- are occluded. While keypoints provide sparse but stable visual cues for the
elled articulated robot arm with accurate position and torque fingers, edges provide pixel-accurate estimation of the upper arm. No joint
sensing to execute trajectories with high precision. This encoder sensing was used here.
approach has been most successful in industrial automotive
manufacturing but typically does not use any exteroception.
In this work we focus on visually-driven manipulation where age learned discriminative information to fit kinematically
the scene is understood through visual object detection and plausible states to the observed data.
fitting and the articulated robot arm is tracked visually. Many Approaches such as [5] and [6] use 2D keypoint estima-
compliant robot arms suffer from structure bending and are tion to predict the 2D pixel location of joints but do not
not millimetre-precise while in some industrial scenarios leverage the kinematic and visual information provided by
manipulators are entirely devoid of sensing such as in nuclear the manufacturer’s 3D model of the robot. This kinematic
decommissioning [1]. In these scenarios the vision-only information is only learned indirectly from a large training
manipulator tracking would be useful. set and therefore needs to be explicitly enforced using an
We explore model-based articulated arm tracking based additional kinematic solver. Although keypoints can provide
entirely on RGB-D cameras passively detecting the arm. reliable constraints for the kinematic solver, they are only
The goal is to determine the configuration of the robot arm sparsely distributed. We therefore propose to use additional
model which best matches the observed state. One particular dense edge correspondences as a second tracking objective.
challenge is that a variety of different joint configurations These two objectives are visualised in Figure 1 for a cluttered
can lead to visually similar observations. We are motivated environment. While edges provide densely distributed pixel-
by the work of [2], [3], [4] (in the field of human body accurate correspondences, they are impaired by textured ob-
tracking) to develop model fitting approaches which lever- jects and are therefore unreliable as a sole tracking objective,
in which case keypoints provide more stable cues.
Christian Rauch is supported by Microsoft Research through its PhD Local optimisation algorithms leverage gradient informa-
Scholarship Programme. Maurice Fallon is supported by a Royal Society tion which, for kinematic models, can be easily obtained
University Research Fellowship. This research is supported by the Engineer-
ing and Physical Sciences Research Council (EPSRC) through the ORCA
by differentiating the forward kinematics of the articulated
Robotics Hub (EP/R026173/1). model. However, initialising such a local optimisation solely
1 Institute of Perception, Action and Behaviour, School of Informatics,
from visual observations is challenging due to the visual
University of Edinburgh, UK [email protected] ambiguity of shape symmetric robot links and the large
2 Microsoft, Cambridge, UK
3 Oxford Robotics Institute, Department of Engineering Science, Univer- range of possible joint motions. We therefore consider many
sity of Oxford, UK [email protected] possible candidates for initialising the optimisation. These
candidates are drawn from a coarse robot state distribution when mapping between a 2D keypoint to 3D pose by using a
that is predicted from a single depth image. Sampling from line of sight constraint and the camera intrinsics to constrain
this distribution allows us to consider many candidates and the optimisation state space.
select that one that provides the best visual cues. Due to their stability, we propose to rely on keypoint track-
In summary, we contribute: ing as the base objective. After initial optimisation, we then
1) a tracker initialisation strategy using a coarse joint switch to dense but pixel-level accurate edge correspondence
position distribution predicted from a depth image for accurate registration. Compared to our previous work
2) a combined tracking objective that uses stable and [13], which used pixel-wise depth image segmentation as
pixel-accurate cues from colour and depth images in a visual cues, the 2D keypoints proposed in this paper can be
single unified framework located behind occlusions (see Figure 1) while texture/colour
The combination of these stages makes our proposed tracking edges provide sharper contours than the imprecise edges in
approach independent from joint encoder sensing and con- typical depth images.
secutively refines the state from the initially sampled config- Kinematic optimisation: Model fitting approaches, such as
uration via keypoint tracking until the basin of convergence [9], [12], [4], rely on accurate models to find the optimal
for pixel-accurate edge correspondences is reached. state that is kinematically and visually plausible. While
We show that, while keypoints already provide a good global optimisation methods are less prone to local minima,
performance for tracking a manipulator, the additional inte- they are also more difficult to tune and are computationally
gration of edges can reduce the end-effector tracking error expensive. Local optimisation approaches on the other hand
to 2.5cm for grasping scenarios. are well established and make use of gradient information
to quickly converge to a minimum. We use the optimisation
II. R ELATED W ORK toolbox EXOTica [16], which provides a modular way to
A large corpus of work [7], [8], [9], [1] has investigated exchange solvers and objectives, to fit our robot model using
visual tracking for robotic manipulation in recent years. the keypoint and edge objectives. Inspired by [17], we first
Visual tracking approaches ought to be able to mitigate optimise the kinematic chain from the base of the robot to the
effects such as linkage elasticity or joint encoder inaccuracies palm or wrist, before optimising the smaller finger links. This
and ought to enable a more precise manipulation accuracy makes sure that the optimisation of fingers, which have less
and a more holistic representation of the manipulation scene, visual features and are more likely occluded, is initialised
including the manipulandum and obstacles in the scene. from a reasonable state.
Joint distribution prediction: Inspired by previous work
III. M ETHOD
on predicting the state of an articulated model from images
in [2], [10], we propose to predict a distribution over the A. Overview
articulated state space of a robot manipulator. Similar to To find kinematically plausible robot configurations which
[2], we represent this distribution as discretised bins. Instead match the observed depth image, we propose a coarse-to-
of training discrete state regressors for each of these bins, fine inverse kinematic optimisation in three stages (Figure 2).
we propose to directly sample from the distribution that First, we sample from a predicted distribution of joint values
is represented by these bins. Compared to the discrete to propose a set of possible initial configurations (Section III-
states provided by the retrieval forest in [10], our proposed C). These coarse samples are tested in the second stage to
sampling approach provides continuous interpolated samples select the sample which minimises the keypoint tracking
from the state space and hence also includes samples that are objective on the first image in a sequence. In the third
not exactly part of the training set. stage, we minimise the combined keypoint and edge objec-
Visual features: Different sparse and dense visual features tive (Section III-F) consecutively on a sequence of images
have been used in tracking literature to establish correspon- (Section III-G). All three stages use visual cues that we
dences between the observed and estimated state of a 3D extract from the depth image using a multitask convolutional
model. Early work in this area used dense features like neural network (Section III-B).
colour image edges [11], [1], [12] and depth images [9].
These correspondences are based on the local appearance B. Multitask Prediction
of the estimated state and change with each iteration of the Predicting the joint position distributions and keyoint
optimisation. This results in many local minima which can heatmaps is done in parallel on a depth image. Since these
be mitigated by introducing discriminative information [13]. tasks share depth image features, they are commonly trained
Sparse keypoint features, learned from data, are commonly in a multitask setup (Figure 3). In our architecture, we use a
used for human pose estimation [5], [6] and used to es- ResNet-34 [18] to extract 256 feature maps that are used
timate the skeleton configuration from 2D images. These by the task specific branches. As the type of a keypoint
approaches do not resolve 3D ambiguity, nor do they provide relates to the link it belongs to, we train an additional
the exact visual representation that is required for robotic segmentation task to segment the depth image into robot
grasping tasks. An additional 3D pose estimation stage links and background.
[14] can regress from these keypoint locations to 3D joint The segmentation and keypoint heatmaps provide infor-
coordinates. As proposed in [15], we resolve the ambiguity mation in the image space about pixels being occupied by
 0.4 interpolated PDF
samples

probability
0.3

0.2

0.1
prediction 0.0
−π −34 π −12 π −14 π 0 +14 π +12 π +34 π +π

keypoints joint position [rad]

1 2 3
(x,y,z) Fig. 4. Example distribution (orange) and samples (green) of a lower arm
(x,y,z) joint position, predicted and sampled from an image of the occluded bottle
sequence (Figure 8). The distribution shows two strong modes at 1.2rad and
−1.8rad since the link has a similar visual appearance at half rotations.

joints keypoints edges

Fig. 2. An overview of the sources of information extracted from an the network to predict a distribution of joint states. For each
observed RGB-D image pair. From left to right, they provide increasingly of the NJ = 15 joint positions, we place a 1D Gaussian
detailed visual cues for the tracking system. with σ = 0.1rad on the true joint position. This Gaussian
is then discretised in the value range [−π, π)rad into 60
segmentation
3x3x20

bins which results in a resolution of 6deg ≡ 0.1rad. All

Conv

Upscale 240
640
feature maps 20 320
discretised joint positions are serialised into a single vector
480
ResNet-34 15 R900×1 (15 × 60) and then reshaped into a matrix R15×60
keypoints
3x3x23

20 containing the score values of discretised joint positions.

Conv

256
Upscale 240
After prediction we can treat the scores of each joint as a
3x3x256

3x3x256

3x3x128

3x3x128

Upscale
23 320
ConvT

ConvT

ConvT

ConvT

FC pose probability distribution function (PDF) from which we can

AvgPool
FC joints
(15 x 60) sample joint states. Since we are sampling from a continuous
distribution, the resolution of the discretised bins does not
Fig. 3. Our approach uses multitask prediction to obtain segments, directly affect the sampled states, as long as it is small
keypoints and joint estimates from a single depth image. All three tasks enough to represent multiple modes of the joint positions.
use the same depth features extracted by a ResNet-34 [18].
The inference for all tasks takes 0.07s per image.
C. Sampling of Initial Configuration
a link or a 2D keypoint, respectively. The feature maps Robotic joints have a large range of possible joint po-
are therefore individually upscaled by 3 × 3 transposed sitions and configurations that are far away in joint space
convolutions to 128 task specific feature maps of a quarter can lead to the same global end-effector pose with very
of the original depth image resolution. For the segmentation similar local appearance. Trying to directly predict the joint
this is followed by a regular 2D 3 × 3 convolution and a position from a single depth image is therefore an ambiguous
softmax layer for providing the probabilities for the NL = 18 task. Although we provide a single-mode 1D Gaussian joint
robot links, the background and the object (20 classes in position distribution as training target, it is likely that a multi-
total). To reuse information about the location of links for the modal distribution is predicted for visually similar appearing
keypoint localisation, we concatenate the upscaled heatmap configurations of a link (Figure 4).
features with the segmentation features and apply 3 × 3 2D Since the mean or the strongest mode of such a distribution
convolution with a sigmoid activation. might not correspond to the observed link state, we propose
The 22 3D keypoints are manually placed on the surface to sample independently from each joint’s distribution to
of the 18 links. During training, they are transformed via the initialise the optimisation several times. To obtain samples
true state into the camera frame and projected onto the 2D from the PDF represented by the score bins, we linearly
image plane. 2D Gaussians with σ = 3px are centred on interpolate the cumulative sum of the bin scores, which
each of the 22 keypoint pixel locations to obtain the final provides the cumulative distribution function (CDF), and
heatmaps [5]. An additional background heatmap is created sample uniformly in U(0, 1) from the inverse CDF.
to represent the probability of a pixel not being assigned to
any keypoint. During prediction, we can only recover the D. Training
line of sight from camera origin to the 2D keypoint on the The segmentation, keypoint localisation and joint distri-
image plane. This ambiguity is resolved during optimisation bution prediction is trained on approximately 125000 syn-
of point-to-line distances in a subsequent stage. thetically rendered depth images in the manner described in
The third branch provides a joint state distribution to our previous work [13]. These synthetic images show the
initialise the optimisation, and the 6D robot pose as support. robot at different configurations sampled from a wide range
Since a regular regression of the joint state only provides of states where the palm is inside the camera frustum. As
the state vector itself without a confidence measure, we train proposed in [19], we randomly select one of 30 objects and
place the object at a random pose inside the hand to simulate provided by rendering the estimated state in the image frame.
interaction with a manipulandum. We do not discriminate We iterate through these edge pixels and assign them to
between these 30 objects, but treat them as a single object. the closest observed edge pixel if the angle between their
During training we minimise a weighted cross entropy normals is smaller than 8 degree, i.e. if they point roughly
for the segmentation task, the mean absolute error on the in the same direction. This is similar to the orthogonal
keypoint heatmaps and the mean squared error on the discre- line search proposed in [12]. The edge-to-edge association
tised joint scores. The cross entropy is weighted by median provides multiple P2L tasks per link, which are updated at
frequency balancing to increase the classification accuracy each iteration by rendering the new estimated state. To make
for smaller links such as the middle finger and finger tip. these objectives more robust, we reject keypoints with scores
smaller than 0.5 and reject edges pixels with more than 5cm
E. Tracking Objective
distance.
The observed robot state is provided by the colour and
depth images, and the predicted keypoints and joint position F. Resolving Robot State Ambiguity
distribution. The estimated robot state is initially provided by Each P2L task minimises the distance dv between the
the sampled configurations and thereafter from the optimisa- observed line of sight rays in the camera frame and their cor-
tion on consecutive image frames. The visual representation responding 3D points p in the link frame (after transforming
of the estimated state is obtained by rendering the link them into the same camera frame using the estimated joint
meshes at their estimated pose. configuration) with respect to the robot state q. The gradient
The objective for the optimisation is to minimise the of this task is derived using the chain rule:
distance between observed 2D keypoints and edges and  
their corresponding estimated visual 3D representation. Since ∂dv ∂p l ∂p
= · l− . (4)
the depth of these keypoints and edges cannot be fully ∂q ∂q klk2 ∂q
determined, e.g. a keypoint might be occluded, the objective ∂p
is formulated using the line of sight. ∂q is given by the kinematic Jacobian Jkin (q) ∈ R3×6+NJ ,
The line of sight l of a 2D keypoint or 2D edge pixel is i.e. by differentiation of forward kinematics, where
∂pi
the ray that passes the camera origin and the 2D point on the Ji,j = ∂q j
with i ∈ [1, 3] (task space φ) and j ∈ [1, 6 + NJ ]
image plane. These lines are defined in the camera frame and (robot state space q), and NJ the number of joints.
obtained via back-projection using the camera intrinsics. The The keypoint and edge correspondences provide  NT P2L
∂dv,t
start of this ray ls can be constrained using corresponding tasks for each of the NL links. The gradients ∂q and
depth information. The previously defined link keypoints directions (dv,t ) of these tasks (t ∈ [1, NT ]) are averaged
and the link meshes are transformed from the link to the per link l and objective type:
camera frame at each optimisation iteration and provide the
NT ,l
corresponding estimated 3D visual representation. 1 X ∂dv,t
Given the 3D line of sight l and a corresponding 3D point JP 2L,l = (5)
NT,l t ∂q
p, the projection of p on l, pv NT ,l

(ls − p) · l
 1 X
t = min 0, − (1) φP 2L,l = dv,t . (6)
klk2 NT,l t
pv = ls + tl (2) Since the tasks are shared among the keypoint (k) and edge
is the point on the line closest to p. The vector (e) objective, their relative contribution is weighted by a link
specific weight α ∈ {0, 1} and a global weight w ∈ {0, 1}:
dv = pv − p (3)
Jopt,l = αk JP 2L,l,k + we αe JP 2L,l,e (7)
points from the estimated 3D point to its corresponding
observed 3D point in the camera frame. dv is the tracking φl = αk φP 2L,l,k + we αe φP 2L,l,e . (8)
objective for keypoints and edges which is to be minimised. The weights α are used to switch the tracking objective
We will denote this objective as Point-to-Line (P2L) task. between keypoints (αk = 1, αe = 0) and edges (αk = 0,
1) Keypoints: For each predicted heatmap, we select the αe = 1) individually per link. The weight we is used to
pixel with the highest score and its associated depth reading globally add or remove the edge tracking objective.
to obtain the line of sight l. The corresponding 3D keypoint The distances are minimised iteratively with respect to q
is transformed from its local coordinate frame to the camera
frame via forward kinematics during the optimisation. The †
qi+1 = qi + Jopt φ (9)
P2L keypoint correspondences are established per observed
†
image frame and stay constant during optimisation. using the pseudo-inverse Jopt ∈ R6+NJ ×3NL of all stacked
2) Edge Pixels: Estimated edge pixels are related to Jopt and the stacked P2L distances φ ∈ R3NL ×1 .
their closest observed edge pixel by computing a distance Since the root link of the robot is rotational symmetric
transform on the Canny [20] edges of the observed colour and often not observed in the depth image, we use the true
image. The estimated edge pixels and its 3D coordinate are 6D camera pose and do not optimise these state variables.
G. Tracking Pipeline
The tracking operates on a continuous sequence of depth
and colour images. It is initialised once at the beginning by
sampling 50 configurations from the predicted distribution
(Section III-C) and selecting the state with the smallest key-
point objective (Section III-E.1), i.e. the forward kinematics
state with the smallest average Euclidean distance between
the 3D keypoints and their corresponding line of sight. The
optimisation is then initialised at each new image pair using
the previous solution and iterates for 10 iterations (0.37s).
The tracking objectives are switched at run time for each
link individually. A link switches from keypoint to edge
tracking (αk = 0, αe = 1), if all of its keypoint distances
are closer than 2cm (kdv k ≤ 0.02) and vice versa. Finger
links always use the keypoint objective.
We initially only track the arm and palm and switch to full
tracking when the upper links’ keypoint error is smaller than
2cm, and switch back to arm and palm tracking when this
error becomes larger than 3cm. This low and high threshold
have been chosen to minimise hysteresis.
Fig. 5. Sampling from the joint position distribution. Left: colour image
IV. E VALUATION of the observed scene, Middle: depth image from which we predict the
joint position distribution, Right: observed edges (blue) overlaid with the
We evaluate our tracking approach using a Kuka LWR4 contours of three sampled configurations (green, orange, yellow).
7 DOF arm with a Schunk SDH2 7 DOF end-effector,
which is observed by a fixed Asus Xtion PRO LIVE RGB-D
camera. For further details on the experimental setup we refer 3.1cm to 2.5cm, respectively. Although the occluded bottle
to [13]. We evaluate our tracking approach on four sequences sequence (Figure 8) shows an improved tracking perfor-
that show grasping of different objects and occlusions. mance of the forearm link, this is not propagated to the palm.
In the grasping behind occlusions sequence (Figure 9),
A. Sampling Robot States which is the most challenging of our sequences since it
First we will qualitatively evaluate the first stage of contains distractions of both types (manipulandum and oc-
our proposed pipeline, which proposes initial robot joint clusion), we are still able to track the palm with an average
configurations using the predicted distributions described position error of 4.5cm, which is less than half of the
in Section III-C. Figure 5 shows snapshots of two track- palm length (9.38cm). The keypoint-only baseline performs
ing sequences, each with three sampled configurations. For slightly better in this case.
these visualisations, we sampled 50 configurations from the In summary, our proposed tracking approach is able to
predicted distribution and automatically selected the three reliably track an occluded manipulator in grasping scenes,
configurations with the smallest average edge-to-edge dis- without making any assumptions on the presence of objects
tance. These configurations coarsely align with the observed or the availability of joint encoder readings. This solves a
state and demonstrate that the predicted distribution provides common problem of articulated tracking approaches, which
reasonable robot states to initialise the local optimisation. often need to be initialised from a known robot state. Our
approach is therefore more generally applicable to scenarios
B. Tracking
where direct access to the robot is not available.
We apply the proposed tracking approach on the four
sequences as described in Section III-G. To evaluate the V. C ONCLUSION
contribution of edge tracking, we apply tracking once with We presented a robotic manipulator tracking approach
we = 0 (keypoint-only objective) and we = 1 (combined that solely relies upon visual cues to initialise tracking. It
keypoint and edge objective) with the same sampled starting consecutively updates the estimated state using a combi-
state. Apart from we , we use the same configuration for nation of colour edge and depth keypoint correspondences.
all sequences. Figures 6 to 9 report the position tracking The proposed deep multitask network learns common depth
error against forward kinematics for a forearm link and the image features that can efficiently be used in parallel for
palm (fifth and ninth link in the kinematic chain), with two the coarse initialisation and the keypoint tracking objective.
snapshots of the sequence overlaid with contours of the state Dense colour image edges are then further used to refine
estimated using the combined keypoint and edge objective. the estimated state. Our approach only requires an accurate
By using edges as an additional objective, the average kinematic and visual model to generate training data and to
palm position error in the non-occluded grasping sequences provide the estimated visual representation during tracking.
(Figures 6 and 7) has reduced from 3.7cm to 2.7cm and No real robot data was required to train the network.
 forearm palm forearm palm
0.10 0.10
position error [m]

position error [m]

0.08 0.08

0.06 0.06

0.04 0.04

0.02 0.02

0.00 0.00
0 10 20 30 0 10 20 30 0 20 40 0 20 40
time [s] time [s] time [s] time [s]
Keypoints Keypoints + Edges Keypoints Keypoints + Edges

Fig. 6. Grasping box. Using the additional edge objective reduces average Fig. 8. Occluded bottle. Using additional edge objective reduces average
position error from 5cm to 3.1cm (forearm) and 3.7cm to 2.7cm (palm). position error for the forearm from 3.1cm to 2.1cm but increases the palm
position error from 2.6cm to 2.8cm.

forearm palm
0.10
forearm palm
position error [m]

0.08 0.15
position error [m]

0.06

0.04 0.10

0.02
0.05
0.00
0 20 40 0 20 40
time [s] time [s] 0.00
0 10 20 30 40 0 10 20 30 40
Keypoints Keypoints + Edges
time [s] time [s]
Keypoints Keypoints + Edges

Fig. 7. Grasping bottle. Using additional edge objective reduces average

position error from 2.7cm to 2.3cm (forearm) and 3.1cm to 2.5cm (palm).
Fig. 9. Grasping Alpen box behind occlusion. The additional edge objective
impairs tracking (average palm position error increased from 4.3cm to
4.5cm), when strong visual distractions are present.
We evaluated our approach on four sequences showing
the grasping of different objects and varying occlusions, and
found that the additional edge tracking objective improves
tracking of grasping scenes compared to only keypoint level correspondences.
tracking. Even in cases with strong distractions from a The proposed prediction of a joint position distribution
textured manipulandum and occlusions, which our previous provides samples that are sufficient to initialise tracking,
approach cannot handle, we are able to purely visually track but the discrete bin scores and their interpolated PDF are
the palm with a position error of less than half its size. predicted and sampled independently which makes this sam-
We note that we used the same tracking parameters for pling approach inefficient. The camera pose is currently not
the sequences with and without occlusions and recommend part of the predicted and sampled robot state distribution and
tuning of the combination of the objectives depending on is therefore assumed given and static. In future work we will
the expected amount of occlusions. In future work, we will investigate methods for sampling a holistic robot state that
investigate alternative dense correspondences like histogram can additionally be used during tracking to detect tracking
of oriented gradients (HOG) to provide more robust pixel- loss and reinitialise the tracker.
 R EFERENCES [11] I. Oikonomidis, N. Kyriazis, and A. A. Argyros, “Full DOF tracking of
a hand interacting with an object by modeling occlusions and physical
[1] V. Ortenzi, N. Marturi, R. Stolkin, J. A. Kuo, and M. Mistry, “Vision- constraints,” in 2011 International Conference on Computer Vision,
guided state estimation and control of robotic manipulators which lack Nov 2011, pp. 2088–2095.
proprioceptive sensors,” in 2016 IEEE/RSJ International Conference [12] E. Muoz, Y. Konishi, V. Murino, and A. D. Bue, “Fast 6D pose
on Intelligent Robots and Systems (IROS), Oct 2016, pp. 3567–3574. estimation for texture-less objects from a single RGB image,” in 2016
[2] T. Sharp, C. Keskin, D. Robertson, J. Taylor, J. Shotton, D. Kim, IEEE International Conference on Robotics and Automation (ICRA),
C. Rhemann, I. Leichter, A. Vinnikov, Y. Wei, D. Freedman, P. Kohli, May 2016, pp. 5623–5630.
E. Krupka, A. Fitzgibbon, and S. Izadi, “Accurate, robust, and [13] C. Rauch, T. Hospedales, J. Shotton, and M. Fallon, “Visual articulated
flexible real-time hand tracking,” in Proceedings of the 33rd Annual tracking in the presence of occlusions,” in 2018 IEEE International
ACM Conference on Human Factors in Computing Systems, ser. CHI Conference on Robotics and Automation (ICRA), May 2018, pp. 643–
’15. New York, NY, USA: ACM, 2015, pp. 3633–3642. [Online]. 650.
Available: http://doi.acm.org/10.1145/2702123.2702179 [14] C. Zimmermann and T. Brox, “Learning to estimate 3D hand
[3] S. Sridhar, F. Mueller, A. Oulasvirta, and C. Theobalt, “Fast and robust pose from single RGB images,” in IEEE International Conference
hand tracking using detection-guided optimization,” in Proceedings of on Computer Vision (ICCV), 2017. [Online]. Available: http:
Computer Vision and Pattern Recognition (CVPR), June 2015. //lmb.informatik.uni-freiburg.de/Publications/2017/ZB17a
[4] P. Krejov, A. Gilbert, and R. Bowden, “Guided optimisation through [15] J. Brookshire and S. Teller, “Articulated pose estimation via over-
classification and regression for hand pose estimation,” Computer parametrization and noise projection,” in Robotics: Science and Sys-
Vision and Image Understanding, vol. 155, pp. 124 – 138, 2017. tems, 2014.
[5] S. E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional [16] V. Ivan, Y. Yang, W. Merkt, M. P. Camilleri, and S. Vijayakumar,
pose machines,” in 2016 IEEE Conference on Computer Vision and EXOTica: An Extensible Optimization Toolset for Prototyping and
Pattern Recognition (CVPR), June 2016, pp. 4724–4732. Benchmarking Motion Planning and Control. Cham: Springer
[6] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for International Publishing, 2019, pp. 211–240. [Online]. Available:
human pose estimation,” in Computer Vision – ECCV 2016, B. Leibe, https://doi.org/10.1007/978-3-319-91590-6 7
J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International [17] D. Tang, J. Taylor, P. Kohli, C. Keskin, T. Kim, and J. Shotton,
Publishing, 2016, pp. 483–499. “Opening the black box: Hierarchical sampling optimization for esti-
[7] K. Pauwels, V. Ivan, E. Ros, and S. Vijayakumar, “Real-time object mating human hand pose,” in 2015 IEEE International Conference on
pose recognition and tracking with an imprecisely calibrated moving Computer Vision (ICCV), Dec 2015, pp. 3325–3333.
RGB-D camera,” in 2014 IEEE/RSJ International Conference on [18] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
Intelligent Robots and Systems, Sept 2014, pp. 2733–2740. image recognition,” in 2016 IEEE Conference on Computer Vision
[8] J. Bohg, J. Romero, A. Herzog, and S. Schaal, “Robot arm pose and Pattern Recognition (CVPR), June 2016, pp. 770–778.
estimation through pixel-wise part classification,” in 2014 IEEE Inter- [19] F. Mueller, D. Mehta, O. Sotnychenko, S. Sridhar, D. Casas, and
national Conference on Robotics and Automation (ICRA), May 2014, C. Theobalt, “Real-time hand tracking under occlusion from an
pp. 3143–3150. egocentric RGB-D sensor,” in Proceedings of International Conference
[9] T. Schmidt, K. Hertkorn, R. Newcombe, Z. Marton, M. Suppa, and on Computer Vision (ICCV), October 2017. [Online]. Available:
D. Fox, “Depth-based tracking with physical constraints for robot https://handtracker.mpi-inf.mpg.de/projects/OccludedHands/
manipulation,” in 2015 IEEE International Conference on Robotics [20] J. Canny, “A computational approach to edge detection,” IEEE Trans-
and Automation (ICRA), May 2015, pp. 119–126. actions on Pattern Analysis and Machine Intelligence, vol. PAMI-8,
no. 6, pp. 679–698, Nov 1986.
[10] J. P. C. Valentin, A. Dai, M. Nießner, P. Kohli, P. H. S.
Torr, S. Izadi, and C. Keskin, “Learning to navigate the energy
landscape,” CoRR, vol. abs/1603.05772, 2016. [Online]. Available:
http://arxiv.org/abs/1603.05772