SfM-Net: Learning of Structure and Motion from Video

...
Sudheendra Vijayanarasimhan∗ Susanna Ricco∗ Cordelia Schmid†∗
Estimated depth, camera motion,
object motion and segmentation
[email protected] [email protected] [email protected]

Rahul Sukthankar∗ Katerina Fragkiadaki‡

[email protected] [email protected]
arXiv:1704.07804v1 [cs.CV] 25 Apr 2017

Abstract Flow

Masks
Forward-backward
We propose SfM-Net, a geometry-aware neural network
constraints
for motion estimation in videos that decomposes frame- Depth

to-frame pixel motion in terms of scene and object depth, Photometric Photometric
error
camera motion and 3D object rotations and translations. error
Weight sharing
CNN
Given a sequence of frames, SfM-Net predicts depth, seg-
mentation, camera and rigid object motions, converts those
into a dense frame-to-frame motion field (optical flow),
Input
differentiably warps frames in time to match pixels and frames t+1 t
t t +1
back-propagates. The model can be trained with vari-
Figure 1. SfM-Net: Given a pair of frames as input, our model
ous degrees of supervision: 1) self-supervised by the re-
decomposes frame-to-frame pixel motion into 3D scene depth, 3D
projection photometric error (completely unsupervised), 2)
camera rotation and translation, a set of motion masks and cor-
supervised by ego-motion (camera motion), or 3) super- responding 3D rigid rotations and translations. It backprojects
vised by depth (e.g., as provided by RGBD sensors). SfM- the resulting 3D scene flow into 2D optical flow and warps ac-
Net extracts meaningful depth estimates and successfully cordingly to match pixels from one frame to the next. Forward-
estimates frame-to-frame camera rotations and transla- backward consistency checks constrain the estimated depth.
tions. It often successfully segments the moving objects in
the scene, even though such supervision is never provided.
in the case of self-supervision. SfM-Net can take advantage
of varying levels of supervision, as demonstrated in our ex-
periments: completely unsupervised (self-supervised), su-
1. Introduction pervised by camera motion, or supervised by depth (from
Kinect).
We propose SfM-Net, a neural network that is trained to SfM-Net is inspired by works that impose geometric
extract 3D structure, ego-motion, segmentation, object ro- constraints on optical flow, exploiting rigidity of the vi-
tations and translations in an end-to-end fashion in videos, sual scene, such as early low-parametric optical flow meth-
by exploiting the geometry of image formation. Given a ods [2, 19, 23] or the so-called direct methods for visual
pair of frames and camera intrinsics, SfM-Net, depicted SLAM (Simultaneous Localization and Mapping) that per-
in Figure 1, computes depth, 3D camera motion, a set of form dense pixel matching from frame to frame while es-
3D rotations and translations for the dynamic objects in the timating a camera trajectory and depth of the pixels in the
scene, and corresponding pixel assignment masks. Those in scene [9, 26]. In contrast to those, instead of optimizing
turn provide a geometrically meaningful motion field (opti- directly over optical flow vectors, 3D point coordinates or
cal flow) that is used to differentiably warp each frame to camera rotation and translation, our model optimizes over
the next. Pixel matching across consecutive frames, con- neural network weights that, given a pair of frames, pro-
strained by forward-backward consistency on the computed duce such 3D structure and motion. In this way, our method
motion and 3D structure, provides gradients during training learns to estimate structure and motion, and can in prin-
∗ Google Research
ciple improve as it processes more videos, in contrast to
† Inria,
Grenoble, France non-learning based alternatives. It can thus be made robust
‡ Carnegie Mellon University to lack of texture, degenerate camera motion trajectories or

1
dynamic objects (our model explicitly accounts for those), re-projection error of the landmark positions. They use syn-
by providing appropriate supervision. Our work is also in- thetic data to pre-train the 2D to 3D mapping of their net-
spired and builds upon recent works on learning geometri- work. Our work considers dense structure estimation and
cally interpretable optical flow fields for point cloud predic- uses videos to obtain the necessary self-supervision, instead
tion in time [5] and backpropagating through camera pro- of static images. Garg et al. [11] also predict depth from a
jection for 3D human pose estimation [33] or single-view single image, supervised by photometric error. However,
depth estimation [11, 35]. they do not infer camera motion or object motion, instead
In summary, our contributions are: requiring stereo pairs with known baseline during training.
Concurrent work to ours [35] removes the constraint that
• A method for self-supervised learning in videos in- the ground-truth pose of the camera be known at train-
the-wild, through explicit modeling of the geometry of ing time, and instead estimates the camera motion between
scene motion and image formation. frames using another neural network. Our approach tackles
the more challenging problem of simultaneously estimating
• A deep network that predicts pixel-wise depth from a
both camera and object motion.
single frame along with camera motion, object motion,
and object masks directly from a pair of frames.
Geometry-aware motion estimation. Motion estimation
• Forward-backward constraints for learning a consis- methods that exploit rigidity of the video scene and the ge-
tent 3D structure from frame to frame and better ex- ometry of image formation to impose constraints on optical
ploit self-supervision, extending left-right consistency flow fields have a long history in computer vision [2, 3, 19].
constraints of [13]. Instead of non-parametric dense flow fields [14] researchers
have proposed affine or projective transformations that bet-
We show results of our approach on KITTI [12, 21], ter exploit the low dimensionality of rigid object mo-
MoSeg [4], and RGB-D SLAM [27] benchmarks under tion [23]. When depth information is available, motions are
different levels of supervision. SfM-Net learns to predict rigid rotations and translations [15]. Similarly, direct meth-
structure, object, and camera motion by training on realistic ods for visual SLAM having RGB [26] or RGBD [17] video
video sequences using limited ground-truth annotations. as input, perform dense pixel matching from frame to frame
while estimating a camera trajectory and depth of the pixels
2. Related work in the scene with impressive 3D point cloud reconstructions.
Back-propagating through warps and camera projec- These works typically make a static world assumption,
tion. Differentiable warping [16] has been used to learn which makes them susceptible to the presence of moving
end-to-end unsupervised optical flow [34], disparity flow objects in the scene. Instead, SfM-Net explicitly accounts
in a stereo rig [13] and video prediction [24]. The closest for moving objects using motion masks and 3D translation
previous works to ours are SE3-Nets [5], 3D image inter- and rotation prediction.
preter [33], and Garg et al.’s depth CNN [11]. SE3-Nets [5]
use an actuation force from a robot and an input point cloud Learning-based motion estimation. Recent works [7,
to forecast a set of 3D rigid object motions (rotation and 20, 29] propose learning frame-to-frame motion fields with
translations) and corresponding pixel motion assignment deep neural networks supervised with ground-truth motion
masks under a static camera assumption. Our work uses obtained from simulation or synthetic movies. This en-
similar representation of pixel motion masks and 3D mo- ables efficient motion estimation that learns to deal with
tions to capture the dynamic objects in the scene. How- lack of texture using training examples rather than relying
ever, our work differs in that 1) we predict depth and cam- only on smoothness constraints of the motion field, as previ-
era motion while SE3-Nets operate on given point clouds ous optimization methods [28]. Instead of directly optimiz-
and assume no camera motion, 2) SE3-Nets are supervised ing over unknown motion parameters, such approaches op-
with pre-recorded 3D optical flow, while this work admits timize neural network weights that allow motion prediction
diverse and much weaker supervision, as well as complete in the presence of ambiguities in the given pair of frames.
lack of supervision, 3) SE3-Nets consider one frame and an
action as input to predict the future motion, while our model Unsupervised learning in videos. Video holds a great
uses pairs of frames as input to estimate the intra-frame mo- potential towards learning semantically meaningful visual
tion, and 4) SE3-Nets are applied to toy or lab-like setups representations under weak supervision. Recent works have
whereas we show results on real videos. explored this direction by using videos to propagate in time
Wu et al. [33] learn 3D sparse landmark positions of semantic labels using motion constraints [25], impose tem-
chairs and human body joints from a single image by com- poral coherence (slowness) on the learnt visual feature [32],
puting a simplified camera model and minimizing a camera predict temporal evolution [30], learn temporal instance

2
 T
Po ran
i n s fo
t C rm
lo ed
ud
Camera Motion
512

Fl
ow
P
Fr air 512
am of Object Motion
es

Point Cloud
Transformed
38
4
x1
2

O ask
8x

MOTION NETWORK

bj s
6

M
ec
S
Fr ing

t
am le
e
STRUCTURE NETWORK
Point
s =1 deconv Cloud
s=2

38 38
4x D
12 48 48
4 x1 ep

19
19

24 24 28 t
96

x1

96
8x x1 h

2x
2x

x
12

32 6 8 x
x

x3
3 512 512 1024 1024 8 512 6 256

64
6

32

64
x4

64
4

64

2
256 256 128 32
128 128

Figure 2. SfM-Net architecture. For each pair of consecutive frames It , It+1 , a conv/deconv sub-network predicts depth dt while another
predicts a set of K segmentation masks mt . The coarsest feature maps of the motion-mask encoder are further decoded through fully
connected layers towards 3D rotations and translations for the camera and the K segmentations. The predicted depth is converted into
a per frame point-cloud using estimated or known camera intrinsics. Then, it is transformed according to the predicted 3D scene flow,
as composed by the 3D camera motion and independent 3D mask motions. Transformed 3D depth is projected back to the 2D next
frame, and thus provides corresponding 2D optical flow fields. Differentiable backward warping maps frame It+1 to It , and gradients are
computed based on pixel errors. Forward-backward constraints are imposed by repeating this process for the inverted frame pair It+1 , It
and constraining the depths dt and dt+1 to be consistent through the estimated scene motion.

level associations [31], predict temporal ordering of video transforming the point cloud based on camera and object
frames [22], etc. rigid transformations, and back projecting the transformed
Most of those unsupervised methods are shown to be 3D coordinates to the image plane. Then, given the optical
good pre-training mechanisms for object detection or classi- flow field between initial and projected pixel coordinates,
fication, as done in [22, 30, 31]. In contrast and complemen- differentiable backward warping is used to map frame It+1
tary to the works above, our model extracts fine-grained 3D to It . Forward-backward constraints are imposed by repeat-
structure and 3D motion from monocular videos with weak ing this process from frame It+1 to It and constraining the
supervision, instead of semantic feature representations. depths dt and dt+1 to be consistent through the estimated
scene motion. We provide details of each of these compo-
3. Learning SfM nents below.

3.1. SfM-Net architecture.

Our model is shown in Figure 2. Given frames It ,
It+1 ∈ <w×h , we predict frame depth dt ∈ [0, ∞)w×h , Depth and per-frame point clouds. We compute per
camera rotation and translation {Rtc , tct } ∈ SE3, and a set frame depth using a standard conv/deconv subnetwork oper-
of K motion masks mkt ∈ [0, 1]w×h , k ∈ 1, . . . , K that de- ating on a single frame (the structure network in Figure 2).
note membership of each pixel to K corresponding rigid ob- We use a RELU activation at our final layer, since depth
ject motions {Rtk , tkt } ∈ SE3, k ∈ {1, . . . , K}. Note that values are non-negative. Given depth dt , we obtain the 3D
a pixel may be assigned to none of the motion masks, de- point cloud Xit = (Xti , Yti , Zti ), i ∈ 1, . . . , w × h corre-
noting that it is a background pixel and part of the static sponding to the pixels in the scene using a pinhole camera
world. Using the above estimates, optical flow is computed model. Let (xit , yti ) be the column and row positions of the
by first generating the 3D point cloud corresponding to the ith pixel in frame It and let (cx , cy , f ) be the camera intrin-
image pixels using the depth map and camera intrinsics, sics, then

3
  i  K maps it is assigned to the static background whose mo-
xt
Xti tion is a function of the global camera motion alone. We
 
i − cx
d w
Xit =  Yti  = t  yti − c  (1)
 
i f h y allow a pixel to belong to multiple rigid body transforms
Zt f in order to capture composition of motions, e.g., through
kinematic chains, such as articulated bodies. Learning the
where dit denotes the depth value of the ith pixel. We use required number of motions for a sequence is an interest-
the camera intrinsics when available and revert to default ing open problem. We found that we could fix K = 3 for
values of (0.5, 0.5, 1.0) otherwise. Therefore, the predicted all experiments presented here. Note that our method can
depth will only be correct up to a scalar multiplier. learn to ignore unnecessary object motions in a sequence
by assigning no pixels to the corresponding mask.
Scene motion. We compute the motion of the camera
and of independently moving objects in the scene using a
Optical flow. We obtain optical flow by first transforming
conv/deconv subnetwork that operates on a pair of images
the point cloud obtained in Equation 1 using the camera and
(the motion network in Figure 2). We depth-concatenate
object motion rigid body transformations followed by pro-
the pair of frames and use a series of convolutional layers to
jecting the 3D point on to the image plane using the camera
produce an embedding layer. We use two fully-connected
intrinsics. In the following, we drop the pixel superscript i
layers to predict the motion of the camera between the
from the 3D coordinates, since it is clear we are referring to
frames and a predefined number K of rigid body motions
the motion transformation of the ith pixel of the tth frame.
that explain moving objects in the scene.
We first apply the object transformations:
Let {Rtc , tct } ∈ SE3 denote the 3D rotation and PK
X0t = Xt + k=1 mkt (i)(Rtk (Xt − pk ) + tkt − Xt ). We
translation of the camera from frame It to frame It+1
then apply the camera transformation:
(relative camera pose across consecutive frames). We
represent Rtc using an Euler angle representation as X00t = Rtc (X0t − pct ) + tct .
Rtc x (α)Rtc y (β)Rtc z (γ) where Finally we obtain the row and column position of the
pixel in the second frame (xit+1 , yt+1
i
) by projecting the
corresponding 3D point Xt = (Xt , Yt00 , Zt00 ) back to the
00 00
 
cos α − sin α 0
Rtc x (α) =  sin α cos α 0 ,
image plane as follows:
0 0 1
   00 
cos β 0 sin β "
xit+1
# X  
Rtc y (β) =  0 1 0 , f  t00  c
− sin β 0 cos β
w
i = 00 Yt + x
yt+1 Zt cy
  h f
1 0 0
cz
Rt (γ) = 0 cos γ − sin γ  ,

0 sin γ cos γ The flow U, V between the two frames at pixel i is then
i
(Ut (i), Vt (i)) = (xit+1 − xit , yt+1 − yti ).
and α, β, γ are the angles of rotation about the x, y, z-axes
respectively. The fully-connected layers are used to predict 3.2. Supervision
translation parameters tc , the pivot points of the camera ro-
SfM-Net inverts the image formation and extracts depth,
tation pc ∈ R3 as in [5], and sin α, sin β, sin γ. These last
camera and object motions that gave rise to the observed
three parameters are constrained
temporal differences, similar to previous SfM works [1, 6].
to be in the interval [−1, 1] by using RELU activation
Such inverse problems are ill-posed as many solutions of
and the minimum function.
depth, camera and object motion can give rise to the same
Let {Rtk , tkt } ∈ SE3, k ∈ {1, ..., K} denote the 3D rigid
observed frame-to-frame pixel values. A learning-based so-
motions of up to K objects in the scene. We use similar rep-
lution, as opposed to direct optimization, has the advan-
resentations as for camera motion and predict parameters
tage of learning to handle such ambiguities through par-
using fully-connected layers on top of the same embedding
tial supervision of their weights or appropriate pre-training,
E. While camera motion is a global transformation applied
or simply because the same coefficients (network weights)
to all the pixels in the scene, the object motion transforms
need to explain a large abundance of video data consistently.
are weighted by the predicted membership probability of
We detail the various supervision modes below and explore
each pixel to each rigid motion, mkt ∈ [0, 1](h×w) , k ∈
a subset of them in the experimental section.
{1, . . . , K}. These masks are produced by feeding the em-
bedding layer through a deconvolutional tower. We use sig-
moid activations at the last layer instead of softmax in or- Self-Supervision. Given unconstrained video, without
der to allow each pixel to belong to any number of rigid accompanying ground-truth structure or motion informa-
body motions. When a pixel has zero activation across all tion, our model is trained to minimize the photometric error

4
between the first frame and the second frame warped to- Supervising camera motion. If ground-truth camera
wards the first according to the predicted motion field, based pose trajectories are available, we can supervise our model
on well-known brightness constancy assumptions [14]: by computing corresponding ground-truth camera rota-
tion and translation Rtc−GT , tc−GT
t from frame to frame,
and constrain our camera motion predictions accordingly.
1 X Specifically, we compute the relative transformation be-
Lcolor
t = kIt (x, y) − It+1 (x0 , y 0 )k1 tween predicted and ground-truth camera motion {terr =
w h x,y t
inv(Rtc )(tc−GT
t − tct ), Rterr = inv(Rtc )Rtc−GT )} and mini-
mize its rotation angle and translation norm [27]:
where x0 = x + Ut (x, y) and y 0 = y + Vt (x, y). We use
differentiable image warping proposed in the spatial trans- Lct trans = kterr
t k2 
crot
 
trace(Rterr )−1
 (2)
former work [16] and compute color constancy loss in a Lt = arccos min 1, max −1, 2
fully differentiable manner.

Spatial smoothness priors. When our network is self- Supervising optical flow and object motion. Ground-
supervised, we add robust spatial smoothness penalties on truth optical flow, object masks, or object motions re-
the optical flow field, the depth, and the inferred motion quire expensive human annotation on real videos. How-
maps, by penalizing the L1 norm of the gradients across ever, these signals are available in recent synthetic
adjacent pixels, as usually done in previous works [18]. datasets [20]. In such cases, our model could be
For depth prediction, we penalize the norm of second or- trained to minimize, for example, an L1 regression loss
der gradients in order to encourage not constant but rather between predicted {U (x, y), V (x, y)} and ground-truth
smoothly changing depth values. {U GT (x, y), V GT (x, y)} flow vectors.

3.3. Implementation details

Forward-backward consistency constraints. We incor- Our depth-predicting structure and object-mask-
porate forward-backward consistency constraints between predicting motion conv/deconv networks share similar
inferred scene depth in different frames as follows. Given architectures but use independent weights. Each consist
inferred depth dt from frame pair It , It+1 and dt+1 from of a series of 3 × 3 convolutional layers alternating
frame pair It+1 , It , we ask for those to be consistent under between stride 1 and stride 2 followed by deconvolutional
the inferred scene motion, that is: operations consisting of a depth-to-space upsampling,
X concatentation with corresponding feature maps from the
LF
t
B
= 1
wh | (dt (x, y) + Wt (x, y)) − convolutional portion, and a 3 × 3 convolutional layer.
x,y Batch normalization is applied to all convolutional layer
dt+1 (x + Ut (x, y), y + Vt (x, y))| outputs. The structure network takes a single frame as
input, while the motion network takes a pair of frames.
where Wt (x, y) is the Z component of the scene flow ob- We predict depth values using a 1 × 1 convolutional layer
tained from the point cloud transformation. Composing on top of the image-sized feature map. We use RELU
scene flow forward and backward across consecutive frames activations because depths are positive and a bias of 1 to
allows us to impose such forward-backward consistency cy- prevent small depth values. The maximum predicted depth
cles across more than one frame gaps, however, we have not value is further clipped at 100 to prevent large gradients.
yet seen empirical gain from doing so. We predict object masks from the image-sized feature map
of the motion network using a 1 × 1 convolutional layer
with sigmoid activations. To encourage sharp masks we
Supervising depth. If depth is available on parts of the multiply the logits of the masks by a parameter that is a
input image, such as with video sequences captured by a function of the number of step for which the network has
Kinect sensor, we can use depth supervision in the form of been trained. The pivot variables are predicted as heat maps
robust depth regression: using a softmax function over all the locations in the image
followed by a weighted average of the pixel locations.
1 X
Ldepth = dmaskGT GT
t (x, y)·kdt (x, y)−dt (x, y)k1 ,
t
w h x,y 4. Experimental results
The main contribution of SfM-Net is the ability to explic-
where dmaskGTt denotes a binary image that signals pres- itly model both camera and object motion in a sequence, al-
ence of ground-truth depth. lowing us to train on unrestricted videos containing moving

5
objects. To demonstrate this, we trained self-supervised net- Approach Log RMSE
works (using zero ground-truth supervision) on the KITTI KITTI 2012 KITTI 2015
datasets [12, 21] and on the MoSeg dataset [4]. KITTI with stereo pairs 0.31 0.34
contains pairs of frames captured from a moving vehicle seq. with motion masks 0.45 0.41
in which other independently moving vehicles are visible. seq. without motion masks 0.77 1.25
MoSeg contains sequences with challenging object motion, Table 1. RMSE of Log depth with respect to ground truth for our
including articulated motions from moving people and ani- model with stereo pairs and with and without motion masks on
mals. sequences in KITTI 2012 and 2015 datasets. When using stereo
pairs the camera pose between the frames is fixed and the model is
equivalent to the approach of Garg et al. [11]. Motion masks help
KITTI. Our first experiment validates that explicitly improve the error on both datasets but more so on the KITTI 2015
modeling object motion is necessary to effectively learn dataset which contains more moving objects.
from unconstrained videos. We evaluate unsupervised
depth prediction using our models on the KITTI 2012 and RGB frame Predicted Depth
KITTI 2015 datasets which contain close to 200 frame se- (stereo pairs) (sequence)
quence and stereo pairs. We use a scale-invariant error met-
ric (log RMSE) proposed in [8] due to the global scale am-
biguitiy in monocular setups which is defined as
!2
1 X ¯ 1 X ¯
Escaleinv = kd(x, y)k2 − kd(x, y)k1 ,
N x,y N x,y

where N is the number of pixels and d¯ = (log(d) −

log(dGT )) denotes the difference between the log of
ground-truth and predicted depth maps. We pre-train the
our unsupervised depth prediction models using adjacent
frame pairs on the raw KITTI dataset which contains ∼
42, 000 frames and train and evaluate on KITTI 2012 and
2015 which have depth ground truth.
We compare the the results of Garg et al. [11] who use
stereo pairs to estimate depth. Their approach assumes the Figure 3. Qualitative comparison of the estimated depth using our
camera pose between the frames is a known constant (stereo unsupervised model on sequences versus using stereo pairs in the
baseline) and optimize the photometric error in order to es- KITTI 2012 benchmark. When using stereo pairs the camera pose
timate the depth. In contrast, our model considers a more between the pair is constant and hence the model is equivalent to
challenging “in the wild” setting where we are only given the approach of Garg et al. [11]. For sequences, our model needs
sequences of frames from a video and camera pose, depth to additionally predict camera rotation and translation between the
two frames. The first six rows show successful predictions even
and object motion are all estimated without any form of su-
without camera pose information and the last two illustrate failure
pervision. Garg et al. report a log RMSE of 0.273 on a sub- cases. The failure cases show that when there is no translation
set of the KITTI dataset. To compare with our approach on between the two frames depth estimation fails whereas when using
the full set we emulate the model of Garg et al. using our ar- stereo pairs there is always a constant offset between the frames.
chitecture by removing object masks from our network and
using stereo pairs with photometric error. We also evaluate
our full model on frame sequence pairs with camera motion when no motion masks are used. The gap between the two
estimation both with and without explicit object motion es- approaches is wider on the KITTI 2015 dataset which con-
timation. tains more moving objects. This shows that it is important
Table 1 shows the log RMSE error between the ground- to account for moving objects when training on videos in
truth depth and the three approaches. When using stereo the wild.
pairs we obtain a value of 0.31 which is on par with exist- Figure 3 shows qualitative examples comparing the
ing results on the KITTI benchmark (see [11]). When us- depth obtained when using stereo pairs with a fixed base-
ing frame sequence pairs instead of calibrated stereo pairs line and when using frame sequences without camera pose
the problem becomes more difficult, as we must now in- information. When there is large translation between the
fer the unknown camera and object motion between the two frames, depth estimation without camera pose information
frames. As expected, the depth estimates learned in this is as good as using stereo pairs. The failure cases in the last
scenario are less accurate, but performance is much worse two rows show that the network did not learn to accurately

6
 Predicted Motion Masks Ground Truth Mask Predicted Flow Ground Truth Flow

Figure 4. Ground truth segmentation and flow compared to predicted motion masks and flow from SfM-Net in KITTI 2015. The model
was trained in a fully unsupervised manner. The top six rows show successful prediction and the last two show typical failure cases.

predict depth for scenes where it saw little or no translation Analysis of our failure cases suggest possible directions
between the frames during training. This is not the case for improvement. Moving objects introduce significant oc-
when using stereo pairs as there is always a constant offset clusions, which should be handled carefully. Because our
between the frames. Using more data could help here be- network has no direct supervision on object masks or object
cause it increases the likelihood of generic scenes appearing motion, it does not necessarily learn that object and camera
in a sequence containing interesting camera motion. motions should be different. These priors could be built into
our loss or learned directly if some ground-truth masks or
Figure 4 provides qualitative examples of the predicted object motions are provided as explicit supervision.
motion masks and flow fields along with the ground-truth in
the KITTI 2015 dataset. Often, the predicted motion masks
are fairly close to the ground truth and help explain part of MoSeg. The moving objects in KITTI are primarily ve-
the motion in the scene. We notice that object masks tended hicles, which undergo rigid-body transformations, making
to miss very small, distant moving objects. This may be due it a good match for our model. To verify that our network
to the fact that these objects and their motions are too small can still learn in the presence of non-rigid motion, we re-
to be separated from the background. The bottom two rows trained it from scratch under self-supervision on the MoSeg
show cases where the predicted masks do not correspond to dataset, using frames from all sequences. Because each mo-
moving objects. In the first example, although the mask is tion mask corresponds to a rigid 3D rotation and transla-
not semantically meaningful, note that the estimated flow tion, we do not expect a single motion mask to capture a
field is reasonable, with some mistakes in the region oc- deformable object. Instead, different rigidly moving object
cluded by the moving car. In the second failure case, the parts will be assigned to different masks. This is not a prob-
moving car on the left is completely missed but the motion lem from the perspective of accurate camera motion esti-
of the static background is well captured. This is a partic- mation, where the important issue is distinguishing pixels
ularly difficult example for the self-supervised photometric whose motion is caused by the camera pose transformation
loss because the moving object appears in heavy shadow. directly from those whose motion is affected by indepen-

7
 RGB frame Predicted flow Motion masks Seq. transl [27] rot [27] transl. ours rot ours
360 0.099 0.474 0.009 1.123
plant 0.016 1.053 0.011 0.796
teddy 0.020 1.14 0.0123 0.877
desk 0.008 0.495 0.012 0.848
desk2 0.099 0.61 0.012 0.974
Table 2. Camera pose relative error from frame to frame for vari-
ous video sequences of Freiburg RGBD-SLAM benchmark.

the next and compare with the predicted camera motion

from our model, by measuring translation and rotation error
of their relative transformation, as done in the correspond-
ing evaluation script for relative camera pose error and de-
tailed in Eq. 2. We report camera rotation and translation
error in Table 2 for each of the Freiburg1 sequences com-
pared to the error in the benchmark’s baseline trajectories.
Our model was trained from scratch for each sequence and
used the focal length value provided with the dataset. We
observe that our results better estimate the frame-to-frame
translation and are comparable for rotation.
Figure 5. Motion segments computed from SfM-Net in MoSeg [4].
The model was trained in a fully unsupervised manner.
5. Conclusion

dent object motions in the scene. Current geometric SLAM methods obtain excellent ego-
motion and rigid 3D reconstruction results, but often come
Qualitative results on sampled frames from the dataset
at a price of extensive engineering, low tolerance to mov-
are shown in Fig. 5. Because MoSeg only contains ground-
ing objects — which are treated as noise during recon-
truth annotations for segmentation, we cannot quantitatively
struction — and sensitivity to camera calibration. Further-
evaluate the estimated depth, camera trajectories, or optical
more, matching and reconstruction are difficult in low tex-
flow fields. However, we did evaluate the quality of the ob-
tured regions. Incorporating learning into depth reconstruc-
ject motion masks by computing Intersection over Union
tion, camera motion prediction and object segmentation,
(IoU) for each ground-truth segmentation mask against the
while still preserving the constraints of image formation,
best matching motion mask and its complement (a total of
is a promising way to robustify SLAM and visual odome-
six proposed segments in each frame, two from each of the
try even further. However, the exact training scenario re-
three motion masks), averaging across frames and ground-
quired to solve this more difficult inference problem re-
truth objects. We obtain an IoU of 0.29 which is similar to
mains an open question. Exploiting long history and far in
previous unsupervised approaches for the small number of
time forward-backward constraints with visibility reason-
segmentation proposals we use per frame. See, for exam-
ing is an important future direction. Further, exploiting a
ple, the last column of Figure 5 from [10], whose proposed
small amount of annotated videos for object segmentation,
methods for moving object proposals achieve IoU around
depth, and camera motion, and combining those with an
0.3 with four proposals. They require more than 800 pro-
abundance of self-supervised videos, could help initialize
posals to reach an IoU above 0.57.
the network weights in the right regime and facilitate learn-
ing. Many other curriculum learning regimes, including
Kinect depth supervision. While the fully unsupervised those that incorporate synthetic datasets, can also be con-
results show promise, our network can benefit from extra sidered.
supervision of depth or camera motion when available. The
improved depth prediction given ground truth camera poses
on KITTI stereo demonstrate some gain. We also experi-
mented with adding depth supervision to improve camera Acknowledgements. We thank our colleagues Tinghui
motion estimation using the RGB-D SLAM dataset [27]. Zhou, Matthew Brown, Noah Snavely, and David Lowe for
Given ground-truth camera pose trajectories, we estimated their advice and Bryan Seybold for his work generating syn-
relative camera pose (camera motion) from each frame to thetic datasets for our initial experiments.

8
References [23] T. Nir, A. Bruckstein, and R. Kimmel. Over-Parameterized
Variational Optical Flow. IJCV, 76(2):205–216, 2008.
[1] I. Akhter, Y. A. Sheikh, S. Khan, and T. Kanade. Nonrigid
[24] V. Patraucean, A. Handa, and R. Cipolla. Spatio-temporal
structure from motion in trajectory space. In NIPS, 2008.
video autoencoder with differentiable memory. CoRR,
[2] J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. Hi-
abs/1511.06309, 2015.
erarchical model-based motion estimation. In ECCV, 1992.
[25] A. Prest, C. Leistner, J. Civera, C. Schmid, and V. Fer-
[3] M. Black, Y. Yacoob, A. Jepson, and D. Fleet. Learning
rari. Learning object class detectors from weakly annotated
parameterized models of image motion. In CVPR, 1997.
video. In CVPR, 2012.
[4] T. Brox and J. Malik. Object segmentation by long term
[26] T. Schöps, J. Engel, and D. Cremers. Semi-dense visual
analysis of point trajectories. In ECCV. 2010.
odometry for AR on a smartphone. In ISMAR, 2014.
[5] A. Byravan and D. Fox. SE3-Nets: Learning rigid body mo-
[27] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cre-
tion using deep neural networks. CoRR, abs/1606.02378,
mers. A benchmark for the evaluation of RGB-D SLAM
2016.
systems. In IROS, 2012.
[6] J. Costeira and T. Kanade. A multi-body factorization
method for motion analysis. ICCV, 1995. [28] D. Sun, S. Roth, and M. J. Black. Secrets of optical flow
[7] A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazirbas, estimation and their principles. In CVPR, 2010.
V. Golkov, P. Smagt, D. Cremers, and T. Brox. FlowNet: [29] J. Thewlis, S. Zheng, P. H. Torr, and A. Vedaldi. Fully-
Learning optical flow with convolutional networks. In ICCV, trainable deep matching. In BMVC, 2016.
2015. [30] J. Walker, C. Doersch, A. Gupta, and M. Hebert. An uncer-
[8] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction tain future: Forecasting from static images using variational
from a single image using a multi-scale deep network. In autoencoders. In ECCV, 2016.
NIPS, 2014. [31] X. Wang and A. Gupta. Unsupervised learning of visual rep-
[9] J. Engel, T. Schöps, and D. Cremers. LSD-SLAM: Large- resentations using videos. In ICCV, 2015.
scale direct monocular SLAM. In ECCV, 2014. [32] L. Wiskott and T. J. Sejnowski. Slow feature analysis:
[10] K. Fragkiadaki, P. A. Arbeláez, P. Felsen, and J. Ma- Unsupervised learning of invariances. Neural Comput.,
lik. Spatio-temporal moving object proposals. CoRR, 14(4):715–770, 2002.
abs/1412.6504, 2014. [33] J. Wu, T. Xue, J. J. Lim, Y. Tian, J. B. Tenenbaum, A. Tor-
[11] R. Garg, B. V. Kumar, G. Carneiro, and I. Reid. Unsuper- ralba, and W. T. Freeman. Single image 3D interpreter net-
vised cnn for single view depth estimation: Geometry to the work. In ECCV, 2016.
rescue. In ECCV, 2016. [34] J. J. Yu, A. W. Harley, and K. G. Derpanis. Back to ba-
[12] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for au- sics: Unsupervised learning of optical flow via brightness
tonomous driving? The KITTI vision benchmark suite. In constancy and motion smoothness. In ECCV, 2016.
CVPR, 2012. [35] T. Zhou, M. Brown, N. Snavely, and D. Lowe. Unsupervised
[13] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsuper- learning of depth and ego-motion from video. In CVPR 2017.
vised monocular depth estimation with left-right consistency.
CoRR, abs/1609.03677, 2016.
[14] B. K. Horn and B. G. Schunck. Determining optical flow.
Artificial Intelligence, 17, 1981.
[15] M. Hornacek, A. Fitzgibbon, and C. Rother. SphereFlow: 6
DoF scene flow from RGB-D pairs. In CVPR, 2014.
[16] M. Jaderberg, K. Simonyan, A. Zisserman, and
K. Kavukcuoglu. Spatial transformer networks. In
NIPS, 2015.
[17] C. Kerl, J. Sturm, and D. Cremers. Dense visual SLAM for
RGB-D cameras. In IROS, 2013.
[18] N. Kong and M. J. Black. Intrinsic depth: Improving depth
transfer with intrinsic images. In ICCV, 2015.
[19] L. Z. Manor and M. Irani. Multi-Frame Estimation of Planar
Motion. PAMI, 22(10):1105–1116, 2000.
[20] N. Mayer, E. Ilg, P. Häusser, P. Fischer, D. Cremers,
A. Dosovitskiy, and T. Brox. A large dataset to train convo-
lutional networks for disparity, optical flow, and scene flow
estimation. In CVPR, 2016.
[21] M. Menze and A. Geiger. Object scene flow for autonomous
vehicles. In CVPR, 2015.
[22] I. Misra, C. L. Zitnick, and M. Hebert. Unsupervised learn-
ing using sequential verification for action recognition. In
ECCV, 2016.