See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/2903196

Determining Constant Optical Flow

Article · April 2004

Source: CiteSeer

CITATIONS READS
144 159

1 author:

Berthold K. P. Horn
Massachusetts Institute of Technology
205 PUBLICATIONS 30,657 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Kinematics, Statics, and Dynamics of simple kinematic chain robot arms View project

Satellite Image Interpretation View project

All content following this page was uploaded by Berthold K. P. Horn on 13 December 2012.

The user has requested enhancement of the downloaded file.

 Determining Constant Optical Flow
Berthold K.P. Horn
Copyright © 2003

The original optical flow algorithm [1] dealt with a flow field that could
vary from place to place in the image, as would typically occur when a
camera is moved through a three-dimensional environment—or if objects
moved in front of a fixed camera.
A related, but simpler problem, is that of recovering the motion of an
image, all parts of which move with the same velocity (section 4.3 in [2]).
The solution of this problem enables “optical mice,” as well as motion
compensation in hand-held video cameras—critical to their operation.
We provide here background on moving images, a method for the esti-
mation of image velocity using least squares applied to brightness deriva-
tives, and an analysis of the sensitivity of the estimated image velocity to
noise in brightness measurement.

Constant Brightness Assumption

As in the solution of the general optical flow problem [1], we start with
the “constant brightness assumption.” If E(x, y, t) is the brightness at
image point (x, y) at time t, then we assume that
d
E(x, y, t) = 0 (1)
dt
that is, we assume that as the image of some feature moves, it does not
change brightness.
If the image moves δx in the x-direction and δy in the y-direction
during the interval δt, then
E(x + δx, y + δy, t + δt) = E(x, y, t) (2)
Taylor series expansion yields
∂E ∂E ∂E
E(x, y, t) + δx + δy + δt + . . . = E(x, y, t) (3)
∂x ∂y ∂t
where the ellipsis (. . .) indicates terms that are of higher order in the in-
crements. Cancelling E(x, y, t) from both sides, we obtain
∂E ∂E ∂E
δx + δy + δt + . . . = 0 (4)
∂x ∂y ∂t
Dividing through by δt and considering the limit as δt → 01 :
∂E dx ∂E dy ∂E
+ + =0 (5)
∂x dt ∂y dt ∂t

1 We could have obtained eq. 5 more directly by expanding the total derivative
in eq. 1 using the chain rule, see [1].
 2

Brightness Change Constraint Equation

This result, called the “brightness change constraint equation” can be writ-
ten in the more compact form
uEx + vEy + Et = 0 (6)
if we let
∂E ∂E ∂E
Ex = , Ey = , and Et = , (7)
∂x ∂y ∂t
be the brightness gradient (Ex , Ey ) and the time-rate of change of bright-
ness at a point in the image, and if we let
dx dy
u= and v = (8)
dt dt
be the image velocity components in the x and y directions.
The brightness change constraint equation is a linear constraint on
the image velocity components u and v with the partial derivatives of
brightness, Ex , Ey , Et as coefficients. This constraint equation applies at
every picture cell in each image frame.

Numerical Estimation of Derivatives

Before we go on to use the brightness change constraint equation, let us

consider just how we can estimate the partial derivatives of brightness.
The derivatives can be estimated from images using first differences. It
is, however, important to be consistent in how the three derivatives are
estimated. The three derivative estimates should be “centered” at the
same point in space and time. This can be accomplished by considering
a 2 × 2 × 2 cube of values of brightness in (x, y, t) space (see Fig. 1).

Each of the derivatives is based on the difference between the average

value over one 2 × 2 side of the cube and the opposite 2 × 2 side.
 3

For example,

1 1 
{Ex }i,j,k ≈ Ei,j+1,k + Ei+1,j+1,k + Ei,j+1,k+1 + Ei+1,j+1,k+1
x4
 (9)
1 
− Ei,j,k + Ei+1,j,k + Ei,j,k+1 + Ei+1,j,k+1
4
where Ei,j,k is the brightness at pixel (i, j) (that is, the pixel in row i and
column j) in frame k, and x is the pixel spacing in the x direction.
Note that the estimates of the derivatives are centered at points that
are in-between pixels and in-between image frames—rather than being
aligned with them. In a rectangular array of pictures cells, there will be
one fewer row and one fewer columns of derivative estimates than there
are picture cell rows and columns. So {Ex }i,j,k is the estimate of the x-
derivative of E(x, y, t) at pixel (i + 1/2, j + 1/2) at frame (k + 1/2).
Averaging four values2 has the added benefit of reducing the standard
deviation of random noise in the signal by one half. Additional smoothing
may be applied to the image before estimating the derivatives in order to
further reduce noise, as long as enough bits of precision are retained.

Analysis of Constraint and “Normal Flow Magnitude”

A single constraint does not, by itself, provide enough information to pin

down both components of image velocity (u, v). The constraint can be
expressed as a straight line in image velocity space (see Fig. 2).

All points (u, v) on that line are valid solutions. The constraint reduces
the possibilities for (u, v) from two degrees of freedom to one.
2 Onemay want to just keep the sum of the four values (rather than dividing by
four) in order to retain more precision when using integer arithmetic.
 4

We can see more clearly just what constraint a single measurement

provides by dividing the brightness change constraint equation by the
magnitude of the brightness gradient:
1 Et
 (Ex , Ey ) · (u, v) = −  (10)
Ex2 + Ey
2
Ex2 + Ey
2

The term to the left of the dot is a unit vector in the direction of the
brightness gradient3 . The dot-product yields the component of (u, v)
in the direction of the brightness gradient. So the component of image
velocity in the direction of the brightness gradient can be determined.
The component orthogonal to this direction (i.e. parallel to (Ey , −Ex )),
however, can not. That is, the component in the direction along isophotes
(contours of constant brightness) cannot be determined. The component
in the direction of the brightness gradient (i.e. the right hand side term
in eq. 10) is sometimes referred to as the “normal flow”—i.e. the flow
“normal” or orthogonal to the isophotes.

Using Brightness Gradients at Two Points

To determine both components of image velocity, we need one more con-

straint. One way to obtain another constraint is to consider another point
in the image—one where the local brightness gradient (Ex,2 , Ey,2 ) is dif-
ferent from that at the first point (Ex,1 , Ey,1 ). The brightness change con-
straint equation applied at these points—assuming that the image velocity
is the same in the two places—give us two linear equations in u and v:
uEx,1 + vEy,1 = −Et,1
(11)
uEx,2 + vEy,2 = −Et,2
or, equivalently two straight lines in velocity space.
We can write the two equations compactly in matrix-vector form as
Ms = b (12)
where s = (u, v)T , M is the 2 × 2 matrix of coefficients, and b has as
components the right hand side values in eq. 11. The equations can be
solved for the intersection of the lines as long as the determinant
Δ = Ex,1 Ey,2 − Ex,2 Ey,1 (13)
of the matrix M is non-zero. Then
s = M −1 b (14)
3 One can think of E(x, y) as the height of a surface as a function of x and
y. Then isophotes are contours of constant height and the direction of the
brightness gradient is the direction of steepest ascent, while the magnitude of
the brightness gradient is the slope in that direction.
 5

or, in component form,

Δ u = Ey,1 Et,2 − Ey,2 Et,1
(15)
Δ v = Ex,2 Et,1 − Ex,1 Et,2
The determinant will be zero when the two brightness gradients are par-
allel, that is, when the ratio Ey,2 : Ex,2 equals the ratio Ey,1 : Ex,1 . In that
case the two equations in eq. 11 are not independent.
The determinant is essentially the magnitude of the cross-product of
the two brightness gradients (Ex,1 , Ey,1 ) and (Ex,2 , Ey,2 ). Consequently,
the determinant will be small when the gradients are near parallel—or
when the gradients are weak. In this case noise in measurements will be
greatly amplified when computing u and v because of the division by the
small determinant. So this method works best when the two brightness
gradients point in very different directions—and are strong.
Another way of looking at this is to consider that the solution of the
pair of equations in eq. 11 gives the intersection of two constraint lines in
velocity space. The position of that intersection will be more sensitive to
small errors in estimates of the coefficients of the implicit equations for
the lines (eq. 11) when the angle between the two lines is small.

Using Constraint from All Picture Cells

Images are noisy, and derivatives estimated from them noisier still, so a
velocity estimate based on measurements at just two image points will not
be very reliable. Since we have many picture cells in an image we can use a
least squares approach that adds up contributions from all places where
image derivatives can be estimated. That is, we can consider this a highly
overconstrained problem with many more equations than unknowns. We
assume here that u and v are the same for all points in the image—unlike
the situation in the general optical flow problem [1].
First, consider the continuous case. The integral

(uEx + vEy + Et )2 dx dy (16)
I

should be zero if we plug in the correct values for u and v—and if there
are no measurement errors. In the presence of inevitable noise in bright-
ness measurements, there will not be any values of u and v that make
the integral exactly zero. However, the values of u and v that minimize
the integral provide good estimates of the actual image velocity. Differ-
entiating w.r.t. u and v and setting these derivatives equal to zero, we
 6

obtain: 
2 (uEx + vEy + Et )Ex dx dy = 0
I
 (17)
2 (uEx + vEy + Et )Ey dx dy = 0
I
or     
u+ Ex2 Ex Ey v = − Ex Et
I I I
     (18)
2
Ey Ex u + Ey v = − Ey Et
I I I
These are two linear equations in u and v with integrals of products of
brightness gradients as coefficients4 .
We can write the two equations compactly in matrix-vector form as
Ms = b (19)
where s = (u, v) , M is the 2 × 2 matrix of coefficients and b has as com-
T

ponents the right hand side values of eq. 18. The pair of linear equations
for u and v in eq. 18 can be solved as long as the determinant
   2
2 2
Δ= Ex Ey − Ex Ey (20)
I I I
of the matrix M is non-zero. Then
s = M −1 b (21)
or in component form,
   
2
Δu = Ex Ey Ey Et − Ey Ex Et
I I I I
    (22)
2
Δv = Ex Ey Ex Et − Ex Ey Et
I I I I
It is easy to see that the determinant in eq. 20 is zero if the brightness
gradient has the same direction everywhere, that is if the ratio Ey (x, y) :
Ex (x, y) is constant. It can in fact be shown that this is the only situation
in which the determinant is zero.
This makes sense, since motion along isophotes does not produce any
change in image brightness and if the isophotes are all parallel, then mo-
tion of the whole image along that direction produces no change. Hence
such motions cannot be detected. The image has to have enough “two-
dimensional texture”—or “curvature of isophotes”—to allow us to esti-
mate image velocity (At the same time, motion in the direction of the
brightness gradient can be estimated).

4 The extremum found in this way is in fact a minimum, rather than say a max-
imum, since the integral in eq. 16 can be made as large is we like simply by
making u and/or v large—so there can’t be a finite maximum.
 7

Discrete Case

In practice, image brightness is known only on a discrete grid of picture

cells. In this case the double integrals in the above analysis can be replaced
with double sums over all the places where the brightness derivatives are
known. So we now wish to minimize
n−1 m−1
 2
uEx + vEy + Et dx dy (23)
i=1 j=1

In the sum, Ex , Ey , and Et are to be read as the estimates of these deriva-

tives at picture cell position (i + 1/2, j + 1/2), as discussed above. Note
again that in an image of size n × m (rows × columns), derivatives can be
estimated at (n − 1) × (m − 1) points.
By the way, the expression in eq. 23 is the weighted sum of the squares
of the perpendicular distances of the point (u, v) from the constraint lines
(defined by eq. 6) in image velocity space. The weights are the squares
of the magnitudes of the gradients—so contributions from picture cells
where the gradient is stronger are more heavily weighted.
The minimum of eq. 23 occurs at
n−1 m−1 n−1 m−1 n−1 m−1 n−1 m−1
2
Δu = Ex Ey Ey Et − Ey Ex Et
i=1 j=1 i=1 j=1 i=1 j=1 i=1 j=1
(24)
n−1 m−1 n−1 m−1 n−1 m−1 n−1 m−1
Δv = Ex Ey Ex Et − Ex2 Ey Et
i=1 j=1 i=1 j=1 i=1 j=1 i=1 j=1

where
⎛ ⎞2
n−1 m−1 n−1 m−1 n−1 m−1
Δ= Ex2 2
Ey −⎝ Ex Ey ⎠ (25)
i=1 j=1 i=1 j=1 i=1 j=1

To summarize, working with two successive image frames of n rows

and m columns, the algorithm is:

(i) Estimate Ex on a grid of (n−1) rows and (m−1) columns using eq. 9.
Use corresponding equations to similarly estimate Ey and Et ;

(ii) Accumulate the five sums of products needed in eq. 24 and eq. 25;

(iii) Find u and v from eq. 24, using Δ computed using eq. 25.

The method fails if the determinant is zero, which happens only if the
direction of the brightness gradient is the same at all points in the image.
 8

Stability and Accuracy

We have already seen that the equations cannot be solved for the image
velocity if the brightness gradient has the same direction everywhere in
the image. This occurs when
E(x, y) = f (ax + by) (26)
for some arbitrary function f (.). In this case, brightness is constant along
parallel straight lines defined by (ax + by) = c. Then
Ex (x, y) = af  (ax + by)
(27)
Ey (x, y) = bf  (ax + by)
and so the ratio Ey : Ex is constant.
Clearly there must be borderline cases where the determinant, while
not exactly zero, is quite small. In this case, while we can solve the equa-
tions, we cannot rely on the results, since the estimated image velocities
will be very sensitive to measurement errors. The reason is that in solving
for u and v we divide by the determinant. We conclude that one (crude)
measure of the quality of the result is the magnitude of the determinant
Δ of the coefficient matrix M. The larger the determinant, the smaller the
error in the estimated image velocity.
But this is not the whole story. It is possible, for example, that the
velocity component in one direction is quite well constrained while it may
not be in another direction. A single measure of quality cannot capture
this kind of detail. The question is how much the inverse M −1 amplifies
small errors in the right-hand side terms of the equations, and how that
amplification depends on direction.
This is controlled by the eigenvalues and eigenvectors of the real sym-
metric matrix M. Let λ1 and λ2 be the two eigenvalues, and let e1 and e2
be the corresponding (unit) eigenvectors5 . Then
Me1 = λ1 e1 and Me2 = λ2 e2 (28)
Note that the determinant equals the product of the eigenvalues:
Δ = λ1 λ2 (29)
The inverse matrix M −1 has the same eigenvectors—but its eigenvalues
are the algebraic inverse of the eigenvalues of M. That is
1 1
M −1 e1 = e1 and M −1 e2 = e2 (30)
λ1 λ2

5 Thetwo eigenvectors will be orthogonal if the eigenvalues are distinct. If the

eigenvalues happen to coincide, then the eigenvectors are not uniquely deter-
mined, but one can always pick two that are orthogonal.
 9

Now any vector b can be decomposed into a weighted sum of the two
eigenvectors (because they form an orthogonal basis):
b = α1 e1 + α2 e2 (31)
Then
α1 α2
M −1 b =
e1 + e2 (32)
λ1 λ2
We conclude that error components in b in the direction of the eigen-
vector with the smaller eigenvalue have a larger effect than error com-
ponents in the direction of the other eigenvector. Correspondingly, the
image velocity is less well constrained in the direction of the eigenvector
corresponding to the smaller eigenvalue.
So another measure of the quality of the estimated image velocity
is the magnitude of the smaller eigenvalue. The larger this is, the less
the error amplification. Since the determinant is the product of the two
eigenvalues, it also will be small when the smaller of the two eigenvalue
is small. But the determinant is also effected by the larger of the two
eigenvalues, so the magnitude of the smaller eigenvalue by itself—while
requiring more computation—is a more useful measure of quality.

Finding the Eigenvalues

The eigenvalues of the real symmetric matrix

 
a b
M= (33)
b c
are the roots of the characteristic polynomial6
λ2 − (a + c)λ + (ac − b2 ) = 0 (34)
Hence
(a + c) ± d
λ+,− = (35)
2
where 
d = (a − c)2 + 4b2 (36)
In our case,
n−1 m−1 n−1 m−1 n−1 m−1
a= Ex2 , b= Ex Ey , c= 2
Ey (37)
i=1 j=1 i=1 j=1 i=1 j=1
So a and c are non-negative, and so is d. Further λ+ ≥ 0, λ− ≤ λ+ and
|λ− | ≤ λ+ (38)
So we can easily find the smaller eigenvalue, the one that is a measure of
quality of image velocity estimation. It is only a bit more work to compute
than the determinant, Δ = ac − b2 , and it is more useful.

6 The characteristic polynomial is the determinant of the matrix (M − λI).

 10

Finding the Eigenvectors

If we wish to find the directions in which the image velocity is most con-
strained and least constrained, then we also have to determine the eigen-
vectors of the matrix M. To find the eigenvector corresponding to the
eigenvalue λ we solve the homogeneous linear equations
    
a−λ b x 0
= (39)
b c−λ y 0
Now
(a − c) ∓ d
a − λ+,− =
2
(40)
(c − a) ∓ d
c − λ+,− =
2
so solutions of eq. 39 can be written in either of the two forms
   
2b (a − c) ± d
or (41)
(c − a) ± d 2b
The magnitude squared of the first form is 2d(d ± (c − a)), while that of
the second is 2d(d ± (a − c)). Correspondingly, the unit eigenvectors can
be written in either of the forms
 
1 ±2b
√  (42)
2d d ± (c − a) d ± (c − a)
or  
1 d ± (a − c)
√  (43)
2d d ± (a − c) ±2b
In each occurence of the ± sign, the plus sign is chosen if the plus sign is
chosen in the expression for the eigenvalue (i.e. λ+ ). The direction with
the weaker constraint is that associated with λ− , while stronger constraint
on the image velocity exists in the direction associated with λ+ .

References
[1] Horn, B.K.P. & B.G. Schunck, “Determining Optical Flow,” Artificial In-
telligence, Vol. 16, No. 1–3, August 1981, pp. 185–203.
[2] Horn, B.K.P. “Parallel Networks For Machine Vision” Artifical Intelli-
gence Memo 1071, Dec 1988. also in Artificial Intelligence at MIT: Expand-
ing Frontiers, edited by Patrick H. Winston and Sarah A. Shellard, MIT
Press, Vol. 2, pp. 531–573, 1990, and in Research Directions in Computer
Science: An MIT Perspective, edited by A. Meyer, J.V. Guttag, R.L. Rivest,
& P. Szolovits, MIT Press, pp. 277-315, 1991. and in Vision Chips: Imple-
menting Vision Algorithms with Analog VSLI Circuits, C. Koch & H. Li (Eds.)
IEEE Computer Society Press, 1995.
 11

[3] Horn, B.K.P. and B.G. Schunck “Determining Optical Flow — a Retro-
spective” Artificial Intelligence, Vol. 59, No. 1–2, February 1993, pp. 81–87.

View publication stats