Optical Flow

Giuseppe Catalano Alessio Gallace Bomi Kim Francesco Santoro Bomi Kim March 23, 2009 Sergio Pedro

Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.1 Motion and motion eld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Optical Flow 1.1 Mathematical formalisation . . . . . . . . . 1.1.1 Optical Flow methods . . . . . . . . 1.1.2 Lucas-Kanade optical ow . . . . . . 1.1.3 Spatial and Temporal derivatives . . 1.1.4 Temporal derivative . . . . . . . . . 1.1.5 Fast optical ow estimation . . . . . 1.2 Implementation . . . . . . . . . . . . . . . . 1.2.1 Lukas-Kanade based implementation 1.2.2 Fast evaluation implementation . . . 1 2 4 4 6 6 7 8 8 10 10 11

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

Introduction
In the eld of computer vision motion detection has a relevant importance. By using information contained in a stand image, we can obtain a lot of informations about what we are observing, but there is no way to automatically infer what is going to happen in the immediate future. On the other hand a sequence of images provide informations about movement of depicted objects. Theres a plenty of techniques to recognize movement in a sequence, some based on feature and pattern recognition, some other based just on pixels, regardless what they mean for a human being. Some of these are Block Matching analysis and Optical Flow esimation; the latter will be object of study in the next sections.

0.1

Motion and motion eld

Figure 1: Velocity projection over image surface When an object is moving in space in front of a camera theres a corresponding movement on the image surface. Give the point P0 and its projecton Pi , by knowing its velocity V0 , we can nd out the vector Vi representing its movement in the image (g. 1). Given a a moving rigid body we can build a motion eld by computing all the motion vectors Vi (img. 2). The motion eld is a way to map the movement in a 3D space, on a 2D image taken on camera: for this reason, since we loose a dimension, we cannot exactly compute motion eld, but just approximate it. Moreover, when considering the movement of an object we have to consider also the movement of the camera, called ego motion: the most reliable estimee of the motion is build up boht this movements, objects and cameras. Here we will not consider ego motion, assuming that we 2

CONTENTS

Figure 2: Example of motion ow have a xed camera, as well as we are not going to consider changing in the illumination and shadowing.

Chapter 1

Optical Flow
Optical ow is a phenomena we deal with every day. It is the apparently visual motion of standing objects as we move through the world. When we are moving all the world around us seems to move in the opposite way: the faster we move, the faster it does. This motion can also tell us how close we are to the dierent object we see. The closer they are, the faster their movement will appear. There is also a relationship between the magnitude of their motion and the angle between the direction of our movement and their relative position to us: if we are moving toward an object, it will appear to stand still, but it will become larger as we get closer (this phenomena is also called FOE, focus of expansion); rather, if the object we are looking at is beside us, it will appear moving faster. In computer vision there are a lot of optical ow estimation techniques applied in elds as behaviour recognition or video surveillance; though they are blinder than pattern recognition based methods, there are fast enough implementations that allow us to build soft real time applications.

1.1

Mathematical formalisation

Optical ow methods try to compute optical ows by using two images taken at time t and t + t; since they deal with the Taylor series, theyre called dierential methods, as they work with the spatial and temporal derivatives. Given an image X, we express the intensity of each of its voxel as it follows:

I(x, y, t) We can now assume that the image intensity of each visible scene point is unchanging over time, so we can state the so called image constraint equation:

I(x, y, t) = I(x + x, y + y, t + t)

(1.1)

CHAPTER 1. OPTICAL FLOW Assuming the movement to be small the 1.1 can be developed in Taylor series as it follows I(x + x, y + y, t + t) = I(x, y, t) + I I I x + y + t + H.O.T. x y t

(1.2)

where H.O.T. means higher order terms, which can be ignored. From the 1.2 and the 1.1, it follows that: I I I x + y + t = 0 x y t that is, by dividing by t, the following: I x I y I t + + =0 x t y t t t which results in I I I Vx + Vy + =0 x y t (1.3)

where Vx and Vy are the component of velocity of optical ow associated with the considered I I voxel, respectively laying on the Xaxis and on the Y axis, and and x , y and I are the t derivatives of the image at (x, y, t). These three measurement can be written also as Ix ,Iy and It , so the 1.3 will become: Ix Vx + Iy Vy = It or, equivalently I T V = It (1.5) (1.4)

Figure 1.1: An example when optical ow is dierent from motion eld We have one equation but two variables, that means we need some other constraints. For this reason optical ow sometimes doesnt correspond to the motion eld. This is the so called

CHAPTER 1. OPTICAL FLOW

aperture problem and we can understand it better by watching at g. 1.1: since the cilinder is rotating, if we consider just the black bars, it would be impossibile to determine whether theyre moving horizontally (as they do), or vertically, as detected by optical ow. It is impossibile to determine the real movement unless the end of the bars become visible.

1.1.1

Optical Flow methods

Since we have to determine system, we can use dierent methods that add some constraint to the problem, in order to estimate the optical ow. Some of them are: Block-based methods - minimizing sum of squared dierences or sum of absolute dierences, or maximizing normalized cross-correlation Dierential methods of optical ow estimation, based on partial spatial and temporal derivatives of the image signal, as: Lucas-Kanade method - dividing image into patches and computing a single optical ow on each of them Horn-Schunck method - optimizing a functional based on residuals from the brightness constancy constraint, and a particular regularization term expressing the expected smoothness of the ow eld Buxton-Buxton method - based on a model recovered from the motion of edges in image sequences General variational methods - a range of modications/extensions of Horn-Schunck, using other data terms and other smoothness terms. Discrete optimization methods - the whole space is quantized, and every pixel is labelled, such that the corresponding deformation minimizes the distance between the source and the target image. The optimal solution is often computed through min-cut max-ow algorithms or linear programming. Here we are going to delve into Lucas-Kanade method.

1.1.2

Lucas-Kanade optical ow

The Lucas-Kanade optical ow estimation method is a dierential two frames algorithm, because it needs two frames in order to work. It introduces an additional hypothesis that allow us to have a fully determined system: given a pixel P , all the pixels in the neighborood of P have the same velocity as P . Thanks to this assumption considering a m m window (m > 1), centered at pixel P we can write the following:

Ix1 Vx + Iy1 Vy = It1 Ix2 Vx + Iy2 Vy = It2 . . . Ixn Vx + Iyn Vy = Itn

CHAPTER 1. OPTICAL FLOW

As we can see there are more than two equations (that would be sucient, in case of not singular system, to nd out the solution), so we have an over-determined system. Hence: Ix1 Iy1 It1 Ix2 Iy2 It2 Vx (1.6) . . V = . . y . . . . . Ixn or Av = b (1.7) A possibile solution could be given by the least squares methods, as shown in the following: AT Av = AT (b)orv = (AT A)1 AT (b) or Vx = Vy
2 Ixi Ixi Iyi

Iyn

Itn

(1.8)

Ixi Iyi 2 Iyi

Ixi Iti Iyi Iti

(1.9)

with the sums running from i=1 to n. In other words, given a window W , we can rewrite the 1.4 as follows: f (u, v) = (Ix u + Iy v + It )2 dxdy (1.10)

where we integrate with respect to the dimension of W . Since we assume that velocity is the same all over W , that is u and v dont depend upon dx nor dy, we can write the following system:

u u

2 Ix dxdy + v

Ix Iy dxdy +
2 Iy dxdy +

Ix It dxdy = 0 Iy It dxdy = 0

Ix Iy dxdy + v

That is easily computable.

1.1.3

Spatial and Temporal derivatives

In this section we are going to discuss some methods to compute derivatives needed in order to estimate optical ow. As previously stated we need to compute the spatial derivatives of an image I at the time t and the temporal derivative between time t and t + t.

Sobel Operator A widely used technique for spatial derivatives computing is the convolution by using the Sobel Operator. It is a dierential operator computing an approximation of the gradient of the image intensity and its very fast to apply since its based on a small window (3 3 kernel) to convolve with the whole image. It is also used to detect edges because by using it well obtain some white where the intensity changes abruptly, and black when it does not have any changes. We refer to two matrices, on for the gradient over the X axis, and one for the gradient over the

CHAPTER 1. OPTICAL FLOW

Figure 1.2: An image and its spatial and temporal derivatives Y axis; given an image I, we are going to convolve it two times in order to obtain Gx and Gy , that are the images containing the approximation of spatial derivatives. +1 +2 +1 +1 0 1 1 1 0 0 0 A and Gx = +2 0 2 A Gy = (1.11) 8 8 1 2 1 +1 0 1
1 The coecient 8 has the purpose to smoothen the derivative in such a way that the peaks of the derivative function are lowered.

1.1.4

Temporal derivative

Though it can be calculated by using the Sobel operator as well, the fastest technique is to do directly the subraction between the frame taken at time t and that taken at time t + t.

1.1.5

Fast optical ow estimation

Another useful technique used to estimate optical ow regardless of the magnitude of motion vectors is the fast optical ow evaluation developed by Mitsubishi Electric Research Laboratory. It enumerates all the possibile movement for each pixels and then computes the vectorial sum in order to nd out the most likely direction for that pixel. The algorithm follows: Subtract the current frame from the previous one, yielding a temporal dierence image. For pixels where the temporal dierence is nonzero, enumerate the possible motion directions consistent with the local image measurements. We derive two rules for motion

CHAPTER 1. OPTICAL FLOW direction estimation:

If the temporal dierence is negative, the motion direction is toward the adjacent pixel with higher luminance in the current frame. If the temporal dierence is positive, the motion direction is toward the adjacent pixel with lower luminance in the current frame. Apply the 1D direction estimation rules to four orientations (vertical, horizontal, and the two diagonals) at each pixel. Treat each possible motion direction estimate as a vector and average the possible motion estimates at each pixel Finally, average the above ow estimate at each pixel with the ow estimates of each of its eight neighbors.

Figure 1.3: Fast optical ow: walk through

CHAPTER 1. OPTICAL FLOW

10

1.2
1.2.1

Implementation
Lukas-Kanade based implementation

As previously described the rst step of the algorithms is computing the spatial and temporal derivatives.

Figure 1.4: Lukas-Kanade algorithm Activity Diagram Then we have to build up the linear system and nd out the velocity of each pixel by solving it. There are several parameters to tune in order to achieve good results and to avoid noise: Neighborhood window: since Lucas-Kanade hypothesis is that all of the pixesl in a m m window have the same velocity components, this parameter allow us to choose how big is the windows size and, as direct consequence, the density of optical ow vectors. Temporal derivative threshold: since between two consecutive frames there can be noise, we can choose how much big has to be the temporal derivative of a pixel to consider it. If a pixel has a temporal derivative lesser than this threshold, it would be set to 0 (that means that pixel has not moved over the time) Gaussian smoothing: this is a matrix w w wide, where w is the neighborhood window, and its used in order to weight the sums in equation 1.9 such that the components far from the central pixel count less than those that are closer. gauss[i][j] = j w 1 ( (i k 1)2 + (j k 1)2 e ) 2 3 2 2 (1.12)

where sigma has the meaning of the standard deviation in the gauss function (g. 1.5). By raising sigma well observe a attening in the gaussian bell such that farther component

CHAPTER 1. OPTICAL FLOW

11

Figure 1.5: Gauss function in 2D and 3D space from the central pixel would cout more. The smaller is sigma, the higher would be the center of the bell and the lesser would be the importance of pixels far from the central one. Testing and results Results are shown in g. 1.6 and 1.7. The former shows optical ow when people taken in the smaller pictures are walking forward, while the latter depicts vectors taken out from people walking backward. Actually there are some numeric problems because of the algorithm deals with very small numbers and very big numbers and so when multiplied between them they can give out some stability problems. This demeanor is mainly found when working with images where spatial derivatives are big.

1.2.2

Fast evaluation implementation

A second implementation has been done by exploiting the fast optical ow evaluation algorithm. This was modied and adapted to deal just with simple binary images and to do blob tracking. In order to detect blobs (the so colled objects in g. 1.8), weve implemented the grass re algorithm [8], to label each pixel according to which object it is part of. The following are the main steps of the algorithm: For each pixel, save in a matrix the dierence between the current luminance value and the previous one; Compute the direction of movement by using the information gathered at the previous step. Thus, for each pixel, if the dierence (calculated in the previous step) is less than 0, the movement is in the direction of the neighbour with higher luminance. If its bigger than 0, the movement is in the direction of the neighbour with lower luminance; Given all the possibile movement for a pixel, we sum all of the motion vectors to nd out the resulting motion of that pixel;

CHAPTER 1. OPTICAL FLOW

12

Figure 1.6: L-K results 1 Find out blobs (objects) by using the grass re algorithm; Compute the average movement for each object by averaging all the movements of all of its pixels. Using the open CV library, draw the arrows that represent the movement of each object. Testing and results

CHAPTER 1. OPTICAL FLOW

13

Figure 1.7: L-K results 2

CHAPTER 1. OPTICAL FLOW

14

Figure 1.8: Flow chart of the fast evaluation / blob detecting implementation

Figure 1.9: Fast Eval / Blob detection results 1

CHAPTER 1. OPTICAL FLOW

15

Figure 1.10: Fast Eval / Blob detection results 2

Bibliography
[1] http : //en.wikipedia.org/wiki/Lucas%E2%80%93Kanadem ethod. [2] http : //en.wikipedia.org/wiki/Opticalf low [3] http : //www.cs.northwestern.edu/ sco590/vision/hw1.pdf [4] http : //microservo.altervista.org/documenti/Relazione4/optical%20f low.htm [5] http : //homepages.inf.ed.ac.uk/rbf /CV online/LOCALC OP IES/OW EN S/LECT 12/node4.html [6] http : //www.centeye.com/pages/techres/opticf low.html [7] Moeslund, T., (2008). Image and Video Processing. Aalborg: Aalborg University. [8] Freeman, W., (1998). Comptuter Vision for Interactive Computer Graphics.Mitsubishi Electric Research Laboratory.

16