Python OpenCV: Epipolar Geometry

The general setup of epipolar geometry. The gray region is the epipolar plane. The orange line is the baseline, while the two blue lines are the epipolar lines. Often in multiple view geometry, there are interesting relationships between the multiple cameras, a 3D point, and that point's projections in each of the camera's image plane. The geometry that relates the cameras, points in 3D, and the corresponding observations is referred to as the epipolar geometry of a stereo pair. The standard epipolar geometry setup involves two cameras observing the same 3D point P, whose projection in each of the image planes is located at p and p' respectively. The camera centers are located at O1 and O2, and the line between them is referred to as the baseline. We call the plane defined by the two camera centers and P the epipolar plane. The locations of where the baseline intersects the two image planes are known as the epipoles e and e'. Finally, the lines defined by the intersection of the epipolar plane and the two image planes are known as the epipolar lines. The epipolar lines have the property that they intersect the baseline at the respective epipoles in the image plane. When the two image planes are parallel, then the epipoles e and e' are located at infinity. Notice that the epipolar lines are parallel to u axis of each image plane. An interesting case of epipolar geometry is shown in Figure 4, which occurs when the image planes are parallel to each other. When the image planes are parallel to each other, then the epipoles e and e' will be located at infinity since the baseline joining the centers O1, O2 is parallel to the image planes. Another important byproduct of this case is that the epipolar lines are parallel to an axis of each image plane. This case is especially useful and will be covered in greater detail in the subsequent section on image rectification. In real-world situations, however, we are not given the exact location of the 3D location P, but can determine its projection in one of the image planes p. We also should be able to know the camera's locations, orientations, and camera matrices. What can we do with this knowledge? With the knowledge of camera locations O1, O2 and the image point p, we can define the epipolar plane. With this epipolar plane, we can then determine the epipolar lines1. By definition, P's projection into the second image p0 must be located on the epipolar line of the second image. Thus, a basic understanding of epipolar geometry allows us to create a strong constraint between image pairs without knowing the 3D structure of the scene. The setup for determining the essential and fundamental matrices, which help map points and epipolar lines across views. We will now try to develop seamless ways to do map points and epipolar lines across views. If we take the setup given in the original epipolar geometry framework (Figure 5), then we shall further deneM andM0 to be the camera projection matrices that map 3D points into their respective 2D image plane locations. Let us assume that the world reference system is associated to the rst camera with the second camera oset rst by a rotation R and then by a translation T. This species the camera projection matrices to be:

M = K[I 0]   M' = K'[R T]

Now we find Fundamental Matrix (F) and Essential Matrix (E). Essential Matrix contains the information regarding translation and rotation, that describes the location of the second camera relative to the first in global coordinates. Fundamental Matrix contains equivalent information as Essential Matrix additionally to the knowledge about the intrinsics of both cameras in order that we will relate the 2 cameras in pixel coordinates. (If we are using rectified images and normalize the point by dividing by the focal lengths, F=E). In simple words, Fundamental Matrix F maps some extent in one image to a line (epiline) within the other image. This is calculated from matching points from both the pictures. A minimum of 8 such points is required to seek out the elemental matrix (while using the 8-point algorithm). More points are preferred and use RANSAC to urge a more robust result. So first we need to find as many possible matches between two images to find the fundamental matrix. For this, we use SIFT descriptors with FLANN based matcher and ratio test. Python3

import numpy as np
import cv2
from matplotlib import pyplot as plt
 
# Load the left and right images
# in gray scale
imgLeft = cv2.imread('image_l.png', 
                     0)
imgRight = cv2.imread('image_r.png',
                      0)
 
# Detect the SIFT key points and 
# compute the descriptors for the 
# two images
sift = cv2.xfeatures2d.SIFT_create()
keyPointsLeft, descriptorsLeft = sift.detectAndCompute(imgLeft,
                                                       None)

keyPointsRight, descriptorsRight = sift.detectAndCompute(imgRight,
                                                         None)
 
# Create FLANN matcher object
FLANN_INDEX_KDTREE = 0
indexParams = dict(algorithm=FLANN_INDEX_KDTREE,
                   trees=5)
searchParams = dict(checks=50)
flann = cv2.FlannBasedMatcher(indexParams,
                              searchParams)
 

# Apply ratio test
goodMatches = []
ptsLeft = []
ptsRight = []
 
for m, n in matches:
    
    if m.distance < 0.8 * n.distance:
        
        goodMatches.append([m])
        ptsLeft.append(keyPointsLeft[m.trainIdx].pt)
        ptsRight.append(keyPointsRight[n.trainIdx].pt)

Image Left Image Right Let's notice the basic Matrix. Python3

        
ptsLeft = np.int32(ptsLeft)
ptsRight = np.int32(ptsRight)
F, mask = cv2.findFundamentalMat(ptsLeft,
                                 ptsRight,
                                 cv2.FM_LMEDS)

# We select only inlier points
ptsLeft = ptsLeft[mask.ravel() == 1]
ptsRight = ptsRight[mask.ravel() == 1]

Next, we find the epilines. Epilines corresponding to the points in the first image is drawn on the second image. So mentioning correct images are important here. We get an array of lines. So we define a new function to draw these lines on the images. Python3

def drawlines(img1, img2, lines, pts1, pts2):
  
    r, c = img1.shape
    img1 = cv2.cvtColor(img1, cv2.COLOR_GRAY2BGR)
    img2 = cv2.cvtColor(img2, cv2.COLOR_GRAY2BGR)
    
    for r, pt1, pt2 in zip(lines, pts1, pts2):
        
        color = tuple(np.random.randint(0, 255,
                                        3).tolist())
        
        x0, y0 = map(int, [0, -r[2] / r[1] ])
        x1, y1 = map(int, 
                     [c, -(r[2] + r[0] * c) / r[1] ])
        
        img1 = cv2.line(img1, 
                        (x0, y0), (x1, y1), color, 1)
        img1 = cv2.circle(img1,
                          tuple(pt1), 5, color, -1)
        img2 = cv2.circle(img2, 
                          tuple(pt2), 5, color, -1)
    return img1, img2

Now we find the epilines in both the images and draw them. Python3

# Find epilines corresponding to points
# in right image (second image) and
# drawing its lines on left image
linesLeft = cv2.computeCorrespondEpilines(ptsRight.reshape(-1,
                                                           1,
                                                           2),
                                          2, F)
linesLeft = linesLeft.reshape(-1, 3)
img5, img6 = drawlines(imgLeft, imgRight, 
                       linesLeft, ptsLeft,
                       ptsRight)
 
# Find epilines corresponding to 
# points in left image (first image) and
# drawing its lines on right image
linesRight = cv2.computeCorrespondEpilines(ptsLeft.reshape(-1, 1, 2), 
                                           1, F)
linesRight = linesRight.reshape(-1, 3)

img3, img4 = drawlines(imgRight, imgLeft, 
                       linesRight, ptsRight,
                       ptsLeft)
 
plt.subplot(121), plt.imshow(img5)
plt.subplot(122), plt.imshow(img3)
plt.show()

Output: