The Photogrammetric Record 25(129): 24–41 (March 2010)

INTEREST OPERATORS FOR FEATURE-BASED

MATCHING IN CLOSE RANGE PHOTOGRAMMETRY
Ida Jazayeri ([email protected])
Clive S. Fraser ([email protected])
University of Melbourne, Australia

Abstract
Although automated object surface reconstruction via feature-based matching is
commonly employed in both stereoscopic close range and topographic photogram-
metry, it has rarely been used in conjunction with multi-image, convergent photo-
grammetric networks. A prerequisite for feature-based matching of distinct points
is the application of interest operators to provide the dense arrays of candidate fea-
ture points within the images forming the network. This paper describes an evalua-
tion of three interest operators, the Förstner, SUSAN and FAST algorithms, the aim
being to assess which is optimal for feature-based matching in convergent close
range photogrammetry. Following a brief review of the development of interest oper-
ators, a description of the three operators is provided, with the recently developed
FAST operator being discussed in more detail. The merits of image pre-processing
via the Wallis filter are also outlined, after which the performance of the interest oper-
ators is experimentally assessed within an eight-image network on the basis of accu-
racy of interest-point localisation, detection rate and speed. The findings of the
evaluation are that, of the Förstner, SUSAN and FAST operators, the FAST, which
has not been employed to a significant extent in photogrammetry to date, is optimal
for feature-based matching in multi-image close range networks.

Keywords: close range photogrammetry, feature-based matching, feature extraction,

interest operators, object reconstruction

Introduction
The automated reconstruction of objects from convergent images remains a
challenging problem in computer vision and photogrammetry. Three-dimensional modelling
is required in applications as diverse as industrial vision metrology, accident scene
reconstruction, object identification and heritage recording. Such applications call for a
methodology that yields high geometric accuracy with a complete modelling of object detail.
Automated reconstruction methods currently in existence are generally limited in their
application and do not yield high accuracy for non-targeted objects. Some applications in 3D
modelling involve the creation of visually pleasing models that focus on the photorealism of
results, rather than upon the metric quality achieved. These models are used for applications
where accuracy is not critical, such as animation and some types of cultural heritage recording.

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.
Blackwell Publishing Ltd. 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street Malden, MA 02148, USA.
 The Photogrammetric Record

The research reported in this paper, which centres on an investigation into the use of interest-
point operators to support feature-based matching in multi-image close range photogrammetry,
focuses upon high quality dimensional characterisation.
Image-based modelling has been defined by Remondino (2006b) as a complete process
that starts with image acquisition and ends with an interactive 3D virtual model. The
photogrammetric approach to creating 3D models frequently involves all the following phases:
image pre-processing, image scanning for feature point detection, network orientation
including camera self-calibration, feature-based matching and triangulation for unsignalised
points, and final bundle adjustment to complete the photogrammetric measurement phase. This
can then be followed by 3D mesh generation and texturing, and then visualisation and analysis.
Feature-based matching techniques can use different types of image features as the entities
to be matched. These include edges, surfaces, contours and distinct points. This paper will
address image scanning for feature point detection, and specifically an evaluation of three
different interest-point operators to support feature-based matching in convergent, multi-image
network configurations.
As a result of a broader investigation of interest operators that have either been employed
or are potentially applicable to close range photogrammetric object reconstruction, three were
chosen for further evaluation based on their performance, and especially taking into account
attributes of speed, feature detection rate/completeness and localisation accuracy. These three
algorithms, the Förstner, SUSAN and FAST operators, are described in subsequent sections,
following a short general introduction to interest operators.
Image enhancement has long been recognised as a valuable pre-processing step that
precedes the application of interest-point detection for feature-based matching, and in the
research reported in this paper the Wallis filter has been employed in an experimental testing
programme in which the Förstner, SUSAN and FAST operators were evaluated to ascertain
which was optimal for feature-based matching.

Interest Operators
Interest operators are algorithms that detect features of interest in an image, such as
corners, edges or regions. For high-accuracy 3D object reconstruction, interest operators are
required as a preliminary step in the surface measurement process in that they detect features
that serve as suitable corresponding points when matching across multiple images. Interest
operators were first developed in the 1970s and since then a number of different algorithms
have been developed and presented. For more complete accounts of the range of interest
operators that have been proposed, the reader is referred to Schmid et al. (2000), Remondino
(2006a) and Rodehorst and Koschan (2006). As could be anticipated, no single algorithm has
been universally accepted as optimal for all applications.
As illustrated in the development timeline shown in Fig. 1, one of the first interest
operator algorithms was developed by Hans P. Moravec in 1977 (Rodehorst and Koschan,
2006), who introduced the concept of ‘‘points of interest’’ and concluded that these points can
be used to find matching regions in consecutive image frames. The Moravec operator has
subsequently been used in various photogrammetric applications including 3D object
reconstruction (Ohdake and Chikatsu, 2005). This operator is limited, however, in that it
has a poor repeatability rate and is sensitive to noise. The poor repeatability is a consequence of
the anisotropic nature of the algorithm and any rotation in the images will result in a different
set of interest points being found.
The Harris/Plessey operator (Harris and Stephens, 1988) was developed 11 years later in
an attempt to address the limitations of the Moravec operator and generate a more desirable

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. 25
Jazayeri and Fraser. Interest operators for feature-based matching in close range photogrammetry

Fig. 1. Development timeline for selected interest operators.

feature point detector. It has proven to be more accurate and has also been shown to find more
interest points with improved localisation and a higher repeatability rate, but at a greater
computational cost. In spite of the Harris/Plessey operator being computationally intensive, it
has been adopted within the photogrammetric community, mainly for its simplicity (for
example, El-Hakim, 2002; Remondino, 2004).
The Förstner operator (Förstner and Gülch, 1987) has been widely adopted in
photogrammetry and computer vision over the past two decades and it is currently used in
many 3D object reconstruction applications (for example, Rottensteiner, 1996; Grün et al.,
2004; Remondino, 2004; Mayer and Reznik, 2007). It was developed with the aim of creating
a fast operator for the detection and precise location of distinct points, including corners and
centres of circular image features, with photogrammetric image matching applications in mind.
The SUSAN operator was developed in the mid-1990s by Smith and Brady (1997) in an
attempt to create an entirely new approach to low-level image processing. It too has been
adopted in photogrammetry (for example, Diener et al., 1998; Wendt, 2004). The SUSAN
algorithm is an accurate, noise-resistant and fast edge and corner detector that addresses some
limitations of other operators, such as high computation time.
The FAST interest operator, which was only recently developed by Rosten and
Drummond (2006), was designed as a high-speed feature detector for application in real-time
frame-rate digital photogrammetry and computer vision. It employs a high-speed algorithm
that is computationally simple and effective for real-time applications. The FAST operator
has been found generally to outperform other interest operators in speed, being faster than
the Harris, SUSAN and SIFT interest operators commonly used in imaging applications
(Rosten and Drummond, 2006). Beyond its speed, the FAST operator is also invariant to
rotation and changes in scale, a critical element for image-based modelling from convergent
images. This attribute of the FAST algorithm enables the operator to perform better than
many preceding algorithms, including the SIFT operator (Lowe, 2004), which has been
shown to have accuracy limitations when applied to convergent imagery (Sotoodeh et al.,
2008).

Förstner Operator
An overview of each of the three interest operators to be investigated is now provided,
commencing with the Förstner operator. The Förstner algorithm identifies interest points, edges
and regions using the autocorrelation function A. The derivatives of A are computed and
summed over a Gaussian window. Error ellipses are computed, and based on the size and shape
properties of each ellipse the interest features found are classified as points, edges or regions.
The size and shape of the error ellipse is calculated using two eigenvalues k1 and k2, as well as
the inversion of A, with the size being determined by

26  2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.
 The Photogrammetric Record

1 detðAÞ
w¼ ¼ ; w > 0: ð1Þ
k1 þ k2 traceðAÞ
The error ellipse shape (roundness of the ellipse) is obtained as
 
k1  k2 2 4  detðAÞ
q¼1 ¼ ; 0  q  1: ð2Þ
k1  k2 traceðAÞ2
Based on the values of w and q, the algorithm classifies each area as follows (Rodehorst and
Koschan, 2006):
(a) Small circular ellipses define a salient point.
(b) Elongated error ellipses suggest a straight edge.
(c) Large ellipses mark a homogeneous area.
An interest point is only precisely located when the given threshold values wmin and qmin are
exceeded. Practical experience with the Förstner operator has suggested that one of its less
desirable features is relatively high computation time, since it requires complicated
implementation and is generally slower than other detectors (Remondino, 2006a, 2006b).

SUSAN Operator
The SUSAN operator is based on the concept that each point of interest in the image will
have associated with it a local area of similar brightness values and that these areas can be used
as a guide to help find features of interest such as corners and edges in the image (Smith and
Brady, 1997). The operator searches for areas of similar brightness, and consequently for
interest points within a weighted circular window. The algorithm denotes the central pixel in
the search window as the nucleus. The area within the window that has similar intensity values
to the nucleus is computed and referred to as the USAN (Univalue Segment Assimilating
Nucleus). A low value for the USAN indicates a corner since the central pixel would be very
different from its surroundings. After assessing results and eliminating outliers, the local
minima of the SUSANs (smallest USAN) remain as valid interest points. The comparison
between pixel brightness values is computed using the following equation:

1 if jIð~rÞ  Ið~
r0 Þj  t
cð~ r0 Þ ¼
r;~ : ð3Þ
0 if jIð~rÞ  Ið~
r0 Þj >t
Here,~r0 is the position of the nucleus in the two-dimensional image,~ r the position of any other
point within the circular window, Ið~ rÞ the brightness value of any pixel, t the brightness value
threshold and c the output of the comparison. The comparison is calculated for each pixel in the
circular window and the total number of pixels with similar brightness values as the nucleus is
summed by
X
r0 Þ ¼
nð~ cð~ r0 Þ:
r;~ ð4Þ
~
r

r0 Þ value is compared with a geometric threshold, g. The algorithm

In the next step, the nð~
uses a threshold value in order to distinguish between features that make suitable interest
points, and non-suitable features. To find a corner in the image, the threshold value g is set
to half of the maximum value of nð~ r0 Þ, nmax. If nmax is less than the threshold g, then a
corner exists.

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. 27
Jazayeri and Fraser. Interest operators for feature-based matching in close range photogrammetry

FAST Operator
The FAST algorithm was developed by Rosten and Drummond (2006) as a high-speed
feature detector with strong repeatability properties suited to real-time frame-rate applications.
Early application of the FAST operator indicated that it could fully process live PAL video, at
the interest-point detection stage, at full frame rate using less than 7% of the processing time
available, in comparison to the Harris detector (120%) and SIFT operator (300%). It was
further found by the developers that the FAST operator outperformed existing feature detectors
in the area of reliability, namely, in its ability to detect the same corresponding interest points
from different viewpoints. This is an essential attribute for 3D object reconstruction
applications.
The algorithm functions in a similar way to the SUSAN algorithm in that the detector
examines a small patch in an image and assesses whether or not it ‘‘looks’’ like a corner. A
circular window is scanned across the image and the intensity values of the pixels within or
around the window are compared to that of the central pixel. It has already been noted that the
SUSAN algorithm is efficient and fast, and in contrast to other interest operator algorithms,
there is no necessity with the SUSAN operator to calculate a second derivative in the
computational process as only a small number of pixels are examined for each detected corner.
The development of the FAST operator grew out of research that aimed to improve the speed
and invariance to rotation, transformation and changes in scale of the SUSAN operator. In
comparison to the SUSAN detector and other operators that follow a similar computation
process, the FAST operator is indeed proving to be faster and more reliable and it is thus being
adopted in a variety of applications including real-time frame-rate imaging (Smith et al., 2006).
Also, inductive machine-learning methods that extract patterns from large data-sets are
employed in the FAST algorithm for corner detection. The algorithm operates by considering a
circle of 16 pixels around the corner candidate p, as illustrated in Fig. 2. An interest point is
deemed to exist when a set of n contiguous pixels in the circle are all brighter than the intensity
of the candidate pixel Ip plus a threshold t, or all darker than Ip £ t.
For each location on the circle x 2 {1 … 16}, the pixel at that location relative to p
(denoted p fi x) can have one of the three states:

Fig. 2. Sixteen-pixel circle used by FAST operator.

28  2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.
 The Photogrammetric Record

8
< d; Ip!x  Ip  t (darker)
Sp!x ¼ s; Ip  t < Ip!x < Ip þ t (similar) : ð5Þ
:
b; Ip þ t < Ip!x (brighter)
For each x, Sp fi x is computed for all p 2 P, where P represents the set of all pixels in all
training images. This divides P into three subsets Pd, Ps or Pb where each interest-point
candidate p is assigned a PSp fi x value.
In the next stage of computation, a Boolean variable Kp is used and assigned a true value
if p is an interest point, and a false value if p is a non-interest point. The algorithm used selects
an x-value which yields the most information about whether the candidate pixel is an interest
point. The x-value is selected by measuring the entropy of Kp. This begins with computation of
the entropy H(P) of K for the set P:
H ðP Þ ¼ ðc þ cÞ log2 ðc þ cÞ  c log2 c  c log2 c ð6Þ
where
  
c ¼  p  Kp is true  ðnumber of corners)
  
c ¼  p  Kp is false  ðnumber of non-corners):
The choice of x yields the following information:
H ðP Þ  H ðPd Þ  H ðPs Þ  H ðPb Þ: ð7Þ
Once the x-value yielding the most information has been selected, the process is applied
recursively on all three subsets and it only terminates when the entropy of a subset is zero; that
is, when all p-values in the subset have the same value as Kp they are all interest points or non-
interest points. The output of this process creates a decision tree which classifies all detected
points. The decision tree is then converted into C-code which is compiled twice for
optimisation and used as a corner detector.
With other interest operators, the final computation stage of the algorithm is generally to
compute a corner-response function, where non-maximal suppression can be applied, setting a
quality measure directly to each candidate corner. Non-maximal suppression sets the quality
measure for each point to zero if the value computed is not larger than the quality measure of
all points within a certain distance. The final interest points found, that is the remaining non-
zero points, form the resulting output.
The FAST interest operator does not compute a corner-response function and therefore a
non-maximal suppression cannot be directly applied to the resulting features. Instead, a score
function V is computed for each detected interest point and non-maximal suppression is applied
to remove points that have an adjacent point with a higher V. The score function is simply the
sum of the absolute difference between the pixels in the contiguous arc and the centre pixel,
being given by
0 1
X   X  
V ¼ max@ Ip!x  Ip   t; Ip  Ip!x   tA: ð8Þ
x2Sbright x2Sdark

This score value can be used as a quality control measure in post-processing to remove interest
points below a chosen threshold, keeping only high quality interest points with high V values;
that is, points with the highest absolute difference between pixels in the contiguous arc and the

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. 29
Jazayeri and Fraser. Interest operators for feature-based matching in close range photogrammetry

centre pixel. Since the score function values vary with different images and since the user
would not necessarily know which absolute values for V constitute a high score, a percentage
filtering can be applied. This allows the user to determine what percentage of points is to be
retained based on the scores of all detected points, so a 90% filter value will eliminate all but
the 10% of points with the highest score function values. This ensures that only the highest
quality interest points are retained for subsequent feature-based matching and photogrammetric
triangulation.
In order to measure the FAST operator’s repeatability, Rosten and Drummond (2006) used
a technique to test the feature detector’s ability to deal with affine warps under realistic
conditions. Repeatability was defined as the ability to detect the same real-world features from
multiple views. A 3D surface model was used in the test to compute where detected features
should appear in other views. The results indicated that the FAST operator can exhibit high
levels of repeatability under large variations in aspect and for different kinds of features. The
results also indicated, however, that the robustness of the FAST operator may suffer in the
presence of noise.

Image Enhancement via the Wallis Filter

An image pre-processing algorithm can be applied to enhance the images for subsequent
feature extraction. Research conducted within the reported project has found that a filter is
warranted to provide greater detail in shadowed and saturated areas, and thus to allow a greater
number of interest points to be detected.
The Wallis filter (Wallis, 1976) is a digital processing function that enhances the contrast
levels of an image. It is also applied in order to optimise images for subsequent image
matching (for example, Baltsavias, 1991; Baltsavias et al., 1996). Studies have found that
interest operators typically find more suitable points in imagery that has been pre-processed
with the Wallis filter (for example, Seiz et al., 2002; Ohdake and Chikatsu, 2005; Remondino,
2006b). The algorithm is adaptive and adjusts pixel brightness values in local areas only, as
opposed to a global contrast filter which applies the same level of contrast throughout an entire
image. The resulting image contains greater detail in both low- and high-level contrast regions
concurrently, ensuring that good local enhancement is achieved throughout the entire image.
After testing a number of smoothing filters, the authors found the Wallis filter to be a most
suitable choice for enhancement of performance of the three interest operators being evaluated.
The input of a target mean and standard deviation by the user is required for the Wallis
filter, which then adjusts local areas to match the target values accordingly. A weight value is
used to control to what extent the target values are used and how much of the original image is
kept. The general form of the Wallis filter is given by
iWallis ðx; yÞ ¼ ioriginal ðx; yÞr1 þ r0 ð9Þ
where
csoriginal
r1 ¼  starget ð10Þ
csoriginal þ
c
and
r0 ¼ bmtarget þ ð1  b  r1 Þmoriginal : ð11Þ
In equations (9) to (11), iWallis and ioriginal are the filtered and the original images, respectively;
r0 and r1 the additive and multiplicative parameters; moriginal and soriginal the mean and standard

30  2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.
 The Photogrammetric Record

deviation of the original images; mtarget and starget the user-specified target mean and standard
deviation values for the filtered output images; c the contrast expansion constant; and b a
weight parameter called the brightness forcing constant.
The Wallis filter first divides the input image into neighbouring square blocks of user-
defined size in order to calculate local statistics. The choice of block size is governed by the
amount of detail and the level of contrast that the user requires. Small blocks, for example,
5 · 5 pixels in size, result in a strong enhancement of small and typically unimportant features
which can be problematic in subsequent feature extraction and image matching procedures.
Conversely, large blocks, for example, 120 · 120 pixels in size, result in a significant loss of
detail. A medium-sized block of 31 · 31 pixels has been found to be small enough to recover
the required detail in the input image and large enough to appropriately filter out and ignore
unimportant features.
The original mean and standard deviation of the unfiltered image, moriginal and soriginal, are
calculated for each individual block based on pixel grey values, the resulting values being then
assigned to the central cell in each block. The corresponding values for all other cells in the
block are then calculated from this central cell by bilinear interpolation. In this way, each
individual pixel is assigned its own initial local mean and standard deviation based on
surrounding pixel values. The target mean and standard deviation values, specified by the user,
are employed to adjust the brightness and contrast of the input cells. The target mean value can
be set at 127 by default and the user may input any value between 0 and 255. Higher local
mean values than 127 will brighten the image, while an input of less than 127 will create a
darker output. The target standard deviation value is typically set at 60 by default and may be
assigned any value within the data range of the image. Higher values for this parameter result
in a greater contrast stretch, producing higher local contrast and greater detail throughout the
output.
The resulting Wallis-filtered image is a weighted combination of the mean and standard
deviation of the original image and the target mean and standard deviation values specified by
the user. The weight is determined by the brightness forcing constant b, which can take on any
value between 0 and 1; a weight value of 1 generates an output image equal to the Wallis filter
target values specified, while a weight of 0 will keep the original pixel values. A weight value
of b = 0Æ6, for example, will calculate an output image with 60% of the user-specified target
values and 40% of the original unaltered pixel values.
The contrast expansion constant c is a general contrast function that can take any value
between 0 and 1. Lower values such as 0Æ1 produce an overall grey image where only the
outlines of objects are visible, with very little contrast and detail. Values closer to 1 produce a
high contrast image with greater detail and distinct black and white areas.

Experimental Testing Programme

The experimental testing programme undertaken comprised two phases. The first assessed
the degree to which the Wallis filter enhanced interest operator performance and the second
compared the Förstner, SUSAN and FAST interest operators to ascertain which is optimal for
close range photogrammetric object reconstruction. In the first phase, a multi-storey building
was photographed and the images were filtered using the Wallis algorithm. The images were
recorded in the afternoon, when the building had the greatest variation in pixel brightness
values, with the west face being in direct sunlight and the south side being in the shade (in the
southern hemisphere). This proved to be a good testing ground to ascertain how well the Wallis
filter could simultaneously adjust both ends of the brightness spectrum to produce an enhanced
normalised image with better contrast and greater detail. The three interest operators were then

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. 31
Jazayeri and Fraser. Interest operators for feature-based matching in close range photogrammetry

applied to both the original and the Wallis-filtered images to compare the number of interest
points detected.
The second phase of the experimental programme involved an assessment of the
performance of the Förstner, SUSAN and FAST algorithms in the automatic 3D reconstruction
of a number of objects via multi-image matching of detected feature points in convergent
imaging networks. The objects ranged in size from a 20 cm Persian jewellery box to
architectural façades and structural features with dimensions of up to 10 m. One of these object
reconstruction tests will be reported on here, namely, an 80 cm diameter section from a
damaged aluminium liner plate, shown in Fig. 3. These thin liners are used as surface
protection layers in various engineering applications and surface deformation measurement is
required for quality control and monitoring, especially of tearing. The plate is shown in Fig. 3
surrounded by the coded targets which facilitated automatic network orientation and self-
calibration (Cronk et al., 2006). A piece of graph paper was affixed to the surface of the plate
simply to allow for a quantitative analysis of each phase of testing and to provide texture for
feature extraction.
The plate was imaged at a scale of close to 1:50 using a Nikon D200 camera in an eight-
station convergent network configuration, the average convergence angle for object points
being close to 45, as illustrated in Fig. 4. Whereas the standard error rXYZ of object point
determination exceeded 0Æ02 mm for the coded targets in the network orientation, that for
detected feature points subjected to matching was anticipated to be closer to 0Æ05 mm as a
consequence of the lower image coordinate measurement precision for feature points. A high
quality network exterior orientation is a prerequisite for successful feature-based matching for
3D surface reconstruction in such circumstances, for in spite of tight geometric constraints
within the image point correspondence determination process a large number of erroneous
matches are initially found owing to the high density of detected feature points. The final
bundle adjustment of all object points, targeted and feature points, can be effectively used to
filter out erroneous matches, which are also drastically reduced once the minimum number of
accepted rays to a matched point is set to four or more.
In this second phase of the evaluation of the three interest operators, three performance
criteria were addressed:
(a) Localisation: the accuracy of feature point positioning, in relative and absolute terms,
in both 2D image and 3D object space. The relative accuracy of the surface recon-
struction is the more straightforward to quantify, but in this case the gridded graph

Fig. 3. Deformed aluminium plate with coded targets for automatic network orientation.

32  2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.
 The Photogrammetric Record

Fig. 4. Camera station configuration for plate surface measurement.

paper covering the surface allowed an assessment of absolute positioning performance

as well, since grid intersections constituted distinctive feature points to be detected by
the interest operators.
(b) Feature detection rate: the number of true interest points found relative to the number
of points both missed and wrongly detected, that is to say the completeness of the
feature extraction operation.
(c) Speed: the computational efficiency of the algorithm.
In the context of photogrammetric modelling, localisation performance is generally of
most importance since this measures 3D point determination accuracy. The feature detection
rate is perhaps next in the line of importance since this is a determining factor for the density of
the resulting 3D point array. Speed, while of practical importance, is nevertheless generally a
secondary consideration except in near real-time photogrammetric measurements.

Results
Wallis Filtering
The testing of the impact of Wallis filtering on interest operator performance, using the
image of the building shown in Fig. 5, demonstrated that it can be a very beneficial pre-
processing function that enables the three operators to detect significantly more interest points.
Issues normally arising from changes in image contrast and illumination are largely overcome
when the Wallis filter is applied, leading to more repeatable and reliable results. Shadowed
areas are brightened and good local enhancement is achieved throughout the entire image, as
illustrated in Fig. 5(c). This allows the interest operator to detect suitable candidate points for

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. 33
Jazayeri and Fraser. Interest operators for feature-based matching in close range photogrammetry

(a) (b)

(c) (d)

Fig. 5. (a) Original image; (b) Wallis-filtered image; (c) enlarged area showing results of FAST operator applied to
the original image; and (d) enlarged area showing results of FAST operator applied to the Wallis-filtered image.

feature-based matching in all areas of the image. Figs. 5(c) and (d) show the interest-point
detection results of the FAST operator on an enlarged area of the building. As illustrated, the
algorithm efficiently detects many additional interest points in the Wallis-filtered image. The
results of this and other testing has indicated that the three interest operators can detect up to 20
times more interest points on Wallis-filtered images than on the original RGB images.

Interest Operator Performance

In applying the three interest operators to the images from the plate measurement network,
it was found that the FAST operator outperformed the SUSAN and Förstner operators across all
criteria: in terms of speed and detection rate, and in the localisation accuracy of final 3D
surface point positioning via feature-based matching. The FAST algorithm is indeed both fast
and robust and it yields good localisation and high point detection reliability, as illustrated in
both Fig. 6 and by the results presented in Table I. Shown in Fig. 6 are the interest points
detected by each of the operators in a selected 11 · 11 grid area on the plate surface.
Table I lists the speed and detection rate of each operator when applied to both the
original and Wallis-filtered images. On the original images, the FAST operator detected more
than 5 times the number of points found by the SUSAN operator and more than 15 times the
number of points found by the Förstner operator. These multiples are exceeded on the Wallis-
filtered images with the FAST operator finding up to 20 times more interest points than the

34  2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.
 The Photogrammetric Record

Fig. 6. Enlarged area of plate surface showing points extracted with: (a) Förstner; (b) SUSAN; and
(c) FAST operators.

Table I. Interest operator performance for images of the plate network.

Images Performance criteria Förstner SUSAN FAST
Original RGB Speed (s) 9 0Æ4 0Æ1
Detection rate (total points) 1800 4770 30 580
Wallis-filtered Speed (s) 7 0Æ8 0Æ3
Detection rate 10 600 6700 208 900

other two operators. The FAST operator also displayed speed advantages over both the SUSAN
and Förstner operators; the computation time of the FAST algorithm is a fraction of a second
for the eight images of the plate network, with over 30 000 interest points being found in each
of the original RGB images versus 200 000 or more points being detected in 0Æ3 s in the Wallis-
filtered images. The FAST operator exhibited very good localisation performance and it
detected relatively few erroneous points.
The very large difference in the detection rate for the Wallis-filtered versus original images
is further illustrated in Fig. 7, which shows the sevenfold increase in the number of interest
points found with the FAST operator, with the same parameter settings, when applied to one of
the images of the plate network.
Turning again to interest operator performance within the 11 · 11 graph paper grid patch
shown in Fig. 6, Table II lists the total number of points detected, points correctly detected,
points missed and points incorrectly detected relative to the total number. In this instance, the

Fig. 7. (a) Wallis-filtered image; (b) enlarged area showing FAST operator results on Wallis-filtered image; and
(c) FAST operator results on original image.

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. 35
Jazayeri and Fraser. Interest operators for feature-based matching in close range photogrammetry

Table II. Feature point detection results for the selected 121-point grid patch.
Förstner SUSAN FAST
Total points detected 34 (28%) 53 (44%) 109 (90%)
Correct points detected 32 (26%) 41 (34%) 107 (88%)
Points missed 89 (74%) 80 (66%) 14 (12%)
Incorrect points detected 2 (6%) 12 (23%) 2 (2%)
Localisation (pixels) 1 2 to 4 0Æ5 to 1
Localisation 0Æ006 (1 pixel) 0Æ012 to 0Æ024 0Æ003 to 0Æ006
(2 to 4 pixels) (0Æ5 to 1 pixel)

total is anticipated to be 121, with all grid intersections being detected as interest points. The
values listed represent averages across the eight images forming the plate network. Here the
FAST operator finds nearly all grid intersections (88% true interest points) with very few
erroneous points being detected (2% of total points), and with a low number of points missed
(12% of the total).
Table II also shows the 2D localisation accuracy of each operator, namely, how well it
positions each interest point relative to its visually assumed ‘‘true’’ position, which was measured
manually for each grid node. Given that the generally expected accuracy of feature-based
matching is around 0Æ3 pixels, sub-pixel 2D localisation of interest points is required for optimal
accuracy performance. For the Nikon D200 camera employed in the tests conducted, this
translates to a requirement of 0Æ006 mm or better, a level which is met by the Förstner operator and
surpassed by the FAST detector. Fig. 6 illustrates the high localisation accuracy of the FAST
operator within the selected patch. The Förstner operator also demonstrated very good
localisation, with few wrongly detected points. However, the algorithm also missed some 74%
of the grid intersection points. Whereas the SUSAN operator detected more points than Förstner, it
still missed 66% of the grid nodes and its localisation performance was the poorest of the three.

Impact on 3D Object Point Determination

Shown in Fig. 8 are the resulting 3D point clouds generated by the multi-ray feature-based
matching process for the aluminium plate survey, for the three interest operators. The number

Fig. 8. 3D point clouds of plate surface generated from interest points detected by: (a) Förstner; (b) SUSAN; and
(c) FAST operators.

36  2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.
 The Photogrammetric Record

of triangulated surface points in each case is 947 for Förstner, 1026 for SUSAN and 3805 for
the FAST operator. As will be discussed, in spite of the greater density of surface points for the
FAST operator case, these represent only a small percentage of the total number of interest
points detected.
It could be anticipated that the higher the 2D localisation accuracy, the higher the internal
accuracy in object space, since the better the localisation, the lower image coordinate
misclosure values within the network bundle adjustment. This behaviour was reflected to some
extent in the photogrammetric triangulation results, with the FAST operator producing the
smallest rms value of image coordinate residuals, at 0Æ28 pixels. Surprisingly, however, the
SUSAN operator produced a lower misclosure rms value than the Förstner operator, 0Æ32
versus 0Æ38 pixels. This suggests that with the SUSAN operator there may well have been
systematic bias in 2D localisation which is projectively compensated within the final bundle
adjustment.
Given the demonstrated superior performance of the FAST interest operator, as assessed
by all three performance criteria, the remaining discussion of 3D object point determination
will focus upon the results produced by this interest point detector. Results of the object point
determination via multi-image feature-based matching of interest points detected via the FAST
operator are summarised in Tables III and IV.
As discussed previously, a filtering function can be added to the FAST algorithm to limit
the number of feature points detected. The filter works by assessing the quality of each interest
point found, and eliminating all points below a user-defined quality threshold which is based
on the score function value V (equation (8)). The filtering function operates as a percentage
filter, which allows the user to determine the percentage of points to be retained, that is, if the
user selects a filter value of 80%, only 20% of detected points, those with the highest score
function values, will be kept. For each of four quality filter values, 80, 90, 95 and 99%,
Tables III and IV list the rms value of image coordinate residuals, the mean standard error of
object point coordinates, the corresponding relative accuracy, the number of points resulting
from the final bundle adjustment and the number of erroneous points. The erroneous points
comprise those which met initial geometric criteria set in the feature-based image matching, but
were rejected in the bundle adjustment. In all cases, the minimum number of imaging rays for
an object point was set at 4, the maximum possible being 8.
In applying feature-based matching to the eight images of the plate network, without any
quality filtering, some 64 000 points were found to meet the criteria of acceptable matches for
subsequent 3D determination. The results, as anticipated, were not acceptable, with possibly
60% or more of the 3D points constituting gross errors. When the FAST quality filter is
applied, the number of resulting surface points drops quite dramatically, as does the number of
erroneous points. Thus, it can be seen from Tables III and IV that retention of only 20% of the
candidate 2D FAST-detected interest points results in 1518 3D points in the original, unfiltered

Table III. Object point accuracy for network of original (unfiltered) images.
Quality Rms of image Standard error of Relative Number of Number of
filter coordinate 3D surface object 3D matched points erroneous
value (%) residuals points, point from final points
(pixels) rXYZ (mm) accuracy bundle adjustment
80 0Æ45 0Æ044 1:18 000 1518 36
90 0Æ33 0Æ040 1:20 000 895 4
95 0Æ28 0Æ038 1:21 000 504 0
99 0Æ28 0Æ029 1:26 000 193 0

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. 37
Jazayeri and Fraser. Interest operators for feature-based matching in close range photogrammetry

Table IV. Object point accuracy for network of Wallis-filtered images.

Quality Rms of Standard error Relative object Number of Number of
filter image coordinate of 3D surface point accuracy 3D matched points erroneous
value (%) residuals (pixels) points, rXYZ (mm) from final points
bundle adjustment
80 0Æ79 0Æ077 1:10 000 6798 3051
90 0Æ43 0Æ051 1:16 000 3826 177
95 0Æ29 0Æ048 1:17 000 2278 6
99 0Æ30 0Æ038 1:21 000 248 0

images, with only 36 of these being erroneous. The corresponding figures for the Wallis-
filtered case are 6800 and 3050, meaning that 50% of the points were still erroneous. However,
when the quality factor is set to 0Æ95, the resulting number of valid surface points for the Wallis
filter case falls to a still quite dense coverage of 2280, while the number of rejected points
drops to less than 10. The corresponding number of surface points for the unfiltered case is
lower at 500, but here there are no erroneous 3D point determinations. It can be seen that
retention of only the best 5% or so of the detected interest points can produce a very reliable
3D point cloud, as judged by the absence of point rejections in the final bundle adjustment
process.
As regards the impact on accuracy, as the quality tolerance on the FAST interest points
tightens, so the rms value of image coordinate misclosures in the final bundle adjustment
reduces, again as expected. This has a scaling effect on the a posteriori standard error rXYZ for
the 3D surface points, which in the case considered reached a level of between 0Æ03 and
0Æ05 mm for the best 5% of matched interest points. The corresponding proportional accuracy
range for the 0Æ8 m diameter aluminium plate segment was 1:16 000 to 1:26 000. The majority
of this variation was accounted for by differences in the overall image misclosure values (rms
of image coordinates) rather than by differences in ray intersection geometry. Again for the 5%
of retained interest points, the image coordinate residuals suggest a matching accuracy in
image space of about 0Æ3 pixels, which is consistent with feature-based matching accuracies
found in aerial triangulation and automated digital surface model (DSM) extraction from aerial
and satellite images.
Although higher accuracy can be achieved when more points are filtered, the resulting
point cloud can become too sparse to meet the requirements for high-accuracy mesh generation
in subsequent surface modelling. On the other hand, if less than 90% of the points are filtered,
too many erroneous points may result, thus reducing the final accuracy of the network. This is
highlighted by the 80% quality filter case in Table IV for which almost half of the 3D points
determined from the Wallis-filtered images are rejected as erroneous matches.

Concluding Remarks
The main aim of the research described in this paper has been to establish which of the
three interest operators, Förstner, SUSAN or FAST, is optimal for interest-point detection to
support feature-based matching in multi-image, convergent networks. Based upon the results
obtained, it can be concluded that the FAST operator outperforms the SUSAN and Förstner
operators in the three performance criteria of accuracy of localisation, detection rate and speed.
Moreover, the performance of the FAST operator is considerably enhanced when images are
pre-processed with the Wallis filter. Although a single measurement project has been
considered here, it being somewhat unusual given that the object surface was deformed,
gridded graph paper, a number of similar evaluations of the three interest-point operators have

38  2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.
 The Photogrammetric Record

been conducted on objects of varying geometric shape ranging from small to large. In all cases,
the findings have been in line with those reported here, namely, that the FAST operator
provides the best performance.
One of the main advantages of the FAST operator, its high detection rate, can also be a
drawback in that an excessively high number of feature points are detected, albeit in very little
time. The implementation of the quality filter is a practical means to limit the number of
retained interest points, but in networks of many images and dense point fields it is still
common to have a large percentage of erroneous matches, which then need to be filtered out in
the final bundle adjustment. With the image point correspondence algorithm of feature-based
matching used here being a purely geometric operation, there is some scope to employ
radiometric properties of the feature points as a means to better identify and eliminate
erroneous matches prior to bundle adjustment. The process envisaged is somewhat akin to
intensity-based matching, except the aim is to find radiometrically dissimilar image patches
(around the feature points). Of course such a strategy is compromised to some extent when the
imaging geometry is complex. Nevertheless, the proposal to integrate radiometric parameters
into the multi-image, feature-based matching will be considered within the broader aim of
developing a robust automatic 3D object reconstruction process suited to application with
convergent multi-image photogrammetric networks, in which there need be none of the
constraints upon imaging geometry that apply to intensity-based image matching.

references
Baltsavias, E. P., 1991. Multiphoto geometrically constrained matching. Ph.D. dissertation No. 9561, Institute of
Geodesy and Photogrammetry, ETH Zurich. 221 pages.
Baltsavias, E. P., Li, H., Mason, S., Stefanidis, A. and Sinning, M., 1996. Comparison of two digital
photogrammetric systems with emphasis on DTM generation: case study glacier measurement. International
Archives of Photogrammetry and Remote Sensing, 31(B4): 104–109.
Cronk, S., Fraser, C. and Hanley, H., 2006. Automated metric calibration of colour digital cameras. Photo-
grammetric Record, 21(116): 355–372.
Diener, H., Kothe, U., Ristow, B., Schreyer, M. and Stelbe, U., 1998. ERSO—acquisition, reconstruction
and simulation of real objects. Proceedings of the 24th Annual Conference of the IEEE, Aachen, 4: 2419–2424.
El-Hakim, S. F., 2002. Semi-automatic 3D reconstruction of occluded and unmarked surfaces from widely
separated views. International Archives of the Photogrammetry, Remote Sensing and Spatial Information
Sciences, 34(5): 143–148.
Förstner, W. and Gülch, E., 1987. A fast operator for detection and precise location of distinct points, corners
and centres of circular features. Proceedings of ISPRS Intercommission Conference on Fast Processing of
Photogrammetric Data, Interlaken, Switzerland. 437 pages: 281–305.
Grün, A., Remondino, F. and Zhang, L., 2004. Photogrammetric reconstruction of the Great Buddha of
Bamiyan, Afghanistan. Photogrammetric Record, 19(107): 177–199.
Harris, C. and Stephens, M., 1988. A combined corner and edge detector. Proceedings of the Fourth Alvey Vision
Conference, Manchester, UK. 302 pages: 147–151.
Lowe, D. G., 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer
Vision, 60(2): 91–110.
Mayer, H. and Reznik, S., 2007. Building façade interpretation from uncalibrated wide-baseline image
sequences. ISPRS Journal of Photogrammetry and Remote Sensing, 61(6): 371–380.
Ohdake, T. and Chikatsu, H., 2005. 3D modelling of high relief sculpture using image based integrated
measurement system. International Archives of the Photogrammetry, Remote Sensing and Spatial Information
Sciences, 36(5/W17): 6 pages (on CD-ROM).
Remondino, F., 2004. 3-D reconstruction of static human body shape from image sequence. Computer Vision and
Image Understanding, 93(1): 65–85.
Remondino, F., 2006a. Detectors and descriptors for photogrammetric applications. International Archives of the
Photogrammetry, Remote Sensing and Spatial Information Sciences, 36(3): 49–54.
Remondino, F., 2006b. Image-based modelling for object and human reconstruction. Ph.D. dissertation No.
16562, Institute of Geodesy and Photogrammetry, ETH Zurich. 174 pages.

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. 39
Jazayeri and Fraser. Interest operators for feature-based matching in close range photogrammetry

Rodehorst, V. and Koschan, A., 2006. Comparison and evaluation of feature point detections. Proceedings of
5th Turkish–German Joint Geodetic Days (Eds. L. Gruendig & M. O. Altan). Technical University of Berlin,
Germany. 8 pages.
Rosten, E. and Drummond, T., 2006. Machine learning for high-speed corner detection. Proceedings of the
European Conference on Computer Vision, Graz, Austria. 639 pages: 430–443.
Rottensteiner, F., 1996. Three dimensional object reconstruction by object space matching. International
Archives of Photogrammetry and Remote Sensing, 31(B3): 692–696.
Schmid, C., Mohr, R. and Bauckhage, C., 2000. Evaluation of interest point detectors. International Journal of
Computer Vision, 37(2): 151–172.
Seiz, G., Baltsavias, E. P. and Grün, A., 2002. Cloud mapping from ground: use of photogrammetric methods.
Photogrammetric Engineering and Remote Sensing, 68(9): 941–951.
Smith, P., Reid, I. and Davison, A., 2006. Real-time monocular SLAM with straight lines. Proceedings of British
Machine Vision Conference, Edinburgh, UK. Volume 1, 437 pages: 17–26.
Smith, S. M. and Brady, J. M., 1997. SUSAN—a new approach to low level image processing. International
Journal of Computer Vision, 23(1): 45–78.
Sotoodeh, S., Gruen, A. and Hanucsh, T., 2008. Integration of structured light and digital camera image data
for the 3D reconstruction of an ancient globe. International Archives of the Photogrammetry, Remote Sensing
and Spatial Information Sciences, 37(B5): 367–372.
Wallis, K. F., 1976. Seasonal adjustment and relations between variables. Journal of the American Statistical
Association, 69(345): 18–32.
Wendt, A., 2004. On the automation of the registration of point clouds using the metropolis algorithm. Inter-
national Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 35(B3): 106–111.

Résumé
Bien que la reconstruction automatique de surfaces par appariement à base
d’objets soit couramment employée pour la photogrammétrie stéréoscopique aussi
bien rapprochée que topographique, elle a rarement été employée avec des blocs
photogrammétriques convergents multi-images. Un prérequis pour l’appariement de
différents points à base d’objets est l’application d’opérateurs d’intérêt destinés
à fournir des grilles denses de points candidats au sein des images qui forment le
bloc. Cet article présente une évaluation de trois opérateurs d’intérêt, c’est à dire les
algorithmes de Förstner, SUSAN et FAST, dans le but d’identifier le meilleur pour
l’appariement à base d’objets en photogrammétrie rapprochée convergente. Les
bénéfices du prétraitement d’images par le filtre de Wallis sont aussi soulignés, et les
performances des opérateurs d’intérêt sont évaluées expérimentalement sur un bloc
de 8 images selon des critères de précision de localisation des points d’intérêt, de
taux de détection et de vitesse. Les résultats de cette évaluation sont que parmi les
opérateurs de Förstner, SUSAN et FAST, ce dernier bien que peu employé jusqu’à ce
jour en photogrammétrie, est optimal pour l’appariement à base d’objets dans des
blocs d’images rapprochées.

Zusammenfassung
Obwohl die automatische Oberflächenrekonstruktion mit merkmalsgestützter
Zuordnung üblicherweise in Nahbereichs- und topographischen Anwendungen der
Stereophotogrammetrie eingesetzt wird, wird sie selten für Mehrbildaufnahmen in
konvergenten photogrammetrischen Netzen eingesetzt. Grundlage für die
merkmalsgestützte Zuordnung mit ausgeprägten Punkten ist die Anwendung von
Interestoperatoren zur Schaffung einer dichten Anzahl von möglichen
Merkmalspunkten in den Bildern, die das Netz formen. Dieser Beitrag stellt eine
Bewertung dreier Interestoperatoren (Förstner, SUSAN und FAST) vor, mit dem Ziel

40  2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd.
 The Photogrammetric Record

der optimalen Wahl für die Bildzuordnung in konvergenten, photogrammetrischen

Nahbereichsaufnahmen. Nach einer kurzen Übersicht der Entwicklung von
Interestoperatoren, werden die drei Operatoren beschrieben, wobei der kürzlich
entwickelte FAST Operator detaillierter beschrieben wird. Die Vorteile von
Bildvorverarbeitung mit dem Wallis Filter werden skizziert, gefolgt von der
Leistung der Interestoperatoren, die experimentell mit Hilfe eines Netzes von acht
Aufnahmen geprüft wird. Als Kriterien werden hierzu die Genauigkeit der
Punktlokalisierung, die Detektionsrate und die Geschwindigkeit untersucht. Die
Ergebnisse der Untersuchungen zeigen, dass von den drei Operatoren Förstner,
SUSAN und FAST, der FAST Operator, der bis dato nicht im großen Ausmaß in der
Photogrammetrie eingesetzt wird, optimal für die merkmalsgestützte Zuordnung von
Punkten in Mehrbildaufnahmen der Nahbereichsphotogrammetrie ist.

Resumen
Aunque la reconstrucción automática de superficies de objetos mediante la
correspondencia de caracterı́sticas se ha empleado comúnmente en la fotogrametrı́a
topográfica y estereoscópica de objeto cercano, raramente se ha empleado con las
multiimágenes de redes fotogramétricas convergentes. Uno de los prerrequisitos de
dicha correspondencia requiere aplicar previamente operadores de interés para
obtener conjuntos densos de candidatos en las imágenes que forman la red. En este
artı́culo se hace una evaluación de tres operadores de interés, los algoritmos de
Förstner, SUSAN y FAST, con el objeto de determinar el óptimo para la fotogrametrı́a
convergente de objeto cercano. Tras una breve revisión del desarrollo de los
operadors de interés, se describen los tres operadores, examinándose más
atentamente el recientemente desarrollado operador FAST. También se describen
las ventajas del preprocesado de imágenes mediante el filtro Wallis y, tras su
aplicación, se evalúan experimentalmente los resultados de los operadores de interés
utilizando una red de ocho imágenes en base a la exactitud en la localización de los
puntos de interés, y la tasa y velocidad de detección. Los resultados de la evaluación
indican que, de los tres operadores Förstner, SUSAN y FAST, este último, el menos
utilizado hasta la fecha en la fotogrametrı́a, es óptimo para la correspondencia
basada caracterı́sticas en redes de multiimágenes de objeto cercano.

 2010 The Authors. Journal Compilation  2010 The Remote Sensing and Photogrammetry Society and Blackwell Publishing Ltd. 41