86 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL 11, NO 1.

FEBRUARY 1995

A Survev of Sensor Planning in Commter Vision U I

Konstantinos A. Tarabanis, Peter K. Allen, Member, IEEE, Roger Y . Tsai

Abstract-A survey of research in the area of vision sensor plan-

ning is presented. The problem can be summarized as follows:
Given information about the environment (e.g., the object under
observation, the available sensors) as well as information about
the task that the vision system is to accomplish (i.e., detection of
certain object features, object recognition, scene reconstruction,
object manipulation), develop strategies to automatically deter-
mine sensor parameter values that achieve this task with a certain
degree of satisfaction. With such strategies, sensor parameters
SENSOR I
values can be selected and can be purposefully changed in order PLANNING
to effectively perform the task at hand. Sensory systems are then SYSTEM
able to operate more flexibly, autonomously, and reliably. This
problem has recently become an active area of study with a modelr
number of researchers addressing various aspects of the problem. .. camera pose
apical settings
The focus here is on vision sensor planning for the task of robustly
detecting object features. For this task, camera and illumination
parameters such as position, orientation, and optical settings are Fig. 1. Sensor planning for computer vision
determined so that object features are, for example, visible, in
focus, within the sensor field of view, magnified as required,
and imaged with sufficient contrast. References to, and a brief values are determined by often laborious and time-consuming
description of, representative sensing strategies for the tasks of techniques. Generally, a trial-and-error approach involving
object recognition and scene reconstruction are also presented. human interaction is taken. Sensor locations and settings are
For these tasks, sensor configurations are sought that will prove
chosen and then tested in order to verify whether they meet
most useful when trying to identify an object or reconstruct a
scene. the requirements of the task at hand. The resulting parameter
values are valid for only a specific setup and can potentially
/
become unsatisfactory when errors (e.g., robot inaccuracy) I
I. SENSORPLANNING /

alter the environment. Such procedures constitute a major

bottleneck in system installation and result in a design cost
A. Motivation
which often exceeds that of the equipment. As a result,

S ENSOR planning in computer vision is an emerging

research area that tries to understand and quantify the
relationship between objects to be viewed and the sensors
these applications are costly and have limited intelligence and
flexibility.
Sensing strategies are also required for vision systems that
observing them in a model-based, task directed way (see Fig. perform object recognition since a single sensor configuration
1). The importance of the viewpoint in this object-viewer may not always result in a sufficiently informative view. For
relationship is evident. It is the viewpoint, for the most instance, in the cases of occlusion and near-symmetrical or
part, that creates the quality of the resulting image [43], similar objects, more than one interpretation may be consistent
determining the feasibility of the vision task and facilitating with the sensory data. As a result, additional sensor configu-
its execution. Previous work in computer vision, however, has rations are needed from which observed details of the scene
generally placed an emphasis on the object being observed [26] can help recognize the unknown objects by disambiguating
assuming that the viewpoint is given, suitable for the task, and among multiple interpretations.
not controlled. The issues related to determining viewpoints In addition to the sensor parameters themselves, another
that will be most suitable for the vision task at hand have set of key parameters in reconfigurable vision systems that
received considerably less attention. This latter area includes can be controlled, and thus may need to be planned, are
questions such as “What should the observer pose be?’ or those associated with the illumination of the scene. If the
“What values should other observer attributes have?’ interaction between lighting and the object surface to be
Even in currently employed vision systems that observe imaged is carefully considered, the image quality can be
known objects in known poses (e.g., visual inspection, surveil- substantially improved, leading to more informative images
lance, or monitoring systems), appropriate sensor parameter and making later processing of the image much easier. For
Manuscript received April 21, 1992; revised June 28, 1993. instance, by planning the illumination in a visual inspection
K. Tarabanis and R. Y. Tsai are with the IBM T. J. Watson Research Center; application, features of interest can be made to appear more
Yorktown Heights, NY 10.598. prominent, noise due to extraneous features can be reduced or
P. K. Allen is with the Department of Computer Science, Columbia
University, New York, NY 10027. eliminated, and the accuracy with which an object is measured,
IEEE Log Number 9400866. can be increased. While techniques that vary the illumination
1042-296)(/95$04.00 0 199.5 IEEE
TARABANIS er al.: SURVEY OF SENSOR PLANNING IN COMPUTER VISION 87

(e.g., illuminator location, polarization of illumination, etc.) configuration of a machine vision system by selecting sensors,
have been employed in several vision systems, illumination illumination, optics, and image processing algorithms.
planning has been overlooked to a great extent. Since image Vision sensor planning is also useful for robot-controlled
acquisition is much less costly than the subsequent image vision systems in which cameras and light sources are mounted
analysis, it seems advantageous to dedicate computational on robot manipulators (e.g., [ l 11, 1831). In order for these
effort to determine appropriate values for both the viewpoint systems to perform their task reliably (e.g., vision-guided
and the illumination parameters. remote assembly/disassembly or manipulation of objects in
Sensor planning is pertinent to a number of areas of robotics space or in hazardous environments), selection of the proper
and computer vision that have been studied extensively in vision sensor parameter values is critical. The developed
the past. For instance, the general problem of task planning sensor planning techniques could be used to automatically
in robotics and its component areas of motion planning, position and orient the cameras and light sources as well as
grasp planning, and assembly planning can be viewed as to control the camera optics (e.g., controlling the zoom, focus,
different facets of the sensor planning problem. In addition, and aperture settings of programmable zoom lenses).
recent research has underlined the importance of an area Sensor planning techniques are also applicable to areas
very much related to sensor planning, namely, that of active such as the automatic synthesis of vision programs from
sensing. In active sensing sensor parameters are controlled task specifications and model information 1291, [401, [501.
in response to the requirements of the task 171, [70]. It has For example, a vision program to inspect an object can
been shown in 141 that active sensing can take ill-posed vision be automatically generated, to a certain extent, based on
problems and render them well posed through constraints sensor planning techniques that determine appropriate camera
introduced by the dynamic nature of the sensor. Also, by and illuminator poses, optical settings, and image processing
actively reconfiguring a sensor system its effective scope is algorithms.
increased, and as a result, a range of sensing situations can It is important to note that planning techniques developed
be accommodated. For example, the field of view of a mobile for vision sensors can prove useful in other areas of au-
observer is far more extensive than that of a stationary one, tomation, such as the automated machining and dimensional
while required object detail can be resolved by either reducing inspection of mechanical parts 1181, [67], 1711. For instance,
the viewer-object distance or, when possible, by modifying planning for accessibility of a surface in order to machine or
the associated lens settings. In order to purposefully alter probe it by tactile means is equivalent to planning for visibility
sensor configurations, active sensing requires a sensor plan- of this surface assuming orthographic projection. In these
ning component that determines appropriate sensor parameter latter domains of automation, parameters such as workpiece
values. and probe orientation, machine selection and cutter, or probe-
The goal of sensor planning is to automatically, rather than type selection are determined by the planning systems. Vision
manually, generate appropriate sensor configurations based on sensor planning techniques can also be used in the area of
any a priori known information that is often available. For computer graphics for the automatic generation of viewing
instance, the required geometric and physical information of specifications that result in scene renderings that achieve a
objects can be extracted from CAD/CAM models which are certain intent [64].
often available in today’s manufacturing environment. Camera By employing a sensor planning component in these appli-
and illumination models approximating their physical and geo- cation domains:
metric properties can also provide the planning system with the 1) The development cycle and thus the cost of a sensor
required sensor and illumination characteristics. The planning system is reduced since sensor configuration can be done
algorithms can use this model information and augment it with automatically;
knowledge regarding the functions the system is to perform. In 2) Sensor parameter values can be found that are robust
this way, the sensor-based system will be able to reason about (i.e., satisfying the task requirements at hand even in the
its own configuration in order to achieve the task at hand. presence of uncertainty);
3) Sensor parameter values can be determined that optimize
the sensor output with respect to a given task criterion,
B. Applications (e.g., camera and illuminator poses that enhance the con-
Several researchers have recognized the importance of such trast between task relevant features and the background);
a planning component for a new generation of automated 4) A sensor-based system can adaptively reconfigure itself
visual inspection systems 1561, 1781, 1841. In 1781 a system to accommodate variations in the workplace.
is defined to automatically generate dimensional measurement
applications from CAD models of parts. In 1561 a concept-
based inspection system is outlined in which a model-based C. Scope of this Survey
inspection system would be augmented with a generic set In an attempt to limit the scope of this survey, we have
of principles in order to determine appropriate and flexible focused on sensor planning for vision sensors (e.g., cameras,
behavior in new situations. The need for machine vision range finders, illuminators, etc.). Furthermore, within the area
development tools that will assist machine vision designers of vision sensing, we have tried to limit our study to systems
and allow less specialized labor to be employed is underlined using higher-level model-based approaches as opposed to
in [84]. The tool proposed would automatically generate the pixel-level active vision. We are concerned with finding a
88 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. 11, NO. I , FEBRUARY 1995

generalized viewpoint that includes sensor parameters other aspect graph representation of the object is used. The individ-
than camera placement (e.g., focus, aperture, illumination ual approaches following this general methodology basically
placement). Active vision systems that vary their parameters differ in the hypothesis evaluation metrics, the criterion for the
by an assumed a priori schedule-randomly or heuristically next sensing operation or the stopping condition.
rather than in a knowledge-based manner with an identifiable 1) The quality of the match between image and scene
selection criterion-is outside the scope of this survey. features is evaluated by Cameron in [16] using a cross-
Following the approach of Maver and Bajcsy [54], we can correlation measure. In the work by Hutchinson and Kak [37],
classify the approaches to the sensor planning problem by confidence in matches between model and scene features are
the vision task to be achieved or similarly by the amount based on the similarity of the area and the 3D shape attributes
of a priori known information about the scene. This allows of the surfaces.
us to identify three distinct areas of research: object feature 2) In forming the hypotheses, Magee [51] takes a theorem-
detection, model-based object recognition and localization, and proving approach to establish a symbolic correspondence
scene reconstruction. between scene and image points, while in Kim’s work, the
The first area, object feature detection, seeks to automat- correspondence between object images from one view to the
ically determine vision sensor parameter values for which next is determined by employing correspondence techniques
particular features of a known object in a known pose satisfy used in stereo. In the work by Grimson [30], a set of possible
particular constraints when imaged [6], [SI, [231, 1241, 1401, interpretations of the original sensory data is assumed to have
[45], [57]-1601, [62], [72], 1741, [75], [ S I . For example, been determined by some other means.
the features are required to appear in the image as being 3) The criterion for the next sensing operation is based
visible, in-focus, and magnified to a given specification. These on optimally verifying the current hypotheses according to
planning techniques draw on the considerable amount of a a metric. A very similar metric is employed in most of the
priori knowledge of the environment, the sensors, and the approaches. For example, in [37] this metric involves reducing
task requirements. Because the identities and poses of the the maximum uncertainty of the hypothesis set, while in [ 161,
viewed objects are known, the sensor parameters are usually [30], [44], [51] it entails viewing particular disambiguating
preplanned off-line and then used on-line when the object is features.
actually observed. Our own interest is in this first area, and Similar to the aforementioned sensor planning work for
this survey reflects this interest as well as our belief that it is object recognition, there has also been work [12], [13], [28],
an important emerging research area. In the remainder of this [33], [38], [40] in automatically determining strategies that
section references to, and a brief description of, the other two guide the search for matching image and scene features in
areas is presented. Step 2 of the hypothesize-and-verify paradigm. In this work,
The second area is concerned with developing sensing however, sensing configurations are not planned, but rather, for
strategies for the tasks of model-based object recognition whatever the sensing configuration happens to be, a strategy is
and localization [16], 1171, [30], [37], 1401, 1441, 1511. Here, given to match image and scene features based on this initial
sensing operations are chosen that will prove most useful when set of sensory data without often acquiring additional sensory
trying to identify an object or determine its pose. In this work, data.
the a priori known information about the world in the form of The third area addresses the problem of determining sensing
models of possible objects, sensor models, and information ac- strategies for the purpose of scene reconstruction [l], 1141,
quired to this point are compiled into recognitiodocalization [191, [211, 1311, 1531, [66], [81], [MI. In this case, a model
strategies (see Fig. 1). Most approaches in this area follow a of the scene is incrementally built by successively sensing the
common theme. Namely, a search is performed in the space unknown world from effective sensor configurations using the
of object identitieslposes employing the hypothesize-and-verify information acquired about the world to this point. At each step
paradigm: of the algorithm, new sensor configurations are chosen based
on a particular criterion (e.g., ability to explore the largest area
1) Hypotheses are formed regarding the object identities of unknown space). The sensory information acquired at each
and poses; step is then integrated into a partially constructed model of
2) These hypotheses are assessed according to certain met- the scene and new sensor configurations are generated until
rics; the entire scene has been explored. While there is no a priori
3) New sensing configurations are proposed based on a known scene information that can be used in this problem,
given criterion until a stopping condition is met. the iterative sensing is guided by the information acquired
Since after the first step in the above approach the identities to each stage. With respect to the planning component of
and poses of the objects in the scene have been hypothesized, this work, the various approaches differ in the criterion with
the sensor planning techniques of the first area are also which a new sensor configuration is chosen. The approaches
applicable in this second area. also differ in the way the multiple views are integrated into
In order to limit the search of sensor parameter space in this a scene model. The work described in [3], [34], [52] is
hypothesize-and-verify paradigm, a discrete approximation of related in that it addresses the problem of reconstructing
this space is commonly employed. For example in [30], sensor a scene from multiple views, however, the viewpoints are
configurations are only chosen from a known finite set. In [ 151, predefined (i.e., assumed to be given) and are not planned
[37] a discrete approximation of viewing space based on the in any manner.
TARABANIS al.: SURVEY OF SENSOR PLANNING IN COMPUTER VISION 89

11. SENSOR PLANNING FOR FEATURE

DETECTABILITY B. DeJning the Problem
This review now focuses on vision sensor planning where In this section we describe the problem of sensor planning
the sensing task is image acquisition for robust feature detec- for feature detectability in more detail. In Section I the problem
tion. The techniques that are discussed are mostly applicable to was summarized as automatically determining camera and
vision systems that observe known objects in known poses; for illumination parameters (e.g., position, orientation, settings)
example, in visual inspection systems, surveillance systems, that satisfy certain feature detectability task requirements (e.g.
or monitoring systems. visibility, focus, contrast) by drawing on the knowledge of
In the following sections, common themes are initially un- the environment, the sensors, and the task requirements them-
derlined in the various approaches in order to allow grouping selves. From this description, it can be seen that the problem is
and high-level contrast of methodologies. An in-depth analysis model based and task driven. The models include information
of the problem and a critical review of each approach follows. of the objects in the environment and the employed sensors.
This a priori knowledge, together with knowledge of the task
requirements that need to be satisfied, are incorporated into
A. Overview of Research in Sensor Planning the planning algorithms.
for the Task of Feature Detectability In the following sections, we discuss each of the above
This problem is fairly new in the area of computer vision mentioned components of the problem. These are
but has recently received considerable interest. Several vision 1) The sensor parameters,
planning systems are being developed that draw on the a
2) The sensor and object models, and
priori known information regarding the observed object and
3) The feature detectability constraints.
the employed sensors in order to automatically determine
vision sensor parameter values that satisfy certain feature I ) Sensor Parameters: The parameters that need to be de-
detectability constraints. The developed techniques differ in termined for vision sensors are basically associated with the
their general approach to determining sensor parameter val- point from which the scene is observed (i.e., viewpoint) and
ues. the point from which the scene is illuminated. Customarily,
Several systems [59], [601, [62], [63], [87], [88] take a a viewpoint is considered to be the viewer location alone.
generate-and-test approach, in which sensor configurations are However, it is useful to define the concept of a viewpoint in a
generated and then evaluated with respect to the task con- broader sense that includes not only the viewer orientation, but
straints. In order to limit the number of sensor configurations also the optical settings associated with the viewpoint. These
that are considered, the domain of sensor configurations is settings are also observer attributes affecting the resulting
discretized. The viewing space discretization in these systems image of the scene.
is performed by surrounding the object with a tesselated This broader definition, then, identifies two types of sensor
viewing sphere and limiting the possible sensor locations to parameters: geometric and optical. The geometric parameters
either groups or individual tessels on the sphere. Determination are independent of any of the lens optics. Parameters of this
of the sensor parameter values is formulated as a search over
type the following:
this discretized domain of sensor configurations guided by
task-related heuristics. The three positional degrees of freedom of the sensor
The sensor planding methods described in [5], [6], (r,y ? 2)-this parameter is the position vector of a point
[22]-[24], [72], [74], [75] take a synthesis approach. In this associated with the sensor. An example of a convenient
approach, the task requirements are characterized analytically point for this purpose is a point on the C-mount surface of
and the sensor parameter values that satisfy the task constraints the lens since the relationship between the C-mount and
are directly determined from these analytical relationships. the sensor does not vary. The characteristic points of the
There has also been work related to sensor planning in the lens itself (e.g., the principal points, the nodal points, the
area of sensor simulation systems [36], [41], [ W . In such pupils, etc.) can also be used. However, since these points
systems, a scene is visualized given the description of the are not rigidly fixed to the sensor, the relative movement
objects, sensors, and light sources. These systems provide a of these points as the optical settings of the lens vary
framework for planning of sensor configurations. For instance, must be accounted for.
by taking a generate-and-test approach, satisfactory sensor The three orientational degrees of freedom of the sen-
configurations can be found by creating a simulated view of sor-this parameter can be specified by the pan, tilt, and
the scene and evaluating the task constraints in the simulated swing angles of a vector rigidly attached to the sensor. An
image.
example of a convenient vector of this type is a unit vector
Finally, there has been related work that follows the expert
along the viewing direction (i.e., optical axis). While the
systems paradigm [8], [45], [57]. In such systems, expert
characteristic points of the lens change as the optical
knowledge of viewing and illumination techniques is incor-
settings of the lens vary, the viewing direction remains
porated into an expert system rule base. After acquiring
constant.
information regarding the particular object to be observed,
the expert system provides advice regarding the appropriate The optical parameters, on the other hand, depend on the
sensor configuration. lens settings. Parameters of this type include the following:
90 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. 1 1 , NO. I , FEBRUARY 1995

The back principal point' to image plane distance, &-this The spatial distribution of intensity-for example, the
parameter d is varied in the several focus control mech- above parameter is the most important characteristic of
anisms that are employed in lenses in order to achieve the illumination in the case of structured lighting.
a focused image. Also, d is the image distance that The spectral distribution of intensity-this parameter can
is employed in the Gaussian lens formula governing be varied by employing spectral filters with a certain
focusing of a lens. spectral transmittance.
The entrance pupil diameter2 , of the lens-the entrance Parameters describing the polarization of the illumina-
pupil diameter depends on the size of the aperture (con- tion-the polarization state of the illumination can be
trolled by the aperture setting of the lens) and the lens controlled by employing polarizers.
optics that image the aperture. Among other things, the Thus, in the most general case, planning of camera, lens, and
aperture, and hence the entrance pupil, affect the focus illumination parameters is to be done in a high-dimensional
and the brightness of the image. imaging space [6S]. As will be seen in Section 11-D, however,
The focal length f of the lens-the focal length is a only a subset of the above set of sensor and illumination
fundamental property of a lens and it provides a measure parameters is typically considered in any of the existing sensor
of its refractive ability. In cases where it can be varied planning systems. For example, in many systems the camera
(e.g., zoom lenses) the focal length is a sensor parameter; is assumed to be located at a fixed distance away from the
otherwise, it is a sensor constant. object or the optical axis is assumed to pass through a fixed
Other optical camera parameters that can also be controlled point on the object.
in some cases [6S] include the following: 2) Sensor and Object Models: The sensor models, camera,
1) The exposure time of the sensor, which determines how lens, and illuminator, embody information that characterizes
long the sensor will be exposed to light from the scene; the operation of the sensor. Such information includes the
2) The gain of the video signal by the camera amplifier; following:
3) Parameters adjusting the spectral responsivity of the The sensor and illumination parameters and con-
imaging system (e.g., the spectral transmittance of a stants-for example, the field-of-view angle of the sensor,
fi 1ter) . the sensor pixel size, the lens focal length, the sensor
noise, the spatial extent of illumination, its intensity
The illumination parameters are also of two types: geometric
distribution within this extent, etc.
and radiometric. The geometric parameters are independent
The governing relationships between the sensor param-
of any of the physical characteristics of the illumination.
eters and the sensor constants-for example, the per-
Parameters of this type include the following:
spective projection imaging model, the lens formula for
The three positional degrees of freedom of the illumina- focusing, etc.
tor-this parameter is similarly the position vector of a Object models that are commonly used (e.g., CAD/CAM
point associated with the illuminator. For example, in the models) contain geometric and topological information, which
case of a point light source, this parameter can obviously may be sufficient for some aspects of the sensor planning prob-
be the location of the point light source itself. lem. However, the photometric properties of the object, such as
The three orientational degrees of freedom of the illu- color and reflectivity of its surface, are needed for illumination
minator in the case of directional illuminational-this planning in order to accurately model the interaction between
parameter can be specified by the pan, tilt, and swing light and the object surface and then approximate the resulting
angles of a vector rigidly fixed to the illuminator. An image. In this case, currently used object models need to be
example of a convenient vector of this type is the unit augmented to include such information.
vector along the axis of symmetry of the illumination 3 ) Feature Detectability Constraints: The feature detectabil-
beam. ity constraints discussed in this section are fairly generic to
The geometric characteristics of the illumination beam- most vision tasks. An initial set of such constraints for sensor
for example, the illumination beam may have the shape planning was introduced by Cowan et al. in [22],[24]. Similar
of a conical solid angle with a constant or variable apex to the way sensor parameters have been classified in Section II-
angle. B. 1, the feature detectability constraints can also be collected
The radiometric parameters depend on the physical and optical into two groups depending on whether the illumination source
characteristics of the illumination. Parameters of this type plays a role in the constraint or not.
include the following: The first group consists of the purely sensor constraints for
The radiant intensity-this parameter describes the power which illumination is not a factor. Such constraints include
output of the light source. the following:
Visibility-for a feature to be detectable by the sensor,
it must first be visible to the sensor. This means that
'
The principal points of an optical system are two conjugate points on the all lines of sight from the camera to each point of the
optical axis such that planes drawn perpendicular to the optical axis through
these points are planes of unit magnification. For the case of a thin lens camera
feature are not obstructed (i.e., occluded) by anything in
model, the principal points coincide at a single point. the environment.
2The entrance pupil of a lens is the image of its aperture with respect to Field of view-while the visibility constraint requires that
all elements preceding it in the lens system. the rays of light from the feature reach the sensor, the
TARABANIS er al.: SURVEY OF SENSOR PLANNING IN COMPUTER VISION 91

field-of-view constraint requires that these rays must then edge feature may not be detected in the image when the
image the features onto the active area of the sensor. If contrast between its adjacent faces is not sufficient for
this does not happen, the image of the features will either the edge operator at hand.
be clipped by, or will lie outside, the active sensor area The above constraints can be further classified as geometric
and thus will not be observable. and optical constraints. Geometric constraints depend only on
Focus-the goal of this constraint is to guarantee that the geometric sensor and illumination parameters. Examples
the features are in focus. While there is only one object of such constraints are visibility and illuminability, where
distance for which points are in perfect focus, there is a these depend only on the geometric parameter of position.
tolerance in position for which a feature is still considered Optical constraints depend on both the geometric and the op-
acceptably focused based on the resolution of the image tical parameters. Examples of such sensor constraints include
sensor. This tolerance is referred to as depth of field. Thus resolution, focus, field of view, dynamic range, and contrast.
the focus constraint requires that the features of interest It is important to note that when the sensor and radio-
lie within the depth of field of the lens. metric constraints are formulated in terms of the sensor and
Magnification or pixel resolution-this constraint governs illumination parameters, the sensor constraints involve only
the size of the feature images. Often, a linear feature is sensor parameters, while in general the radiometric constraints
required to appear in the image with a certain length include both sensor and illumination parameters. For example,
(e.g., 5 pixels). This constraint is referred to as the the sensor parameter of aperture is involved in both the sensor
magnification constraint. A related constraint is that of constraint of focus and the radiometric constraint regarding
resolution for which any two points of the feature must the dynamic range of the sensor. This coupling of sensor
be imaged by distinct pixels on the sensor plane. and illumination parameters can also be seen in the case in
Perspective distortion-in many applications (e.g., graph- which the reflectance model of the object surface includes
ics) the images of a scene are considered undesirable a specular component or when lens collection’ is taken into
when the scene is severely distorted under the projec- account. In both cases, the sensor location is coupled into
tion. In such applications a constraint that minimizes the the image irradiance equation. This coupling results from the
perspective distortion of features is appropriate. bidirectional reflectance distribution function in the former
The second group of feature detectability constraints de- case, and in the latter from the fourth power law for the
pends on the illumination. These constraints are referred to cosine of the off-axis angle [35]. There are cases in which
such couplings can be neglected. For example, in the case of
as illumination or radiometric constraints and include the
following: a diffuse object, the scene radiance is the same irrespective of
the camera position. Similarly, if the field of view is narrow,
Illuminability-for a feature to be detectable by the the dependence of the image irradiance on the off-axis angle
sensor, it is not sufficient that it be visible to the sensor is negligible. In general, however, the sensor and illumination
alone. It is also necessary that the feature be visible to planning problems are not separable.
at least some point of the light source as well. If this is
not the case, the feature will not be illuminated and as a
C. Review of Work Taking the Generate-and-Test Approach
result, it cannot be detected. If a feature point is visible to
only some but not all points of the light source, it will be I ) The HEAVEN System: HEAVEN incorporates work by
illuminated by these points and will either be back-facing Sakane et al. [59], [61]-[63] in sensor and illumination plan-
or in shadow for the remaining part of the light source. ning and is the precursor to the V I 0 system that is discussed in
A related constraint is shadow avoidance. For example, the next section. HEAVEN uses a spherical representation that
such a constraint is important in photometric stereo (see is both simple and efficient to model sensor configurations. A
Section 11-C. 1) and-in cases where shadows cast over geodesic dome is created around the object, tesselating the
the feature-may be mistakenly interpreted as the feature sphere with an icosahedron that is further subdivided in a
itself. hierarchical fashion by recursively splitting each triangular
Dynamic range of the sensor-an object point will also face into 4 new faces. This process can be implemented at
not be detectable in the image when the image irradiance whatever level of detail is needed for the task at hand at
from that object point is outside the dynamic range of the the cost of increased processing time. The viewing sphere
sensor. For instance, the image irradiance from an object is centered on the object and its radius is equal to an a
point may be too weak to sensitize the corresponding priori chosen distance from the vision sensor to the target
photoreceptor cells and as a result will appear black in object. This serves as a simple resolution constraint (although
different features on the object will be imaged at different
the image, On the other hand, the image irradiance from
resolutions due to the variation in their distance and orientation
an object point may be too high and thus may saturate the
with respect to the sensor). See Figs. 2 and 3.
photoreceptor area. In both cases, the object point will be
Given this spherical representation, a method is needed to
considered undetectable.
determine the visibility of a target point from each facet of
Contrastdetectability of an edge feature in a scene is
the spherical dome. By projecting a ray from the target point
often determined by the edge contrast in the image, that
of the object (located at the sphere’s center) to the center of
is, the disparity in image intensity values at points in the
neighborhood of the image of the edge. For example, an 3Lens collection [35] relates scene radiance to image irradiance.
92 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. 1 1 , NO. I , FEBRUARY 1995

features, a method has been devised to take the convex hull of

the two regions on the geodesic dome that correspond to the
visibility of the endpoints as the occlusion-free region.
A distance method criterion has been developed to rank
each facet within an occlusion-free region. The idea is that
facets close to a border with an occluded region are possibly
poor choices for sensor location, as noise or small errors in
calibration may cause the occlusion-free facet to be occluded.
The distance used in this criterion is the negated inner product
of the ray from the center of the chosen facet to the center
Fig. 2 . A robot environment and a geodesic dome generated around a target of the nearest occluded facet. This simple test allows each
object in the HEAVEN system (from [ 6 2 ] ) . facet to be sorted by its distance transformation. To compute
this distance, an iterative neighbor growing scheme on the
tesselated sphere is used, propagating from the chosen facet
via each of its three adjacent neighbors, until all paths to the
Viewpoint
Selection Planning and occluded region are computed.
\ Evaluation
The final step in determining placement for a sensor (in
Focus of Attentlon this case, a camera mounted on a robot arm or "eye-in-hand"
configuration) is taken as the intersection of the candidate
facets (sorted by distance), any possible user-chosen locations,
and the workspace of the manipulator encoded as regions on
the geodesic dome also.
The occlusion-free regions for camera placement are also
regions where a point light source can be placed without
Tools
casting shadows on the target. This is made use of in HEAVEN
when light source position planning is performed for a photo-
Geodesic Dome Arm Solution fo metric stereo system. In order to address the case where the
Camera-in-hand
camera occludes the illuminator or vice-versa, the camera and
illuminators are also considered as part of the scene. Since
I)----
--- .. \
shadows caused by surrounding objects need to be avoided in
a photometric stereo system, HEAVEN first computes these
/ shadow-free facets and then ranks them based on two criteria:
Manipulator Modelbase
1) Reliability for recovering the surface normal vectors, and
Object Model 2 ) Detectability of the surface by the camera and the
Camera Model illuminators.
For the first criterion, the photometric stereo problem is
posed in a least squares setting and a singular value decom-
position analysis is applied. The reliability criterion of the
illuminator configuration Cl is formulated as the inverse of
the condition number of the matrix related to the light-source
Real World
positions:

where M , is the matrix that represents the unit vectors directed

Fig. 3. The HEAVEN system (from [ 6 2 ] ) from the object surface to the light sources. Such a criterion
favors the most stable light position in terms of accuracy in
estimating the surface normal vectors.
the facet being processed, all intersections of object faces with On the other hand, the detectability criterion C, is defined
the ray can be computed. Depth buffering can then be used to as the range of surface normal vectors that can be detected by
determine if the facet is occluded or not. This approach can be the photometric stereo setup. A viewing illuminating direction
quite costly, since its complexity is proportional to the product is considered to detect a surface normal vector if the two
of the number of object faces and the number of facets. To make an angle less than 90". (Similar detectability criteria
alleviate this, bounding box representations of the objects to are employed in the VANTAGE system that is discussed in
be tested are intersected with the pyramid volume consisting Section 11-F.1.) In addition to the three light source locations,
of the facet and the center of the dome. If these tests show the criterion C2 takes into account the camera position which
an intersection, then the test is applied to the bounding boxes is also controllable. C, is expressed as the area of intersection
of the next level of the recursive facet tesselation. For linear of four hemispheres on the spherical viewinghlluminating
TARABANIS ri al.: SURVEY OF SENSOR PLANNING IN COMPUTER VISION 93

surface. Each hemisphere corresponds to the range of surface Facets on the tesselated sphere that are associated with
normal vectors that can be detected if a camera or illuminator views containing the same edges, are grouped into regions.
is positioned at the center of the hemisphere. region is then evaluated based on the following criteria:
A composite criterion is formulated that combines C1 and Robustness against potential object placement iincer-
C2. Initially, each criterion is required to satisfy a certain tainty: number of region facets/total number of tesselated
minimum value: sphere facets;
Edge visibility; number of edges visible in a regionhotal
C1 > TH1; C2 > TH2. (2) number of object edges;
The composite criterion C is formulated as Edge occlusion: 1-(number of occluded edgedtotal num-
ber of object edges); and
c = tc, + (1 - t)Cz (3) Contrast: average facet contrast over all facets in this
region, where facet contrast is the normalized sum of all
where t is the weight that controls the relative importance
edge contrast values when the illuminator is placed at
of the criteria. The composite criterion is evaluated for all
this facet.
combinations of occlusion-free positions of the camera and
shadow-free position triplets of the illuminator. The setup with The suitability value associated with a region on the viewing
the highest value of the composite criterion is chosen by the sphere is calculated as a normalized sum of the above attribute
system. HEAVEN also incorporates planning of the window values. Regions that do not meet a specified threshold value
layout for a visual feedback control task by again formulating are eliminated. Each camera and illuminator location is given
the problem in a singular value decomposition framework. a suitability value equal to that of the region in which it lies.
2 ) The V I 0 System: The vision illumination object (VIO) The previous evaluation of camera and illuminator loca-
system, developed by R. Niepold and S. Sakane [60],plans tions does not consider the coupling between the two in an
the setup of both a camera and a point light-source. Given in- actual sensing situation. For this reason, the V I 0 system also
formation regarding the environment (i.e., the objects, sensors, evaluates the camera-illuminator setup as a pair. In order to
and illuminators) and the task at hand, their system determines limit the number of camera-illuminator combinations, only
1) The position of the camera, those that satisfy a set of constraints are considered (e.g.,
2 ) The position of a point light-source, the number of visible edges common to both the camera
3 ) A list of most suitable edge features to monitor on the and illuminator locations must exceed a preselected value, the
camera cannot be placed in front of the illuminator and vice
target object, and
4) A quantitative assessment of each sensor setup. versa). Camera-illuminator pairs that meet these constraints
are then ranked based on another set of attributes associated
The object features to be observed are not fixed in VIO, but
with image features and their combination in the scene. The
may vary as different features are more suitable in different
image feature attributes considered were
object-sensor-illuminator configurations. V I 0 also assigns to
each configuration a number describing the degree of its The length of the feature edge in the image,
suitability to the task at hand based on some chosen criteria. Its contrast,
The camera and illuminator positions are again taken to lie The angle formed between an object edge corresponding
on the surface of a sphere of a chosen radius and centered at to an edge in the image and an expected displacement
an object reference point. In addition, the camera optical axis direction in the case of location uncertainty of the target
is assumed to point at the object reference point. The V I 0 object, and
system thus considers two of the five (three positional and two The angular disparity of object edges: this angular dis-
rotational) degrees of freedom for camera placement and two parity is computed as the normalized sum of angles
of the three positional degrees of freedom of the illuminator. between all pairs of object edges corresponding to the
The objects are polyhedral with Lambertian reflectance. There feature edges in the image.
is no spatial variation of brightness within a polyhedral face All this information regarding a particular sensing situation
since the illumination source is taken as a point distant is collected in a V I 0 cell, which is a data frame representation
from the object, and interreflections and shadow effects are containing all the characteristics of a setup proposal. The
neglected. camera and illuminator suitability values, as well as the
Similar to the HEAVEN system discussed in the previous suitability value of their combination, are combined to obtain
section, sensor illuminator configurations in V I 0 are generated the global V I 0 processability value of the setup. Based on
on an adaptively tesselated spherical surface and evaluated this criterion, setups are ranked and the setup with the highest
according to the task criteria. V I 0 first calculates an image processability value is taken to be the best choice.
representation of the expected scene for each pair of camera In addition to stationary objects, V I 0 also plans a sensor
and illuminator locations, and then evaluates chosen image setup for moving target objects by taking into account the
feature attributes (e.g., edge visibility, edge contrast) to assess previous and future sensor configurations and attempting to
the goodness of each such pair. Threshold values are specified minimize changes in the camera or illumination setup as the
for each image feature attribute so that camera-illuminator target object moves.
pairs that do not meet this threshold condition are eliminated Overall, the V I 0 system has the advantage of providing
from further consideration. both sensor and illumination planning components in a unified
94 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. 1 I , NO. I , FEBRUARY 1995

framework. Furthermore, V I 0 generates a set of candidate CONTRAST

camera-illuminator setups along with a quantitative assess-
ment of each. However, by taking an all-possible-combinations
strategy at various stages of the algorithm, V I 0 may be
confronted by combinatorial problems. The many thresholds
that are employed to mitigate this problem may not work in
some cases as they are preselected in an ad hoc manner.
3 ) The ICE System: The illumination control expert (ICE)
system was developed at the University of Washington
[86]-[88] to achieve automatic sensor and illumination
placement planning for machine vision tasks.
Similar to both V I 0 and HEAVEN, ICE takes a generate- Fig. 4. Contrast criterion in ICE (from [SS]).
and-test approach. Camera and light positions are generated
and then evaluated based on selected criteria. In ICE, as in entire edge is evaluated on a finite number of points along the
V I 0 and HEAVEN, the camera points at an object reference edge and then fitted with piecewise continuous polynomials
point, and the camera and light source are placed on the producing what is defined as the contrast graph for this edge.
surface of a sphere with its center at the origin of the object This graph describes how contrast varies spatially along an
coordinate system. This spherical surface is then approximated edge and is used to determine the distribution of contrast.
by a discrete number of points in such a way that the distance The contrast distribution function is then used to evaluate the
between any two neighboring points is approximately the optimality criteria for illuminator placement that were itemized
same. above.
Overall, the ICE system places emphasis on the illumination On the whole, the ICE system has the advantage of pro-
placement planning component of the problem. For the camera viding an illumination planning system that deals with objects
placement planning component, the task constraint considered with both a general geometry and general reflectance char-
in ICE is edge visibility. The optimal sensor position is acteristics. For instance, objects bounded by curved surfaces
considered to be that from which certain feature edges appear and modeled by a hybrid reflectance model can be han-
least occluded. This criterion is evaluated based on the aspect dled. The reflectance model used in ICE is based on the
graph representation of the object [46], [69]. The regions of Torrance-Sparrow reflectance model [77] with modifications
camera positions that are found using this approach are such made to the specularly reflected component so that polarization
that the set of object edges of interest is either fully or partially can be taken into account. However, in the actual examples
visible. These sensor viewpoints are then assessed based on the given of ICE, the effect of polarization of the incident or
ratios of the unoccluded portion of these edges, as seen from a reflected light is not taken into account and thus its importance
particular viewpoint, to their actual edge lengths. This metric, for illumination placement planning is not clear.
however, varies within an aspect graph cell. Consequently, In ICE the sensor placement planning problem is addressed
the finer spatial subdivision (i.e., tesselation) of the spherical independently of that for illuminator placement. An optimal
viewing surface, discussed earlier, is used when evaluating camera location is found without considering where the light
this metric. source will be placed, and the light source placement prob-
The illumination planning problem is posed in an optimiza- lem is then posed given the sensor location. However, as
was discussed in Section 11-B.3, these two problems are not
tion setting with edge contrast as the criterion to be optimized.
generally separable. Even though specular reflection is taken
Illuminator locations are assessed based on
into account in ICE, the camera placement planning problem is
1) The ratio of the length of an edge for which a given posed independently of the illuminator location and, therefore,
contrast threshold is exceeded to the total edge length the solution obtained may be suboptimal for the combined
and sensor and illuminator planning problem.
2 ) The amount by which the predicted edge contrast ex- 4 ) Remarks on the Generate-and-Test Approach: The
ceeds this threshold over a certain percentage of the generate-and-test method is appealing as an initial solution
length of the edge. to the sensor planning problem because it draws from some
The above optimality criteria are evaluated by simulating well known techniques for discretizing spaces and efficiently
placement of the illuminator at points on the discrete approxi- searching them. Several positive attributes of this approach
mation of the spherical viewing surface (see Fig. 4), while the are listed below.
camera is placed at the one location that was independently 1) Generate and test uses a relatively straightforward ap-
determined. The contrast at an edge point is obtained from a proach to the occlusion problem, performing a search over the
prediction of the image based on the image intensity equation facets of the dome. It is simple to implement algorithmically.
and is computed as the difference between intensities of the 2 ) The tesselation of the sphere is a hierarchical procedure
reflected light from two neighboring small patches, one on that can be used to make search more efficient, since search
each of the faces that meet at this edge. The image intensities over this sphere is at the heart of the method.
are computed based on the illumination and sensor parameters 3) Intersecting multiple constraints posed as regions on the
and the bidirectional reflectance function. Contrast for the sphere to find a feasible region is straightforward.
TARABANIS ef NI.: SURVEY OF SENSOR PLANNING IN COMPUTER VISION 95

4) Lighting constraints can be modeled with the same

representation as visibility constraints.
There are a number of potential problems with generate
and test, however, that will require ongoing research. These
are listed below.
1) The computational cost of finely tesselating high dimen- Fig. 5 . From left to right, examples of the admissible regions for the
resolution, field-of-view, and depth-of-field constraints from [24].
sional parameter spaces and then searching them is high. While
the hierarchical approach can guide this search, it still may
require an exhaustive search (e.g., over all combinations of
facets and possible occluding objects). dimension higher than three. However, by considering a subset
2) The solution space assumes a viewpoint in the center of the sensor parameters, Cowan and Kovesi were able to
of each facet. This approximation may not be useful, for intersect admissible domains of individual constraints in three-
example, in characterizing an entire facet’s visibility. The dimensional space. For instance, the camera orientation is not
general question of scale (i.e., sampling rate) remains for these explicitly taken as a parameter to be planned, but rather is
methods. assumed to take on an implicit value (e.g., in the field-of-
3) The generate-and-test methods work well for point fea- view analysis, the camera viewing direction is taken from
tures; however, they do not extend easily to extended features the viewpoint to the center of the sphere circumscribing all
such as lines and planar faces. With more complex features, features of interest). Similarly, parameters associated with the
the method may become computationally burdensome as well lens optical settings are not planned for the most part. This
as overly conservative in its solutions. approach has the difficulty that the chosen values for such
4) A number of generalized viewpoint parameters are either parameters need to be at least admissible for all constraints.
ignored or assumed fixed in the generate-and-test approach. While a globally admissible value for the camera orientation
In particular, sensor resolution is posed as a global radius is determined, the image plane to perspective center distance
on the object feature, which causes difficulties with multiple is found for each constraint separately and not in a manner
feature observation. Field-of-view constraints are also difficult that guarantees satisfaction of all constraints.
since this encompasses either extended features or multiple For the visibility constraint, Cowan and Kovesi initially
features. In addition, camera orientation parameters are usually computed the three-dimensional region from where a convex
not solved for; they are assumed to be pointing to a designated target can be viewed entirely from above a convex occluding
location on the feature. polygon (the same approach is taken later in [49] in their
work in automatic grasping). For the case where the occluding
D. Review of Work Taking a Synthesis Approach object is not a convex polygon but rather a polyhedron,
An alternative to the discrete approach is to model the the total occluded region is obtained as the union of the
constraints as analytic functions. This is somewhat intimi- component occluded regions of the faces of the occluding
dating because the function spaces are of high dimension (8 polyhedron. The faces that are concave or contain holes
dimensions in the case of a generalized viewpoint that includes must be decomposed into convex shapes for their method
camera location, orientation, and the optical parameters of to be applied. Their algorithm for convex occluding and
focal length, aperture, and focus). However, by characterizing target polygons, as described, has quadratic computational
these spaces (or subspaces formed by restricting the number complexity in the number of edges, which can be improved
of parameters) as analytic functions, a number of benefits are to linear as discussed in [80].
derived. A general comment about the resolution, focus, and field-
I ) The Automatic Sensor and Illuminator Positioning Work of-view constraints is that although they are characterized
of SRI: An important early work in this area is the work of by functional relationships, iterative techniques are then used
Cowan and Kovesi [22], [24] in which camera locations were to solve for the sensor parameters, even when closed-form
automatically generated satisfying several of the geometric solutions are obtainable.
constraints discussed in Section 11-B.3; that is, chosen features For the resolution constraint, Cowan and Kovesi defined
of polyhedral objects are visible, in focus, within the sensor resolution to be the minimum angle subtended by a given
field of view, and spatially resolvable to a given specification. incremental surface length at the perspective center of the
This work was then later extended by Cowan and Bergman lens, rather than the minimum number of pixels per surface
[23] to include planning of illuminator placement. length which is commonly used. This definition can be shown
The general approach is to formulate both problems as con- to be more conservative than the one customarily used. In
straint satisfaction problems. Each task requirement generates addition, it has the advantage of formulating the resolution
an equivalent geometric constraint which in turn is satisfied in constraint independently of the viewing orientation since the
a domain of admissible locations in three-dimensional space angle subtended at the viewpoint does not depend on the view-
(see Fig. 5). The admissible domains obtained for each task ing orientation. As a result, the number of sensor parameters
requirement are then intersected in order to determine locations involved is reduced and the resolution satisfying domains of
that satisfy all constraints simultaneously. individual features can be intersected in a lower dimensional
As discussed in Section 11-B.3, the general camera and parametric space, when globally admissible resolution regions
illuminator planning problem involves a parametric space of are sought.
96 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. 1 I , NO. I , FEBRUARY 1995

Cowan and Kovesi also determined the permissible region planar surface with patches of different reflectance values
for camera placement such that all points of the features to was used as a test object. The camera was then placed in
be observed are in focus. They made use of a depth-of-field the position computed by the sensor planning system without
formula developed by Krotkov [48] that gives the range of illumination constraints. Given this location of the camera, the
distances of feature points that are in focus (i.e., their blur illuminator was moved to positions along the constraint curves
circle diameter is less than the minimum pixel dimension). of minimum and maximum brightness and the resulting images
The set of viewpoints for which a feature has its farthest point were consistent with the expected values (within roughly 5%)).
at a distance equal to the upper limit of this range, and its This work has recently been extended to include constraints
closest point at a distance equal to the lower limit defines on illumination for forming edges (gradients in images) [ 2 5 ] .
the boundary of the locus of admissible viewpoints for this By planning regions of acceptable viewing and illumination
constraint. Although these distances (as any object distances parameters to form edges, a feature based object recognition
governed by the Gaussian lens law) need be measured along strategy can be generated that plans new viewing locations
the optical axis, Cowan and Kovesi measured them radially, of an unknown object. The new viewing position is selected
that is, from the perspective center of the lens to each feature to maximize the number of features (edges) that can be
point. With such a distance definition they developed an seen, given certain strong assumptions about illumination and
iterative procedure to initially obtain the domain of viewpoints reflectance.
that satisfy the focus constraint for a line segment. For this, 2) The MVP System: The authors are developing a vision
it is assumed that the camera optical axis passes through planning system, machine vision planner (MVP), that automat-
the line segment midpoint. In three dimensions, a similar ically determines vision sensor parameter values that satisfy
iterative approach is used to solve the focus constraint at small several of the sensor constraints discussed in Section 11-B.3
increments of azimuth and elevation. The camera optical axis [72], [73], [75], [79], [SO]. MVP takes as input the object
is assumed to pass through the center of a circumscribing circle geometry information from a CAD database, as well as models
or sphere of the features depending, respectively, on whether of the camera and lens, and determines camera poses and
they are coplanar or not. As a result, the orientational degrees settings for which features of interest of polyhedral objects
of freedom of the sensor are omitted and the discretization of are visible (occlusion-free positions of the sensor), contained
sensor configuration space migrates from a genuine synthesis entirely in the sensor field of view, in focus, and resolvable
approach. The sensor planning system has been tested on a by the sensor to a given specification. At this stage the MVP
piece of sheet metal with a number of different size slots and system does not include illumination planning.
holes as well as on a tray of three-dimensional parts. The parameters that are determined include the three po-
The analytical framework set up by the SRI group also sitional degrees of freedom of the sensor-r,,(:c, y. 2)-and
has the advantage of including other constraints on imag- the two orientational degrees of freedom-pan and tilt an-
ing such as illumination. Cowan and Bergman [23] have glesdescribed by a unit vector U along the viewing direction.
extended the results of the sensor planning described above Rotation with respect to the optical axis is not considered. In
by computing the appropriate range of lens aperture settings addition, the three optical parameters-the back principal point
and the three-dimensional region where a point-source light to image plane distance d, the focal length f , and the aperture
may be placed to illuminate the scene. Including illumination of the lens (&-are taken into account. Thus, planning is done
in the planning component is difficult, since it increases the in eight-dimensional imaging space and a point in this space
number of controllable parameters greatly (i.e., position and is defined as a generulized viewpoint V ( r o U, , d. f , a ) .
orientation of the light source, kind of sourcePxtended or In MVP all task constraints are formulated analytically.
point-and source energy). Cowan and Bergman have made Using concepts from geometry, illumination, and optics, each
some reasonable assumptions about the imaging and lighting task requirement is modeled by an equivalent analytical rela-
in a robotic work cell to show the viability of both camera tionship which in turn is satisfied in a domain of admissible
and illumination planning. In their work, the reflectance of a values in the space of parameters to be planned. Generally
surface is modeled as having a diffuse component that follows speaking, for each constraint the admissible domain for sensor
Lambert’s cosine law and a specular component forming a placement and setting is a region in eight-dimensional imaging
lobe within an angular tolerance of the perfect specular angle. space bounded by the hypersurfaces that are determined by
Interreflections are ignored and viewing of the surface is these analytical relationships. The component admissible do-
assumed to be on-axis of the camera. Their analysis shows mains obtained for each task requirement are then combined
that the dynamic range constraint of the sensor (i.e., the in order to find parameter values that satisfy all constraints
brightness-no surface in the scene is too dark or too light) simultaneously. For this purpose, the problem is posed in
from the diffuse component of reflection lies between two an optimization setting in which a globally admissible eight-
closed curves that are proportional to the incident angle of the dimensional viewpoint is sought that is central to the admis-
illumination. The specular region of the reflectance can also sible domain; that is, distant from the bounding hypersurfaces
be computed and is subtracted from the region bounded by the described by the constraint equations. Such a generalized
closed curves to yield a set of admissible regions that satisfy viewpoint is considered desirable since it is robust in the
the illumination constraints. The illumination planning system event of inaccuracy of either sensor placement or setting. The
has been demonstrated with a camera on a robot arm and a analytical relationships for each task constraint provide the
point-source mounted on a second robot arm. A multicolored constraints of the optimization, while the objective function is
TARABANIS et a/ : SURVEY OF SENSOR PLANNING IN COMPUTER VISION

ytz 0
Fig. 7. Visibility region for both feature edges (from [74])

Fig. 6. CAD model of the object used in the MVP system with the feature expressed as follows:
edges to be observed (from [74]).

chosen so as to characterize the distance between a generalized where I = 1 for the resolution constraint, I = 2a, 2b for the
viewpoint and the bounding hypersurfaces. near and far limits of the depth-of-field constraint and z = 3
Once a central generalized viewpoint is determined from for the field-of-view constraint. It should be noted that there is
the optimization, it then needs to be realized in the actual a g1 relationship for each linear feature to be resolved, while
sensor setup. While the task constraints are expressed in for the depth-of-field and field-of-view constraints there is a
terms of the generalized viewpoint, the parameters that can unique set of relationships g 2 0 , g211,and g3 for all features. In
be controlled in an actual sensor setup are generally different. addition to the above inequality constraints, there is also an
In order to achieve these planned sensor parameter values, a auxiliary equality constraint g, = 0 which expresses the unit
mapping is established between the planned parameters (e.g., vector condition for the viewing vector. ,
/'
camera pose and optical settings) and the parameters that While the constraints address the admissibility of the com-
can be controlled (e.g., end effector pose, zoom, and focus puted solution, the optimization function is constructed in a
settings). This mapping between the two parameter spaces is way that characterizes the robustness of the computed solution.
provided by calibration models that embody knowledge of the The measure used to assess the robustness of a solution
geometric relationships of the manipulator, sensor, illuminator, with respect to the resolution, field-of-view, and depth-of-field
and optical relationships of the lenses. constraints is the value of each constraint relationship g l , I =
Objects in MVP are modeled as general polyhedra with 1, 2n, 2h, 3. This is appropriate since a large positive value of
concave and convex faces, with or without holes. The visibility g2 indicates that the constraint is satisfied comfortably, a small
planning algorithm of MVP first considers a sufficient subset positive value indicates marginal satisfaction, and inadmissible
of the faces of the observed polyhedron as polygons in solutions give rise to negative values. Similarly, for the
three-dimensional space that are potentially occluding the visibility constraint a measure of this type is also formulated.
feature to be observed. The algorithm then determines the For this purpose, the minimum distance cl,, from the viewpoint
three-dimensional occluded regions between these occluding to the polyhedron describing the visibility region is chosen:
polygons and each target feature. The individual occluded g4 = fd,,, where +d7, or -d,> depending on whether the
regions of these faces are unioned in order to generate the point is inside or outside the visibility volume, respectively.
occluded region of the target feature for the polyhedron as a The optimization function is taken to be a weighted sum of
whole. The complement of the occluded region is the visibility the above component criteria, each of which characterizes
region from where the entire target feature can be viewed. The the quality of the solution with respect to each associated
visibility region is polyhedral and defines a piecewise analytic requirement separately. Thus, the optimization function is
constraint for viewpoint placement. The visibility region for written as
the object features of Fig. 6 is shown in Fig. 7. In this case,
the visibility region consists of two connected components that
f = max ( ( k l c / l + ( t 2 a 9 2 a + ff26(12b + (tJb!].3 + (?4.(14) (5)
correspond to viewing the edge features through the small hole subject to 9, 2 0, I = 1. 2(1, 2b. 3 , 4 and ~5 = 0, where ( Y ? ,
S H and the large hole L H of the object (see Fig 6). are the weights. These weights are currently chosen so that
The resolution, depth of field, and field of view constraints the contribution of each constraint to the objective function
are characterized by analytic closed-form relationships. These is of the same order of magnitude and prevents a subset
relationships are derived using the geometry of perspective of the constraints from dictating the optimization. Given the
transformation and concepts from optics. The locus of gen- above formulation, the optimization starts with an initial point
eralized viewpoints that satisfies each constraint separately is in the domain of possible generalized viewpoints and then
 98 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. I I , NO. I , FEBRUARY 1995

visible an entire object, whenever this is possible, and will

minimize the maximum local distortion in the image due to
perspective projection. An object point is considered visible
in this analysis if and only if it lies in front of the observer;
that is, occlusion is not considered and there are no field of
view limits. As a result, the viewing direction that is sought
is such that all object points are in front of the viewpoint and
perspective distortion is minimized. The viewpoint location is
assumed to be known. This is often the case in many graphics
applications, where the viewpoint and the object are fixed and
an orientation is sought for which the object is centered in the
field of view.
Fig. 8. Camera view of the features from viewpoint I-,, computed by the Anderson proposes this viewing direction rather than view-
MVP system (from [74]).
ing directions through the center of mass or the center of
the smallest box containing the object, since the latter are
generates a generalized viewpoint that is globally admissible ad hoc and are often inadequate when the viewpoint is
and locally optimal, as described by the optimization function. close to the object. Anderson shows [SI that minimizing the
For this computed viewpoint, all constraints are satisfied with maximum perspective distortion is equivalent to minimizing
the largest margin in the neighborhood of the initial point. the maximum value of the off-axis angle of any object point
As an example, for the object and the edge features shown (i.e., the angle between the viewing direction and the vector
in Fig. 6, the two initial viewpoints that are chosen to start from the viewpoint to an object point). Consequently, the
the optimization, I& and K 2 , and the corresponding camera computed viewing direction corresponds to the axis of the
viewpoints that are computed by the MVP system, Vf1 and narrowest circular cone emanating from the viewpoint and
Vf2,are shown in Fig. 7 along with their associated viewing containing the object. By considering that the viewpoint can
vectors. It can be seen from Fig. 7 that the initial guess be taken as the origin and that the vectors to the object
viewpoint V,l is chosen to lie inside the visibility region points can be normalized to unit length without affecting the
with a viewing vector in the - 2 direction. For V,, the field- solution, the problem can be reduced to a problem in spherical
of-view and focus constraints are violated. The initial guess geometry. More specifically, the narrowest cone enclosing the
viewpoint VL2is chosen to lie outside the visibility region object points is generated by finding the smallest circle on the
with a viewing vector again in the --z direction. For &2 all unit sphere that encloses the object point projections onto this
constraints are violated. The viewpoints f l and Vf, that are sphere.
determined by the MVP system can be seen in Fig. 7 to lie Anderson provides an algorithm in [SI to find the narrowest
inside the visibility region with viewing vectors in the direction circular cone directly and several more efficient algorithms
of the features. Camera views taken by a robot vision system in [6] that are based on the aforementioned reduction of
from the computed viewpoints and with the computed optical the original problem to a problem in spherical geometry.
settings, verify that the constraints are indeed satisfied. The These latter algorithms are based on existing algorithms for
view of the object taken by the robot vision system from the analogous plane geometry problems (e.g., finding the spherical
computed viewpoint Vf2 is shown in Fig. 8. convex hull of a set of points and the spherical Voronoi
With this global optimization formulation of the problem, diagram of a set of points).
the MVP system provides a framework in which individual In summary, in this work by Anderson the viewing direction
constraints can be combined and other constraints can be for the constraint of minimizing perspective distortion is deter-
* incorporated. However, further work is needed in order to mined. Requirements such as magnifying the object features,
study the convergence of the method, the dependence of the having them in focus, and avoiding occlusion of the features
method on the quality of the initial guess, and the selection of are not considered.
appropriate weight settings. 4 ) Remarks on the Synthesis Approach: On the whole, the
The MVP system is notable for its generality of handling synthesis approach forces an understanding of the causal
general viewing configurations (e.g., nonorthogonal viewing, relationships between the parameters to be planned and the
general convex, and concave polyhedral objects) and determin- goals to be achieved. In this way an insight is provided to
ing a complete set of imaging sensor parameters as described the problem, rather than just blindly applying search. To this
by its concept of a generalized viewpoint, Another advantage end, the task constraint are modeled analytically, bringing to
of the MVP system is that it addresses the associated sensor bear a powerful characterization that can be utilized to answer
modeling problems (e.g., in [79], models for a programmable more questions (e.g., “Using a quantitative measure, how
zoom lens are described). Together with the SRI system of comfortably does the computed viewpoint satisfy the depth-
Cowan and Kovesi, MVP studies the broadest range of feature of-field constraint?’) As discussed in Section 11-D.2, with such
detectability constraints. an analytical formulation of the problem, techniques such as
3) Automatic Determination of Viewer Orientation by Ander- optimization can be used as a framework in which several task
son: Anderson [ 5 ] ,[6] describes algorithms for the automatic constraints can be combined. Or if globally admissible values
‘ determination of the viewing direction which will render can be generally determined for a subset of the parameters,
TARABANIS et al.: SURVEY OF SENSOR PLANNING IN COMPUTER VISION 99

the method discussed in Section 11-D.1 has the advantage of tions and test modifications to existing ones. Most of the early
reducing the dimensionality of the search space and providing systems [55] modeled simple distance and tactile sensors by
approximations of the entire admissible space of solutions in employing the solid modeling operation of intersection in order
the dimensions that it considers. In addition, the accuracy of to detect contact between the sensor (e.g., the robot gripper
the techniques taking the synthesis approach can be directly surface or the light beam of a noncontact distance sensor)
computed from the analytical relationships and easily adjusted and objects in the robot workcell. More recently, sensor
to the specifications of the task. simulation systems have been built that include simulation of
more complex sensors such as cameras in conjunction with
E. Review of Work Taking an Expert Systems Approach light sources. In this section, we shall discuss two such sensor
I ) The Lighting Advisor: Batchelor [8], [9] has built an simulation systems: VANTAGE and ROSI.
expert system that suggests possible lighting and viewing 1 ) The VANTAGE System: VANTAGE [40], [41] is a geo-
configurations to the designer of a machine vision system metric and sensor modeler for model-based vision systems.
for industrial inspection. This expert system includes the In general, vision systems deal with two-dimensional appear-
LIGHTING ADVISOR which provides advice regarding what ances of an object and these appearances are determined by
lighting configurations are required in given circumstances. both the object itself (e.g., its geometric and photometric
Two other advisors which will provide advice about the camera properties) and the sensor used to observe it. Typically, the
and the lens are also being developed. object properties are part of the geometric models that are
The program asks a series of questions regarding the object used in model-based vision systems. However, the sensory
to be inspected and the inspection task to be performed; for information that also determines the object appearances is
example, information regarding the reflectance characteristics not included in such modelers. VANTAGE provides sensory
of the object (specular, diffuse, opaque, transparent, translu- information such as the feature types that a sensor detects, the
cent) as well as the type of feature or defect that is to be particular object features that are detectable at a given sensor
highlighted. The program then displays a line drawing which configuration, and the reliability with which the detectable
shows a sketch of the recommended lighting configuration. features can be observed.
Batchelor intends to incorporate more rules in the LIGHT- In VANTAGE, a feature configuration space is first defined
ING ADVISOR in order to actually implement the lighting in order to represent the spatial relationship between the
arrangement that has been recommended by automatically sensor and the feature. In this representation only the angular
controlling the illumination in a flexible inspection cell. This relationship between the sensor and feature coordinate systems
cell provides facilities such as a pick-and-place arm, a 3- is considered. A way to then specify sensor detectability and
degree-of-freedom ( : E , y, 6 ) table, and computer-controlled sensor reliability over this space is given. On the one hand,
lighting with which a wide range of illumination and viewing sensor detectability specifies what features can be detected for
techniques can be implemented. With these additional expert a given sensor configuration, while sensor reliability specifies
system rules, the recommended illumination will be converted the expected uncertainty in sensory measurement and the
into a set of control signals for the lamps. uncertainty of the scene features that are derived from this
2 ) Other Expert Systems for Sensor Planning: Along simi- measurement.
lar lines to the expert system approach of Batchelor, there Both illuminators and detectors are considered in a uni-
has been work by Penn Video, Inc., a subsidiary of the Ball form manner as generulized sources with two properties: the
Corporation, that sells a lighting advisor program described by illumination direction and the illumination configurations. For
Novini in [57].In [45] another expert system is presented for light sources, the illumination direction is the direction of
the design of vision algorithms. This system provides advice the light source, while the illuminated configurations are the
on the selection of effective features and the image processing collection of features that can be illuminated for the particular
operations to extract the chosen features. In addition, the illuminator pose, assuming that the illumination direction is
Industrial Technology Institute in Dearbom, MI, has compiled not occluded. In the case of detectors, these terms correspond,
a database of lighting and viewing techniques as described in respectively, to the viewing direction of the detector and the
1201. set of features that are visible to the detector. With these
3 ) Remarks on the Expert Systems Approach: The systems properties, feature detectability with respect to a generalized
following an expert system approach address the high-level source can be characterized as follows: A feature is illuminated
aspects of the problem in which a particular viewing and by a generalized source if the feature coordinate system lies
illumination technique is chosen from a catalogue [lo], for in- in the illuminated configurations of the illuminator and the
stance, whether front or back illumination is more appropriate illumination direction is not occluded.
for the particular object and feature to be observed. However, Feature detectability for a general vision sensor is decom-
these qualitative approaches need to be extended in order to posed into such illumination conditions of its component
represent and determine the exact spatial relationships between generalized sources and Boolean operations between them.
the sensor, the illuminator, and the object. This decomposition defines the sensor composition tree in
VANTAGE. For example, as shown in Fig. 9, a light-stripe
F. Review of Related Work in Sensor Simulation Systems range finder can detect the portions of an object surface onto
Several sensor simulation systems have been developed and which the illuminator projects light directly and which the
have been used as tools to plan new robot workcell installa- camera can observe. Thus, the detectability of such a vision
 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. I I , NO. I. FEBRUARY 1995

this sensor detectability information, and applying the sensor

composition tree to the scene. This is done in three steps:
The three-dimensional face segments that are illuminated
by each component generalized source are determined,
These segments are then combined according to the
Boolean operations of the sensor composition tree, and
finally,
The three-dimensional face segments that have been
determined to be visible and either illuminated or in
shadow are projected onto the image plane in order to
generate the two-dimensional appearance of the object.
The ability of VANTAGE to predict the two-dimensional
appearances of objects when viewed by a sensor can be
readily employed as the kernel of a tool that plans sensor
configurations. Alternatively, in [39], VANTAGE is used to au-
tomatically generate object recognition programs. The various
appearances of objects are generated and then categorized into
possible aspects, where each aspect is defined to have the same
combination of detected two-dimensional faces. The recog-
(b) nition strategy is then comprised of an aspect classification
Fig. 9. Sensor detectability conditions for a camera and light stripe illumi-
component which is represented as an interpretation tree, and
nator pair from VANTAGE [41]. a linear shape change component which is generated by using
correspondences between image regions and model faces. This
recognition strategy is then converted into a program using an
sensor is decomposed into two generalized source illumination object library.
conditions and Boolean AND operation between them as 2 ) The ROSI Sensor Simulation System: Raczkowsky et al.
shown in the sensor composition tree of Fig. 9. Faces which
[36], [58] have developed ROSI, a sensor simulation system
are both illuminated by the light source and observable by
for robotic applications that includes simulation of a camera
the camera are then projected (PROJ operation in Fig. 9)
in conjunction with light sources. In the ROSI system, an
onto the image plane in order to generate the two-dimensional
industrial scene can be visualized with a simulated camera
appearance of the object.
given:
VANTAGE also takes into consideration sensor noise when
determining feature detectability. This is required because 1) The geometric description of the cell objects,
sensor noise may interfere with detectability even if the above 2) The optical properties of their surfaces,
detectability constraints are satisfied. Since most vision sensors 3) The physical and geometric description of the light
detect features based on brightness values, detectability of a sources, and
feature will depend on the distribution of feature brightness. In 4) The physical and geometric properties of the sensors.
VANTAGE a feature is taken to be detectable if its brightness The geometric description of an object is given by a
is above a threshold. Thus, the feature brightness distribution polyhedral boundary representation based on CAD generated
is modeled as the probability of the feature brightness being data. The surface optical properties include the coefficient
above this threshold. Feature brightness is computed by as- of diffuse reflectance, the index of refraction of the surface
‘suming that the object surface is Lambertian and adding a material, the ratio of diffuse to specular reflectance, and a
zero-mean Gaussian noise term. factor describing the rms slope of the distribution function
In addition to sensor detectability, sensor reliability is also of the reflectance. These properties are used in the bidi-
modeled in the VANTAGE system. Sensor reliability addresses rectional reflection model of Torrance and Sparrow [77].
the uncertainty in sensory measurement and its propagation Inter-reflections are not considered. The light sources are
to uncertainty of the scene features. The sources of such modeled as Lambertian, emitting light evenly, and are defined
uncertainty include the variance in brightness values and the by the center point, the area, the normal vector, and the spectral
variance in light source direction. These two variances are, radiation density of the emitting surface. The camera model
respectively, the major causes of uncertainty for vision sensors. consists of three parts: geometric information regarding the
The uncertainty in sensory measurement for a light-stripe camera pose, parameters that describe the lens system, and
range finder and a photometric-stereo sensor is characterized parameters describing the electronic hardware. In particular,
in [40]. In addition, for the case of the light-stripe range the lens system is modeled by a thin lens with an aperture
finder the propagation of uncertainty from a two-dimensional diaphragm. The parameters characterizing the lens system are
region image feature to a three-dimensional feature area is the focal length, the aperture setting, and the distance (i.e.,
modeled. focus setting) of the lens. This last parameter is equivalent
Appearances of the object when observed by the sensor to the back nodal point to image plane distance d, which
at hand can be generated using the object geometry and is discussed in Section 11-B. 1. The electronic hardware is
TARABANIS er al.: SURVEY OF SENSOR PLANNING IN COMPUTER VISION 101

described by the number of rows and columns of pixels, the fix or assume some of the necessary sensor parameters, per-
dimensions of the rows and columns, and the sensitivity of forming an efficient search of the remaining parameter space
the sensor. for feasible solutions. The other family of solutions has tried
For the given configuration of the camera and the light to analytically model the task constraints (using a variety of
sources, and with the object and sensor model information, assumptions about parameters) and then find an analytical
the system: description of the feasible solutions. In the latter case, the high
Performs a hidden-surface removal to determine the dimensionality of these constraint surfaces requires a nonlinear
surfaces of the scene visible to the camera, optimization approach, which is also a form of efficient search.
Performs a similar hidden-surface removal, with the light It is not clear which of these methods is currently more viable.
source as the viewpoint, in order to determine the light The simplicity of the discrete space approach is appealing,
sources illuminating the individual visible surfaces, as is its hierarchical implementation. On the other hand, the
Clips all visible surfaces to the view volume of each inaccuracy of the tesselations, assumption of certain viewing
individual pixel in order to determine the list of polygons parameters and inability to easily deal with multiple features
illuminating each pixel, make it less desirable. The analytical approach is elegant and
Determines the radiation flux illuminating the individ- extensible to multiple features and off-axis viewing, albeit at
ual pixels of the sensor plane using the Torrance and the cost of constrained nonlinear optimization.
Sparrow reflectance model, While both methods will continue to be used by researchers,
Generates the corresponding gray or color levels of further work is needed in order to achieve robust and gen-
the pixels based on the incoming radiation flux, the eral purpose camera and illumination planning strategies and
minimum and maximum irradiance of a pixel, and its systems. There are common open problems that need to be
relative spectral sensitivity. addressed by future systems. We include a partial list of these
The ROSI sensor simulation system has been used as a tool areas that we believe are fruitful for future research.
to support the planning process of robot cells. While it does
not synthetically determine robot cell configurations that are
A. Modeling and Incorporating Other Constraints
suitable for a particular task, ROSI provides the capability of
varying camera and light-source parameters in order to search The sensor planning systems that have been developed
for a satisfactory simulated view of the scene. have chosen a subset of the feature detectability constraints
3) Remarks on the Sensor Simulation Systems: The sys- discussed in Section 11-B.3. However, there are several other
tems in this category have many commonalities with various feature-detectability constraints as well as a broad range of
,/
approaches taken to the general graphics problem. While constraints that arise in other sensing tasks, which could be /

concurrent work in the computer graphics community has modeled and incorporated in sensor planning systems.
looked at the realistic rendering of scenes, the algorithms An example of a constraint that arises in all sensor planning
employed have a high computational cost (i.e., ray tracing) systems involving movement as part of sensor reconfiguration
which may make them infeasible for this application. Also, is collision avoidance. The values of the sensor parameters
sensor models like those developed in VANTAGE are outside resulting from planning must be such that the sensor can be
the scope of research in computer graphics. These issues are positioned without collision with the environment. Similarly,
essential, however, for sensor simulation systems since their no point in the path to the final sensor configuration should
goal is to simulate real-world sensors in order to determine lead to collision with the environment. One way to address
and evaluate a sensor system and its operation. this constraint would be to draw on previous work in path
Compared with the other approaches reviewed previously, planning in order to model collision-free space, and then treat
the sensor simulation systems use much more illumination collision-free space in a manner similar to the way visibility
information. However, both these systems and the systems tak- regions are dealt with in the MVP system.
ing a generate-and-test approach can easily incorporate more If the environment, the features of interest, or both, move
complete reflectance and illumination models (e.g., mutual in time along paths that in some cases are known in advance,
reflections, area light sources) much in the same way that work this gives rise to a family of time-dependent constraints for
in computer graphics has done. dynamic sensor planning. With the exception of the initial
work in the V I 0 system, the methods presented in this survey
deal mostly with planning of parameters for static scenes. The
111. CONCLUSIONS AND FUTURE DIRECTIONS
IN
sensor planning capabilities of the MVP system have recently
SENSOR PLANNING FOR FEATURE DETECTABILITY been extended [ 2 ] to function in environments where objects
The area of sensor planning for machine vision tasks re- in the environment move in a way that is known a priori,
mains a very rich research area that spans several fields but the features to be observed are not moving. The problem
of study: computer vision, robotics, optics, geometry, and of dynamic sensor planning is of importance because many
computer graphics. The methods presented in this survey sensing environments in practice are more often dynamic than
represent, in most cases, the initial efforts of researchers to not. Thus, the usefulness of these systems will be increased
attack a difficult problem with many degrees of freedom. greatly if planning can be done in a dynamic fashion to
For the task of feature detectability, one family of solutions include moving sensors, light sources, targets, and obstacles.
to this problem has attempted to discretize space and either If planning can be done efficiently and cheaply, then complex
I02 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. I I . NO. 1, FEBRUARY 1995

tasks such as assembly can be monitored in real-time using needs to be investigated. Polyhedral approximations may often
these methods. be sufficient, since in many cases the resulting sensor param-
An example of a constraint that is generic to all sensor eter values may be robust enough to be admissible for the
planning systems is the constraint associated with the oper- constraints of the original object as well. Recent work by [27],
ating range of the employed sensor. If such constraints are [47], [68] in analyzing the visibility of more complex shapes,
ignored, it is often the case that the computed sensor parameter such as generalized cylinders, may be useful in extending the
values are not achievable by the sensor at hand. For example, planning systems to include curved surface objects.
the optical parameter f , d, and a discussed in Section 11-B.l 4 ) Constraint Integration: Techniques that combine the ad-
often cannot simultaneously assume the planned values for a missible domains of individual constraints in order to deter-
given lens. Techniques to characterize the operating range of mine globally admissible solutions need to be investigated
a general lens fall in the area of sensor modeling which is further. Optimization, used in MVP and ICE, provides one
discussed at the end of this section. such framework. However, further work is needed in order
to study convergence properties, appropriate weight values,
B. Modeling and Incorporating Other Sensors and the dependence of the final solution on the quality of
the initial guess. Also, the optimization formulation employed
While this survey has focused on planning using vision
currently constitutes a simple integration scheme based on
sensors, a host of other sensors are currently being used in
the assumption that multiple and coupled objectives can be
many robotics and manufacturing applications. These include
combined in an additive sense into a single global objective.
tactile sensors, three-dimensional range sensors, force-torque
Such a formulation has inherent problems with conflicting
sensors, and acoustic sensors. Some of the techniques de-
objectives. Alternative optimization schemes such as multiple
scribed above extend nicely for planning with these sensors
objective optimization should also be explored with each task
(e.g., the visibility line of sight can be thought of as an
constraint constituting an individual objective.
approach vector for a tactile probe). However, further work
Techniques that determine admissible regions rather than an
needs to be done to properly integrate these sensors and their
optimal point; should also be investigated. A generalization
unique constraints into the overall planning system.
of the method discussed in Section 11-D.l could yield the
admissible space of solutions for a subset of the parameters
C. Relaxing Some of the Assumptions Made having first determined globally admissible values for the other
in Current Approaches parameters. In order to approximate the admissible region
1) Feature Uncertainty: Underlying most of the work done of all constraints, interval-based techniques for solving large
in this area of sensor planning has been the assumption that systems of nonlinear constraints such as those described in
the environment is known. This is often not the case in [32] will prove useful.
actual sensing situations. As a first step towards relaxing this The question of weighting constraints according to the
assumption, sensor planning systems could investigate how importance of each in the general planning objective is a
to incorporate feature uncertainty. One approach to address notable one. It may be more important to have a feature
feature uncertainty could attempt to combine the deterministic visible at a poor resolution than not being visible at all. In
sensor planning techniques with approaches that use a decision determining solutions to parameter values, feasible solutions
theoretic framework [15], [31] in order to address the inherent may be found but ranking them as to suitability is a more
uncertainty in sensing. In this respect, deterministic sensor difficult task. While this may be somewhat task dependent, it
planning systems adopting a synthesis approach provide the may be important to isolate some parameters and solve for
advantage of including analytical relationships that character- them alone. When optimization is used for integration, this
ize the admissible loci. When feature uncertainty is included, amounts to determining the weights on each of the constraints
these relationships can be used in order to compute the that make up the objective function that needs to be optimized.
variation of the admissible loci based on known perturbations
of the features and the environment.
2 ) Accuracy: The previously proposed extension leads to E. Illumination Planning
the related problem regarding the accuracy of the solutions While some of the systems mentioned previously have
for sensor parameter values. In the case of the discrete state either explicitly or implicitly dealt with planning of lighting
approach, characterization of the error induced by the dis- parameters, current work in illumination planning is quite
cretization is needed. In the analytic approach, creating tol- restricted. It should be recognized, however, that the problem
erance constraint volumes (as in geometric tolerancing with of planning of general lighting for machine vision is extremely
CAD/CAM systems) may be useful in determining how robust difficult. Most of the work has used point sources of light that
a solution may be to sensor and calibration error. are incident on convex Lambertian surfaces. These models,
3 ) Complex Object Models: Most of the current work has while useful, are not analogous to actual lighting conditions
considered polyhedral objects and polyhedral type features. seen in current applications. Higher order lightingheflectance
Objects with curved surfaces and their corresponding object models that include such phenomena as multiple sources (both
features need to be included as well. As a first step in this point and extended), specularity, and interreflections from
direction, the approach of posing the problem in terms of concave surfaces need to be found to properly plan lighting
polyhedral approximations of objects with curved surfaces parameters.
TARABANIS er a[.:SURVEY OF SENSOR PLANNING IN COMPUTER VISION 103

F. Sensor and Illumination Modeling [I51 A. Cameron and H. Durrant-Whyte, “A Bayesian approach to optimal
sensor placement,” Rept. OUEL 1759/89, Department of Engineering
A subtle, but important problem in this area is that of Science. Oxford Universitv. Oxford. UK. 1989.
modeling the sensors and mapping the planned sensor pa- [I61 A. Cameron and H. L. ‘Wu, “Identifying and localizing electrical
components: A case study of adaptive goal-directed sensing,” Tech. Note
rameters to real hardware systems. The coupling between the TN-90-085, Philips Laboratories, Briarcliff, NY, 1990.
sensor planning and modeling problems becomes apparent [I71 0. I. Camps, L. G. Shapiro, and R. M. Haralick, “PREMIO: An
when the planned parameters that have been determined by overview,” in Proc. IEEE Workshop on Directions in Automat. CAD-
Based Vision, 1991, pp. 11-21,
some means are to be achieved in an actual setup. For [ 181 L. L. Chin and T. C. Woo, “Computational geometry on the sphere with
this, the planned parameter values need to be mapped to application to automated machining,” Tech. Rept. TR 89-30, Industrial
controllable sensor settings using the sensor models. For and Operations Engineering, Univ. of Michigan, Ann Arbor, MI, Aug.
1989.
some parameters (camera location and orientation) this can [I91 J. J. Clark and N. J. Fenier, “Modal control of an attentive vision
be relatively straightforward. However, parameters such as system,” in Int. Con$ Comput. Vision, 1988, pp. 514-523.
entrance pupil diameter, focal length, and image plane to back [20] M. Coletta and K. Hading, “Picking the right lighting,” MVMSME’s
Vision Technol. Quart., pp. 1-3, Jan. 1990.
principal point distance can be difficult to automatically set [21] C. I. Connolly, “The determination of next best views,” in Proc. 1985
without extensive calibration effort. This is particularly true for IEEE Int. Conf Robotics and Automat., 1985, pp. 432435.
zoom lens cameras, which provide more degrees of freedom [22] C. K. Cowan, “Model based synthesis of sensor location,” in Proc. 1988
IEEE Int. Con$ Robotics and Automat., 1988, pp. 900-905.
for general imaging situations but at the cost of mapping [23] C. K. Cowan and A. Bergman, “Determining the camera and light
planned parameters to their actual settings [76], [82]. A related source location for a visual task,” in Proc. IEEE Int. Conf Robotics
problem is the creation of accurate sensor noise models that and Automat., pp. 509-14, 1989.
[24] C. K. Cowan and P. D. Kovesi, “Automatic sensor placement from
can be combined with the other known imaging constraints to vision task requirements,” IEEE Trans. Putt. Anal. Mach. Intell., vol.
accurately predict correct parameters. 10, pp. 407-16, May 1988.
While sensor modeling can utilize much of the work that [25] C. K. Cowan and B. Modayur, “Edge-based placement of camera and
light-source for object recognition and location,” in Proc. 1993 IEEE
has been done in camera and lens calibration, there has been Int. Con$ Robotics and Automat., 1993.
very little previous work in the area of illumination modeling. [26] R. I. D. Cowie, “The viewer’s place in the theories of vision,” in Int.
Much in the same way that research in camera modeling Joint Cant Artificial Intell., 1983, pp. 952-958.
[27] D. Eggert and K. Bowyer, “Computing the orthographic projection as-
has developed calibration methods in order to position and pect graph of solids of revolution,” in Proc. Workshop on Interpretation
set cameras and their lenses, work in illumination modeling o f 3 D Scenes, 1990, pp. 102-108.
[28] C. Goad, “Special purpose, automatic programming for 3D model-based
will also need to develop similar calibration methods with vision,” in Proc. DARPA Image Understanding Workshop, 1983, pp.
which to characterize the attributes of an illumination source 94-104
and to position illuminators in light-in-hand arrangements. [29] J. L. Gordillo and A. LUX,“Synthesizing vision programs from robot
task specifications,” in 3rd In?. Symp. Robotics Res., 1985, pp. 149-154.
For this purpose, drawing On the wide body Of [30] W. E. L. Grimson, “Sensing strategies for disambiguating among
in photometry and radiometry [42] will prove useful. multiple objects in known poses,” ZEEE J. Robot. Automat., vol. RA-2,
no. 4, Dec. 1986.
REFERENCES [31] G. D. Hager, Task Directed Sensor Fusion and Planning. Norwell, MA:
Kluwer Academic Publishers, 1990.
[32] -, “Interval-based techniques for sensor data fusion,” Tech. Rep.,
[I] A. L. Abbott and N. Ahuja, “Active surface reconstruction by integrating
Yale University, New Haven, CT, 1990.
focus, vergence, stereo, and camera calibration,” in Znt. Proc. Con$
[33] C. Hansen and T. Henderson, “CAGD-based computer vision,” in Proc.
Compur. Vision, 1990, pp. 489492.
IEEE Workshop on Computer Vision, 1987, pp. 100-105.
[2] S. Abrams, P. K. Allen, and K. Tarabanis, “Dynamic sensor planning,”
[34] T. H. Hong and M. 0. Schneier, “Describing a robots workspace using
in Proc. 1993 ZEEE Int. Conf Robotics and Automat., 1993.
[3] N. Ahuja and J. Veenstra, “Generating xtrees from object silhouettes a sequence of views from a moving camera,’’ IEEE Trans. Pan. Anal.
in orthographic views,” IEEE Trans. Putt. Anal. Mach. Intell., vol. 11, Mach. Intell., vol. 7, pp. 721-726, Nov. 1985.
pp. 137-149, Feb. 1989. [35] B. K.P. Hom, Robot W o n . New York: McGraw-Hill, 1986.
[4] J. Aloimonos, I. Weiss, and A. Bandyopadhyay, “Active vision,” in [36] M. Huck, J. Raczkowsky, and K. Weller, “Sensor simulation in robot
Proc. Conf Comput. Vision, 1987, pp. 35-54. applications,” in Proc. Advanced Robotics Program, Workshop on Ma-
[5] D. P. Anderson, “An orientation method for central projection pro- nipulators, Sensors and Steps Towards Mobiliry 1987, pp. 197-209.
grams,” Comput. & Graphics, vol. 6, no. I , pp. 35-37, 1982. [37] S. A. Hutchinson and A. C. Kak, “Planning sensing strategies in robot
[6] D. P. Anderson, “Efficient algorithms for automatic viewer orientation,” work cell with multi-sensor capabilities,” IEEE Trans. Robot. Automat.,
Comput. & Graphics, vol. 9, no. 4, pp. 407413, 1985. vol. 5, no. 6, pp. 765-783, Dec. 1989.
[7] R. Bajcsy, “Active perception,” Proc. IEEE, vol. 76, no. 8, pp. [38] K. Ikeuchi, “Generating an interpretation tree from a CAD model for
9961005, Aug. 1988. 3D-object recognition in bin picking tasks,” Int. J . Comput. Vision, vol.
[8] B. G. Batchelor, “A Prolog lighting advisor,” in Proc. SPZE Intell. Robots I , no. 2, pp. 145-165, 1987.
and Vlll: Syst. and Applicat., vol. 1193, pp. 295-302, 1989. [39] K. Ikeuchi and T. Kanade, “Towards automatic generation of object
[9] -, “Integrating vision and AI for industrial applications,” in Proc. recognition programs,” Tech. Rept. CMU-CS-88-138, Carnegie Mellon
SPIE Intell. Robots and Comput. Vision Vlll: Syst. and Applicat., vol. University, Pittsburgh, PA, May 1988.
1193, pp. 168-173, 1989. [40] -, “Modeling sensors: Toward automatic generation of object
[IO] B. G. Batchelor, D. A. Hill, and D. C. Hodgson, Automated Visual recognition program,’’ Comput. Vision, Graphics, Image Process., vol.
Inspection. Bedford, UK: IFS Ltd., 1985. 48, pp. 5 C 7 9 , 1989.
[ I l l A. K. Bejczy, W. S. Kim, and S. C. Venema, “The Phantom robot: [41] K. Ikeuchi and J. C. Robert, “Modeling sensors detectability with the
Predictive displays for tele-operation with time delay,” in Proc. 1990 VANTAGE geometrickensor modeler,” IEEE Trans. Robot. Automat.,
IEEE Int. Conf Robotics and Automat., 1990, pp. 546551. vol. 7, pp. 771-784, Dec. 1991.
[ 121 R. C. Bolles and R. A. Cain, “Recognizing and locating partially visible [42] H. A. E. Keitz, Light Calculations and Measurements. The Nether-
objects: The local-feature-focus method,” Znt. J. Robot. Res., vol. I , no. lands: Philip’s Technical Library, 1955.
3, pp. 57-82, 1982. [43] J. R. Kender and D. G. Freudenstein, “What is a ‘degenerate’ view?’ in
[I31 R. C. Bolles and P. Horaud, “3DPO: A three-dimensional part orienta- Proc. 1987 DARPA Image Understanding Workshop, 1987, pp. 589-598.
tion system,’’ Int. J. Robot. Res., vol. 5, no. 3, pp. 3-26, 1986. [44] H. S. Kim, R. C. Jain, and R. A. Volz, “Object recognition using multiple
[14] A. Califano, R. Kjeldsen, and R. M. Bolle, “Data and model driven views,” in Proc. 1985 IEEE Int. Con$ Robotics and Automat., 1985, pp.
foveation,” in Proc. 10th Int. Conf Putt. Recognition, 1990, pp. 1-7. 28-33.
 IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, VOL. I I , NO. 1, FEBRUARY 199.5

Y. Kitamura, H. Sato, and H. Tamura, “An expert system for industrial [71) K. Tang, T. C. Woo, and J. Gan, “Maximum intersection of spherical
machine vision,” in Proc. 10th Int. Con$ Patt. Recognition, 1990, pp. polygons and workpiece orientation for 4- and 5-axis machining,”
77 1-773. Industrial and Operations Engineering, Univ. of Michigan, Ann Arbor,
J. J. Koenderink and A. J. van Doorn, “The intemal representation of MI, 1990.
solid shape with respect to vision,” Biolog. Cybern.,no. 32, pp. 21 1-216, [72] K. Tarabanis and R. Y. Tasi, “Computing viewpoints that satisfy optical
1979. constraints,” in Proc. CVPR ‘91: The Comput. Soc. Con$ on Comput.
D. J. Kriegman and J. Ponce, “Computing exact aspect graphs of curved Vision and Patt. Recognition, 199 I .
objects: Solids of revolution,” Proc. Workshop on Interpretation of 3 0 [73] __, “Computing occlusion-free viewpoints,” in Proc. CVPR ’92:
Scenes, 1990, pp. 11&122. The Comput. Soc. Con$ on Comput. Vision and Patt. Recognition, June
E. Krotkov, “Exploratory visual sensing with an agile camera,” Ph.D. 15-18, 1992.
dissertation, University of Pennsylvania, 1987. [74] K. Tarabanis, R. Y. Tsai, and S. Abrams, “Planning viewpoints that
C. Laugier, A. I$, and J. Troccaz, “Combining vision based information simultaneously satisfy several feature detectability constraints for robotic
and partial geometric models in automatic grasping,” in Proc. 1990 IEEE vision,” in 5th Int. Con$ Advanced Robotics (ICAR), 1991.
Con$ Robotics and Automat., 1990, pp. 6 7 M 8 2 . [75] K. Tarabanis, R. Y. Tsai, and P. K. Allen, “Automated sensor planning
L. Lieberman, “Model-driven vision for industrial automation,” in for robotic vision tasks,” in Proc. 1991 IEEE Int. Conf Robotics and
Advances in Digital Image Processing, Stucki, Ed. New York: Automat., 1991.
Plenum, 1979, pp. 235-248. 1761 K. Tarabanis, R. Y. Tsai, and D. S. Goodman, “Calibration of a
.~
M. Magee and M. Nathan, “Spatial reasoning, sensor repositioning and computer controlled robotic vision sensor with a zoom lens,” Submitted
disambiguation in 3d model based recognition,” in Proc. Workshop on to Comput. Vision Graphics, and Image Processing, 1992.
Spatial Reasoning and Multi-Sensor Fusion, 1987. [77] K. E. Torrance and E. M. Sparrow, “Theory for off-specular reflection
W. N. Martin and J. K. Aggarwal, “Volumetric description of objects from roughened surfaces,” J. Opt. Soc. Anter., vol. 57, no. 9, pp.
from multiple views,” IEEE Trans. Patt. Anal. Mach. Intell., vol. 5, pp. 1105-1 114, Sept. 1967.
15&158, Mar. 1983. [78] R. Y. Tsai and M. A. Lavin, “Three-dimensional mechanical part
J. Maver and R. Bajcsy, “How to decide from the first view where measurement using a visiodrobot, system,” Tech. Rept. RC 10506, IBM
to look next,” in Proc. 1990 DARPA Image Understanding Workshop, T.J. Watson Research Center, Yorktown Heights, NY, May 1984.
1990, pp. 482496. 1791 R. Y. Tsai and K. Tarabanis, “Model-based planning of sensor placement
-, “Occlusions as a guide for planning the next view,” Tech. Rep., and optical settings,” in Proc. Sensor Fusion It: Human and Mach.
GRASP Lab., Univ. of Pennsylvania, 1991. Strategies, 1989.
J. Myer, “An emulation system for programming sensory robots,” IBM [80] __. , “Occlusion-free sensor placement planning,” in Machine Vi-
J. Res. Develop., vol. 25, no. 6, pp. 955-962, Nov. 1981. sion for Three-Dimensional Scenes, H. Freeman, Ed. Orlando, FL:
J. L. Mundy, “Industrial machine vision-Is it practical?’ in Machine Academic, 1990.
Vision, H. Freeman, Ed. San Diego, CA: Academic, 1988. [81] P. Whaite and F. P. Feme, “From uncertainty to visual exploration,” in
A. Novini, “Lighting and optics expert system for machine vision,” in Proc. Int. Con$ Comput. Vision, 1990, pp. 6 9 M 9 7 .
Proc. Optics, Illumination, Image Sensing, pp. 1005-1019, 1988. 1821 R. G. Wilson and S. A. Shafer, “Modeling and calibration of zoom
J. Raczkowsky and K. H. Mittenbuehler, “Simulation of cameras in lenses,” in Camera Calibration and Orientation Determination, A.
robot applications,” Comput. Graphics Applicat., pp. 16-25, Jan. 1989. Gwen and T. S. Huang, Eds. New York Springer-Verlag, 1993.
S. Sakane, M. Ishii, and M. Kakikura, “Occlusion avoidance of visual [83] W. J. Wolfe, D. W. Mathis, M. Magee, and W. Hoff, “Task panel sensing
sensors based on a hand eye action simulator system: HEAVEN,” Adv. with a movable camera,” in Proc. SPIE Intell. Robots and Comput. Vision
Robot., vol. 2, no. 2, pp. 149-165, 1987. VIII: Syst. and Applicat., vol. 1193, 1989.
S. Sakane, R. Niepold, T. Sato, and Y. Shirai, “Illumination setup 1841 K. H. Womack, “Front-end foresight,” MVNSME’s Vision Technol.
planning for a hand-eye system based on an environmental model,” Quarter., p. 4, Jan. 1990.
Adv. Robot., vol. 6, no. 4, pp. 461482, 1992. [85] S. Xie, T. W. Calvert, and B. K. Bhattacharaya, “Planning views for the
S. Sakane and T. Sato, “Automatic planning of light source and camera incremental construction of body models,” in Proc. 8th Int. Con$ Patt.
placement for an active photometric stereo system,” in Proc. 1991 IEEE Recognition, 1986, pp. 154-157.
Int. Conf Robotics and Autcm”a., 1991, pp. 108G1087. [86] S. Yi, R. M. Haralick, and L. G. Shapiro, “Automatic sensor and light
S. Sakane, T. Sato, and M. Kakikura, “Model-based planning of visual source positioning for machine vision,” Tech. Report EE-ISL-89-04,
sensors using a hand-eye action simulator system,” in 3rd Int. Con$ University of Washington, Seattle, WA, Sept. 1989.
Advanced Robotics (ICAR), 1987. [87] -, “An illumination model for machine vision,” Tech. Rept. EE-
-, “Planning focus of attentions for visual feedback control,” Trans. ISL-89-03, University of Washington, Seattle, WA, Sept. 1989.
Soc. Instrum. Cont. Eng., vol. 24, no. 6, pp. 608415, June 1988. [88] -, “Automatic sensor and light source positioning for machine
D. D. Seligmann and S. Feiner, “Automated generation of intent-based vision,” in Proc. 10th Int. Con$ Patt. Recognition, 1990, pp. 55-59.
3D illustrations,” Comput. Graphics, vol. 25, no. 3, July 1991.
S. A. Shafer, “Automation and calibration for robot vision systems,”
Tech. Rep. CMU-CS-88-147, Camegie Mellon University, Pittsburgh,
PA, May 1988.
A. Shmuel and M. Werman, “Active Vision: 3D from an image
sequence,” in Proc. 10th Int Conf Patt. Recognition, 1990, pp. 48-54. Konstantinos Tarabanis, for a biography and photograph, please see p. 85
A. J. Spyridi and A. G. Requicha, “Accessibility analysis for the of this issue.
automatic inspection of mechanical parts by coordinate measuring
machines,” in Pmc. 1990 IEEE Int. Ciwf Robotics and Automat., 1990,
pp. 1284-1 289.
T. Sripradisvarakul and R. Jain, “Generating aspect graphs for curved
objects,” in Proc. Workshop on Interpretation of 3D Scenes, 1990, pp.
ins-I 15
Peter K. Allen, for a biography and photograph,, please see p. 85 of this issue.
. - ~
[69] J. Stewman and K. Bowyer, “Creating the perspective projection aspect
graph of polyhedral objects,” in Int. Conf Comput. Vision, 1988, pp.
494-500.
[70] M. J. Swain and M. Stricker, “Promising directions in active vision,”
Tech. RePt. C s 91-27, University of Chicago, Chicago, IL, Nov. 1991. Roger Y. Tsai, for a biography and photograph, please see p. 85 of this issue.