SfM_Georef v.2.

http://www.lancs.ac.uk/staff/jamesm/software/sfm_georef.htm

Introduction

Sfm_georef is a gui-based tool for scaling and orienting SfM point clouds to real-world coordinates,
using observations made directly in the SfM image set. Sfm_georef does not carry out any SfM
reconstruction, but uses the camera data output by other software to enable point clouds (as .ply files)
to be scaled or geo-referenced. Output from the following SfM software or reconstruction pipelines
are supported:

Bundler Photogrammetry Package (http://blog.neonascent.net/archives/bundler-

photogrammetry-package/)
SFMToolbox (http://www.visual-experiments.com/demos/sfmtoolkit/)
VisualSFM (http://www.cs.washington.edu/homes/ccwu/vsfm/index.html)
OSM-Bundler (http://code.google.com/p/osm-bundler/)

For scaling a project, at least one distance (a control length) between positions that can be directly
observed in two or more images is needed. For scaling and orienting a project (full geo-referencing),
the real 3D coordinates of three or more points (control points) that can be observed in two or more
images, is necessary. Once the transform is determined, this can be applied to dense point cloud (.ply)
files generated by PMVS2, with provision given for merging multiple files (as produced by CMVS) into
one.

SfM_Georef v2.2 is written in Matlab (release 2010b) and tested on Win.7 64-bit. As Matlab code, it
should work on other operating systems, although visual layout of the controls on the main window
will not be optimal. It has also been compiled (under Windows XP) as a stand-alone executable. To run
the executable, you will need to download and install Matlab runtime libraries (see instructions
below).

Installation (stand-alone executable)

1) Uncompress sfm_georef_2.2.zip to a folder. The executable itself (sfm_georef_2.2.exe) needs

no further installation.
2) Download and install the Matlab Compiler Runtime. For sfm_georef v2.2 the version required
is 7.14 and can be downloaded from:

www.lec.lancs.ac.uk/data/mike_james_project/MCR7.14Installer.exe

The runtime version appropriate to the compiler version used is required. If you already have
Matlab Compiler Runtimes installed, multiple versions can co-exist without difficulty.

1
Use of sfm_georef can be divided into three stages:

1) Reading the SfM project and importing control measurements

2) Identifying the control points in the image set, calculating the equivalent 3d positions in the
SfM coordinate system and calculating the transform that relates the SfM and the control
coordinate systems
3) Transforming and merging dense point clouds associated with the SfM project.

Getting started: Typical use of SfM_Georef

1) Start sfm_georef, import SfM project data and control measurements
Start sfm_georef by double clicking on the .exe file (Windows); the main control window will
appear as well as a console window (black). The console window will display any error or other
information, as you use sfm_georef.

The main SfM_Georef control window:

The control window is divided into a menu bar and three main panels:

Images – controls which images are visible and ready to make measurements in. Image numbers
appended with an asterisk have no orientation information from the SfM process (they
could not be incorporated into the model) and can’t be used for geo-referencing

Points (SfM) – records image observations, controls intersection strategy and gives 3d point
coordinates in SfM coordinate system

Control Data – Lists the control data imported, allows point selection for transform and shows the
resulting residuals

Additionally, there are buttons for:

Transform and conversion – Determines the transform between SfM and control coordinate
systems.

Quit – Use this to exit sfm_georef. Warning – this does not yet prompt you to save any data…
unsaved data will be lost unless you are running in Matlab with return arguments.

2
To geo-reference a new SfM reconstruction, the appropriate files need to be loaded using the File
menu command. For:

SFMToolkit or Bundler Photogrammetry Package projects:

File  Import project folder; select the folder that contains your images. For Bundler
Photogrammetry Package projects this will also have a subfolder ‘bundler’ with the file
‘bundle.out’. For SFMToolkit projects, bundle.out will be in a ‘bundler’ subfolder. The
import also reads the file list.txt in the image directory. If any these files are not present,
the import will fail.

VisualSFM:
File  Import project file; select the .svm file from your VisualSFM project

If all the appropriate data are found, then you will see no error messages, the main sfm_georef
window will become populated with the image numbers found in the project, the first image will be
displayed as well as a 3D plot of the relative camera positions in the project.

1) Image window, showing image 1

Image windows are where observations (mouse clicks) in the images can be made. Use the

standard Matlab buttons to navigate around and zoom in to the image – click the
button once to activate a navigation tool, click again to deactivate. [Note: Use of the other buttons
and menus is not directly supported and may produce undesired behaviour]. To define an
observation of a point, locate the point in the image, ensure that none of the Matlab image
navigation tools are active and that you have a cross-hairs cursor, and click. A red cross will mark
the location. Note – if the SfM project did not contain any orientation information for the image,
observations are not allowed.

2) SfM object space 3d plot

This window can be brought up at any time from the main menu ( Windows  3D SfM space ).
The plot gives the 3D positions of cameras in the SfM project coordinate system. Camera positions
are marked as red dots and any intersected point locations are given as blue circles. Use the
standard Matlab buttons to navigate around the plot.

Importing control data:

Control data can be imported via the File Import Control menu item. Control data can be supplied to
sfm_georef in different formats, but you can start with a text file containing the data in three tab or
space separated columns, described by a header line. For example, for 3 control points with (optional)
descriptors:

x y z descriptor
1.122 2.335 0.363 target_1
0.204 34.24 8.929 target_2
9.324 5.252 3.443 target_4

3
If control distances have been measured rather than point coordinates, these are provided with the
header ‘distances’

distances descriptor
6.993 tar_1_to_tar_2
3.442 tar_1_to_tar_3
9.322 tar_2_to_tar_3

Note: Descriptors cannot contain characters that could be determined as delimiters (e.g. spaces
or tabs – ‘target_1’ is OK, ‘target 1’ would produce an error).

Control data can also be directly imported from previously saved sfm_geof projects by importing the
project file via the File Import Control menu item. In this case, only the control information is
imported from the read project, all other data are neglected.

Saving your sfm_georef project

All measurements, transformations, control and the image set can be saved as a sfm_georef
project file via the File Save SfM_Georef project as menu item. The file produced is a binary
Matlab data file, that can be read by Matlab or by sfm_georef. Note that images themselves are
not stored, but their locations are recorded as full paths. If images or SfM folders are moved,
the images will have to be relinked. Once the current project is saved, File Save SfM_Georef
project can be used to update the project file.

Loading a pre-existing sfm_georef project

A previous project can be loaded via the File Import Control menu item

2) Identifying points in the SfM image set and making point observations
To scale or geo-reference the SfM project, the control points need to be identified in the images so
their coordinates in the SfM coordinate system can be determined.
1) Identify a control point that will be visible in multiple images.
2) Show the appropriate images by selecting their image numbers in the Images list box, using
control-click or shift-click as appropriate for multiple selection. Note that points cannot be
determined on images that have no orientation information from the SfM process (those
marked by asterisks next to their image number in the Images list box).
3) Bring the first image to use to the front by either clicking on the window frame or by changing
the ‘Current’ drop down box to the required image number.
4) Locate the feature in the image and click. The image coordinates will be stored as observations
of Point 1, and listed in the ‘Image measurements’ table.
5) Locate the feature in another image and mouse-click to make the observation. The second
observation will be added to the table. With two observations, an intersection will be
calculated; if successful, the 3D coordinates will be provided in the ‘3d SfM coords’ column.

Note:

Image residuals are calculated by projecting the 3D position of the determined point onto the image
plane. Image residuals are shown by green lines on the images, and their scale can be changed by
changing the number in the ‘Residual scale on images:’ box. The default of 100 draws residual vectors
that are 100x as long as the real image residuals. Usually, residual values are around 1 pixel or smaller.

4
Significantly larger residuals indicate that either the image has not been incorporated well in the SfM
project (i.e. the imported camera model or the camera orientations are inaccurate) or – more usually –
that one of the image measurements is actually of the wrong feature.

The residuals column in the table gives the RMS residual value for all points.

You can show the point IDs on the image windows from the main menu ( Image overlays  Point IDs –
useful if you have many points.

To add more points:

To add another point, click the ‘New point’ button. Another point will be added to the point list box,
and will be highlighted. Any observations in image windows will now be attributed to this point. To
make further observations of a different point, make sure that the point number is selected in the
point list box.

To delete an observation:
Make the image in which you want to delete the observation the current image, select the point
number in the point number list box, and press the ‘delete’ key. Note – the point number list box must
be the active control (i.e. the last control clicked on) for the delete to be successful.

Intersections:
By default, intersection calculations are automatically updated for each change in the image
observations. If intersections want to be carried out manually, select the appropriate strategy from
the Intersections menu. For either Intersections  Active point or Intersections  All points,
intersections are only carried out when the Intersect button below the Image measurements table is
clicked.

Current image:
Use this list box to change the image number that is currently active (i.e. the image in which
measurements are expected). Useful when you have many windows open and the one you want is
buried.

Organising control data:

Control data are listed in the order supplied in the input. Descriptors (very helpful) can be added or
edited by typing in the Descriptor column.

The first column in the table is the ‘active’ column. If ticked, the control point is active and, if it has a
valid link to a point in the SfM project, will be used in the calculation of the transform.

Unselect any control points that you do not want to be included in the calculation.

The Link column indicates which point in the SfM project (i.e. in the Points SfM panel) represents the
same feature as the control point defined by that row. The transform will be calculated to minimise
the square of the distance (i.e. least squares) between the transformed SfM points and their linked,
active control points.

Residuals are given when a transform has been calculated, and represent the ‘mismatch’ distances
between the control points and the transformed SfM points, in the control coordinate system. In the
5
table, the residual units are given as a thousandth of the units used for the control coordinates. For
example, if control data are provided in metres, residuals are quoted in millimetres.

3) Transform and conversion

The Control data table comprises four columns:

Tick box – indicates whether or not this control measurement is to be used in calculations of scale or
geo-reference transform

Link – gives the ID number of the SfM point that represents this control point. If using control
distances, each distance measurement is represented by two lines in the Control data table,
giving the IDs of the two points between which the distance measurement has been made

Descriptor – a text description of the control point.

Residuals – gives the mismatch between the control data and the transformed SfM points, once a
transform has been determined.

If sufficient SfM points and active control exist for a scale or 3D transform to be calculated the
‘Calculate scale/transform’ button will become active. Scale can be determined from one or more
distance measurement or two 3D points, full geo-referencing requires three or more 3D points.

If the transform is successful, residuals will be calculated and a 3D plot given, showing the control
points and the residuals. The File  Transform and export menu item will become active allowing
conversion of:

SfM points – The transformed coordinates of the points identified and intersected in the SfM
project are exported to a text file

SfM project – Not yet implemented.

Ply files – Points in .ply files can be imported, transformed and then written back to a .ply file. If
multiple .ply files are selected then they are merged and the output written as one file. This
feature enables the dense point clouds output by CMVS/PMVS to be transformed to the geo-
referenced coordinate system. Note only the ASCII ply files written by Bundler or PMVS are
explicitly supported.

Feedback and bug reports

All feedback and bug reports are welcome; please send to m.james__at__ lancaster.ac.uk (removing
the underscores and using the ‘at’ symbol).
I will endeavour to do my best, but I cannot guarantee answering all emails. Apologies in advance.

If you use sfm_georef, please cite the following work:

James, M. R. and Robson, S. (2012) Automated image-based 3D surface reconstruction for the
geosciences: Accuracy and application, J. Geophys. Res., 117, F03017, doi:10.1029/2011JF002289.
Mike James, Lancaster University 21/06/12