Int J Appl 

Earth Obs Geoinformation 72 (2018) 1–10

Contents lists available at ScienceDirect

Int J Appl Earth Obs Geoinformation

journal homepage: www.elsevier.com/locate/jag

Assessment of UAV-photogrammetric mapping accuracy based on variation T

of ground control points
⁎
Patricio Martínez-Carricondoa, Francisco Agüera-Vegaa, Fernando Carvajal-Ramíreza, ,
Francisco-Javier Mesas-Carrascosab, Alfonso García-Ferrerb, Fernando-Juan Pérez-Porrasb
a
Department of Engineering, University of Almeria, Campus de Excelencia Internacional Agroalimentario (ceiA3), Almería, Spain
b
Department of Graphic Engineering and Geomatics, University of Cordoba, Campus de Rabanales, Córdoba, Spain

A R T I C LE I N FO A B S T R A C T

Keywords: Civil engineering uses digital elevation models (DEMs) and orthophotos as basic material to be able to design
Unmanned aerial vehicle (UAV) and execute any project. UAV photogrammetry has made it possible to obtain this type of information in an
Photogrammetry economic and practical way. However, it is necessary to know the accuracy of the data and that it is within the
Digital surface model (DSM) admissible limits. There are many factors that affect the accuracy of products resulting from UAV photo-
Orthoimages
grammetry. Of all of these, the effect of the number of ground control points (GCPs) and their distribution in the
Ground control points (GCPs)
study area are especially significant. Different distributions of GCPs have been studied to try to optimize the
products obtained by UAV photogrammetry. Of all the distributions tested, the best results were obtained with
edge distribution and stratified distribution. Therefore, it is necessary to place GCPs around the edge of the study
area to minimize planimetry errors. In addition, it is advisable to create a stratified distribution inside the study
area with a density of around 0.5–1 GCP × ha−1 to minimize altimetry errors. The combination of these two
distributions minimizes the total error obtained.

1. Introduction economically viable alternative. Many such cases, UAVs are more
competitive because they require less time for data acquisition and
Civil engineering uses orthophotos and DEMs as essential informa- reduce costs compared to the use of classical manned aircrafts (Aber
tion to be able to design and execute any project, as well as to peri- et al., 2010).
odically monitor the progress of the works. Therefore, it is necessary to The integration of photogrammetry and computer vision (Atkinson,
know the accuracy of the data and that it is within the admissible limits 2001; Hartley and Zisserman, 2003) has provided advances in auto-
(Hugenholtz et al., 2013; James et al., 2017; Mancini et al., 2013; mation as a result of the possibility of collecting images from different
Mesas-Carrascosa et al., 2014; Nelson et al., 2009; Ruzgienė et al., heights and in different directions as well as greater flexibility and high-
2015). quality results (Fernández-Hernandez et al., 2015). Nowadays, there
In order to obtain DEMs there are several techniques that can be are several low-cost software applications that allow us to carry out 3D
used to achieve high precision results. Such is the case of Global modelling of surfaces from photographs taken with conventional cam-
Navigation Satellite Systems (GNSSs), Terrestrial Laser Scanning (TLS), eras. Most of these software applications are based on special algo-
or Total Stations (TSs) (Lague et al., 2013), and airborne sensors, such rithms such as Structure-from-Motion (SfM) (Fonstad et al., 2013;
as light detection and ranging (LIDAR) or photogrammetric cameras Javernick et al., 2014; Westoby et al., 2012). SfM is a photogrammetric
(Sallenger et al., 2003). UAV photogrammetry has opened a variety of technique that automatically solves the geometry of the scene, the
new applications in the field of close range photogrammetry by com- camera positions, and the orientation without requiring a priori speci-
bining aerial and terrestrial photogrammetry techniques to offer the fication of a network of targets that have known 3D positions (Snavely
advantages of both. Thus, the application of UAV photogrammetry in et al., 2008; Vasuki et al., 2014; Westoby et al., 2012). SfM incorporates
the field of civil engineering can be situated between techniques using multi-view stereopsis techniques (Furukawa and Ponce, 2010), which
classic terrestrial systems and techniques based on photogrammetry derive the 3D structure from overlapping photography acquired from
from images taken from conventional aircraft, representing an multiple locations and angles. Lowe (2004) and Snavely et al. (2008)

⁎
Corresponding author.
E-mail addresses: [email protected] (P. Martínez-Carricondo), [email protected] (F. Agüera-Vega), [email protected] (F. Carvajal-Ramírez), [email protected] (F.-J. Mesas-Carrascosa),
[email protected] (A. García-Ferrer), [email protected] (F.-J. Pérez-Porras).

https://doi.org/10.1016/j.jag.2018.05.015
Received 13 February 2018; Received in revised form 19 May 2018; Accepted 29 May 2018
0303-2434/ © 2018 Elsevier B.V. All rights reserved.
P. Martínez-Carricondo et al. Int J Appl  Earth Obs Geoinformation 72 (2018) 1–10

Fig. 1. Flowchart of the methodology.

applied the scale-invariant feature transform (SIFT) operator for key- precipitation) degrade the final quality and accuracy of a photogram-
point detection to generate 3D point clouds from photographs. Certain metric product by an average of 25%. Tahar and Ahmad (2013) com-
studies concluded that this operator is one of the most robust to large pared two different UAV units: a fixed wing one and a multi-rotor one.
image variations (Juan and Gwun, 2009; Remondino and El-Hakim, Both UAVs were used to obtain 3D coordinates of slope area and to
2006). determine the accuracy of the photogrammetric product produced from
In contrast to classic aerial photogrammetry, which required rig- both UAVs. Eisenbeiss and Sauerbier (2011) carried out an investiga-
orous flight planning and pre-calibration of the cameras (Kamal and tion of UAV systems and flight modes for photogrammetric applica-
Samar, 2008), SfM provides simplicity in the process, with no need for tions.
exhaustive planning or calibration of cameras, even though images One of the factors with the greatest influence on the accuracy of the
from different cameras can be used. DSM and orthophoto resulting from the photogrammetric process is the
The camera positions derived from the SfM algorithm do not have number and distribution of the GCPs. In addition, measurement of the
the scale and orientation provided by the coordinates of ground control coordinates of the GCPs is a time-consuming task and can sometimes be
points (GCPs). Consequently, the 3D point cloud is generated in relative difficult to carry out due to the morphology of the terrain. In view of the
coordinates, referring to the image coordinate system. The georefer- above, it is concluded that it is necessary to deepen the knowledge of
encing of this model is generally carried out using a small number of this relationship. Therefore, the objective of this paper is to analyse the
GCPs located in clearly visible locations both on the ground and in each influence of the number and distribution of GCPs on the accuracy of
of the photographs. DSMs and orthophotos derived from UAV photogrammetry.
Once the resulting products have been exported, it is necessary to
evaluate the accuracy obtained in the photogrammetric project. The 2. Materials and methods
evaluation of accuracy is usually based on the statistical Root Mean
Square Error (RMSE). In the last years, numerous studies have been The methodology used to assess the photogrammetric mapping ac-
undertaken to evaluate the accuracy of products obtained by UAV curacy based on variation of the GCPs’ distribution using UAVs is
photogrammetry. Many authors have studied the influence that certain summarized in Fig. 1.
parameters have on the accuracy of the resulting products. Agüera-Vega
et al. (2016) studied the influence of flight altitude, terrain mor-
2.1. Study site
phology, and number of GCPs on Digital Surface Model (DSM) and
orthophoto accuracy. Amrullah et al. (2016) tested the effect of com-
The study area is located in Campo de Níjar (Almería), southeast
bining non-metric oblique and vertical camera along with the config-
Spain (Fig. 2). The southwest and northeast UTM coordinates (Zone 30,
uration of the GCPs to improve the precision and accuracy in UAV
ETRS89) of this area are (581918, 4,094,212) and (582338, 4,094,632),
photogrammetry projects. Dandois et al. (2015) studied the optimal
respectively. So, the area of the plot was 420 × 420 m, which covers
altitude, overlap, and weather conditions for computer vision UAV es-
17.64 ha. The selection of the study area was based on its morphology,
timates of forest structure. Mesas-Carrascosa et al. (2016) carried out an
which includes a wide range of slope values. The average slope of the
analysis of the influence of flight parameters (flight altitude above
terrain is 6.45% and its standard deviation is 5.399%. The elevation
ground level, forward and side overlap, and the use, or lack thereof, of
range is about 26 m, varying from 220 to 246 m above mean sea level
GCPs) in the generation of UAV orthomosaics to survey archaeological
(MSL). The study area has almost no vegetation, except in the southeast
areas. Vautherin et al. (2016) compared traditional (non-rolling
corner where there are bushes and trees 4–5 meters high.
shutter) camera models against a newly implemented rolling shutter
model with respect to both the accuracy of geo-referenced validation
points and the quality of the motion estimation. Furthermore, they were 2.2. Image acquisition
able to show that the speed of the UAV (and its direction) can be solely
estimated from the rolling shutter effect of the camera. Jaud et al. The images used in this work were taken from a rotatory wing UAV
(2016) and Murtiyoso and Grussenmeyer (2017) assessed the accuracy with eight rotors and MikroKopter electronic boards and motors. The
of 3D reconstruction computed by different software applications. UAV was equipped with a motion compensated gimbal and a Nikon D-
Wierzbicki et al. (2015) studied the influence of UAV image quality on 3100 digital reflex camera with a lens with a fixed focal length of
orthophoto production. Based on the accuracy analysis performed, it 16 mm. The resolution of the camera sensor was 14.2 megapixels
was stated that images acquired in poor weather conditions (cloudy, (4608 × 3072). Fig. 3 shows the system described above.
The flight altitude was 120 m above ground level, which implies a

2
P. Martínez-Carricondo et al. Int J Appl  Earth Obs Geoinformation 72 (2018) 1–10

Fig. 2. Location of the study area. Coordinates are referred to UTM Zone 30 N (European Terrestrial Reference System 1989, ETRS89).

the Spanish National Geographic Institute, are 582,655.945 m,

4,093,630.095 m, and 240.883 m, respectively. Horizontal coordinates
are referred to UTM Zone 30 N (European Terrestrial Reference System
1989, ETRS89) and the elevation is referred to the MSL using the
EGM08 geoid model. Both rover and base GNSS receivers were Trimble
R6 systems. For RTK measurements, these dual-frequency geodetic in-
struments have a manufacturer’s stated accuracy specification of ±
1 cm + 1 ppm horizontal RMS and ± 2 cm + 1 ppm vertical RSM. As
the distance between the base station and the study area was ap-
proximately 670 m, the horizontal and vertical errors were around 1
and 2 cm.

2.3. Photogrammetric processing

Fig. 3. The UAV octocopter used in this work as the photogrammetric platform. The photogrammetric process was carried out using the software
package Agisoft PhotoScan Professional, version 1.2.4.2399 (Agisoft
PhotoScan, 2015). This kind of photogrammetric software based on SfM
surface of 190 × 113.75 m2 covered by every photo and an equivalent
algorithm was used because it has been proven to outperform other
ground sample distance of 3.291 cm. These values correspond to the
software in terms of accuracy (Sona, 2014). The workflow is a three-
UAV take-off point, located in the lower elevation part of the study
step process (Verhoeven, 2011). The first step is the alignment of the
area. According to the flight altitude, UAV speed, and light conditions
images by feature identification and feature matching. While carrying
at the flight time, the shutter speed was adjusted to minimize the
out image alignment, this software estimates both internal and external
blurring effect on the images taken. The flight plan was carried out in
camera orientation parameters, including nonlinear radial distortion.
navigation mode and consisted of 10 passes, and a total of 160 images
Only an approximate focal length value is required, which is extracted
were selected to carry out all the photogrammetric projects. The camera
automatically from the EXIF metadata. This task was carried out with
was triggered every two seconds by a controller on the UAV and the
the PhotoScan accuracy set to high. The result of this step is the camera
flight speed was set to obtain forward and side overlaps of 90 and 80%
position corresponding to each picture, the internal calibration para-
respectively.
meters, and the 3D coordinates of a sparse point cloud of the terrain. In
Prior to the image acquisition, 72 targets were scattered on the
the second step, the sparse point cloud is referenced to a local co-
studied surface for the purpose of georeferencing (GCPs) and assessing
ordinate system (ETRS89 and frames in the UTM, in the case of this
the accuracy of the DSM and orthophotos (Check Points, CPs). The
study) and densification of the point cloud is achieved using the height
targets consisted of an A3 size (420 × 297 mm) red paper with a black
field method, which is based on pairwise depth map computation. This
circle inside. The locations of these targets are shown in Fig. 4. Three-
resulted in a more detailed 3D model. The third step applies a texture to
dimensional coordinates of these points were measured with a GNSS
the mesh obtained in the previous step. Finally, the orthophoto is ex-
receptor working in Real Time Kinematic (RTK) mode, with the base
ported and a grid DSM can be generated from the point cloud. The
station on a geodesic pillar located closer than one kilometre from the
bundle adjustment can be carried out using three GCPs at least, but
studied surfaces. The three-dimensional coordinates of the geodesic
more accurate results are obtained if more GCPs are used, and it is
pillar, named Cerro Gordo II (Instituto Geográfico Nacional, 2017) by
recommended that more of them be used to obtain optimal accuracy

3
P. Martínez-Carricondo et al. Int J Appl  Earth Obs Geoinformation 72 (2018) 1–10

Fig. 4. (a) Locations of the 72 targets used as GCPs or CPs. Coordinates are referred to the UTM Zone 30 N (European Terrestrial Reference System 1989, ETRS89)
and detail of one of the targets. (b) Geodesic pillar Cerro Gordo II and base GNSS receiver.

Fig. 5. Reference project: (a) Camera locations and image overlap. Blue colour indicates that the terrain point appears in more than nine photographs; (b) DEM (For
interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article).

(Rosnell and Honkavaara, 2012). Taking into account this and the re- (a) In the edge distribution, the GCPs are placed around the edge of the
sults of Tahar (2013) and Agüera-Vega et al. (2017), who observed that study area.
accuracy improved as the number of GCPs increased from four to (b) In the central distribution, the GCPs are placed in the centre of the
twenty, to determine the influence of GCP distribution on the accuracy study area.
of DSMs and orthophotos produced, a first project known as the Re- (c) In the corner distribution, the GCPs are placed in the corners of the
ference Project (RP) was carried out using the 72 targets as GCPs. Fig. 5 study area
shows two details of the RP processing. (d) In the stratified distribution, the GCPs are placed in a stratified way.
Once processed, several photogrammetric projects were carried out (e) In the random distribution, the GPCs are placed randomly
with different distributions of GCPs. The aim of all these projects was to throughout the study area.
evaluate their accuracy and compare it with that obtained in the RP.
Five different GCP distributions were designed for this purpose, as For each type of distribution, the number of GCPs used for the
shown in Fig. 6: photogrammetric bundle adjustment was varied. Twelve different

4
P. Martínez-Carricondo et al. Int J Appl  Earth Obs Geoinformation 72 (2018) 1–10

Fig. 6. Five different distribution types and an example for the combination of 16 GCPs. The shaded area indicates the area where GCPs have been selected. The
numbers of GCPs and CPs varied depended on the case. (a) edge distribution, (b) central distribution, (c) corner distribution (equivalent for all four corners), (d)
stratified distribution, (e) random distribution.

combinations were taken: 4, 5, 6, 7, 8, 9, 12, 16, 20, 24, 30, and 36 account its coordinates X and Y, measured on the orthophoto.
GCPs. In addition, each of these combinations was carried out with five (b) ZGNSSi is the Z coordinate of the ith CP measured with GNSS.
different replicates. In total, 5 (distribution types) × 12 (combinations
of GCPs) × 5 (repeats) = 300 projects were processed. Using these values, RMSET can be obtained from Eq. (5):

2.4. Accuracy assessment RMSET = RMSEXY 2 + RMSEZ 2 (5)

The accuracy of all photogrammetric projects was evaluated using

the surveyed points that had not been used for georeferencing (CPs), 2.5. Comparison of point clouds
using the typical root mean square error (RMSE) formulation (Agüera-
Vega et al., 2017; James et al., 2017; Tahar, 2013).To this end, the CPs In order to compare the clouds generated in the photogrammetric
were identified in the orthoimages and their coordinates were com- projects with the one extracted in the RP, the M3C2 (Multiscale Model
pared to the surveyed GNSS coordinates, resulting in RMSEX, RMSEY, to Model Cloud Comparison) offered by the CloudCompare v2.8
and RMSEXY horizontal accuracy measures, as defined in Eqs. (1)–(3) (Girardeau-Montaut, 2017) plugin was selected as it is the only way to
respectively: compute signed and robust distances directly between two point clouds
n (Lague et al., 2013). This algorithm allows a reliable comparison be-
∑i = 1 (XOi −XGNSSi )2
RMSEX = tween two point clouds, yielding indicators about the degree of con-
n (1)
fidence of the results. In addition, the algorithm indicates positive and
n negative differences according to the cloud chosen as reference. The
∑i = 1 (YOi−YGNSSi )2
RMSEY = analysis is performed on the use of a set of calculation core points for
n (2)
which one distance and confidence interval are calculated. The point
n clouds themselves may also be used directly if so desired. In our case,
∑i = 1 [(XOi −XGNSSi )2 + (YOi−YGNSSi )2]
RMSEXY = the cloud of the RP has been selected as the core points (3,085,618
n (3)
points).
where: The M3C2 algorithm works in two steps:

(a) n is the number of CPs tested for each project. (1) Generation of a normal vector (one for each core point i) to the
(b) XOi and YOi are the X and Y coordinates, respectively, measured in centre of a circle of a certain diameter D (normal scale). The stan-
the orthophoto for the ith CP. dard deviation of the distance of the points from the interpolated
(c) XGNSS and YGNSSi are the X and Y coordinates, respectively, mea- plane could be a measure of the cloud roughness σ(D)i. In our case,
sured with GNSS for the ith CP. a normal scale of 0.828491 m was chosen to take into account a
minimum overlap between the circles generated around every core
Furthermore, the height value was derived from the grid DSM for point.
the X and Y coordinates of the CP on the orthoimage and it was also (2) Once the normal is defined for the core point i, it is used to project i
compared to the GNSS coordinate, producing an RMSEZ accuracy onto each cloud at scale d (projection scale, 1.656982 in this work).
measure for the Z direction, as defined in Eq. (4): Then the distance LM3C2 along the normal within a cylinder of
n
diameter d connecting the clouds is calculated. A maximum cy-
∑i = 1 (ZOi−ZGNSSi )2 linder length can be set to speed up the calculations.
RMSEZ =
n (4)
where: Among other data, the output of M3C2 consists of a text file with X,
Y, Z coordinates for each point of the reference cloud and the associated
(a) ZOi is the height in the ith CP, derived from the DSM, taking into 3D distance to the compared. All the data can be displayed using a
colour scale to differentiate the resulting scalar field.

5
P. Martínez-Carricondo et al. Int J Appl  Earth Obs Geoinformation 72 (2018) 1–10

Fig. 7. RMSEXY obtained for each GCP distribution type: edge (1), central (2), corner (3), stratified (4), and random (5). The horizontal line represents the RMSEXY for
the RP. Only data below 0.10 m have been plotted.

3. Results adding some GCPs in the centre, taking for example 20 GCPs around the
edge and two GCPs in the centre, we obtained an average RMSEXY equal
The accuracy obtained in the bundle adjustment of the RP was to 0.036 m which is practically equal to that obtained when 20 GCPs
0.032 m in planimetry and 0.031 m in altimetry. These values were were placed only around the edge.
obtained using all the targets as GCPs by the formulation described in Fig. 8 shows the average values of the five RMSEz derived from the
Section 2.4. photogrammetric projects corresponding to the five replications of the
Fig. 7 shows the average values of the five RMSEXY derived from the 12 combinations of GCPs and the five distribution types. In order to
photogrammetric projects corresponding to the five replications of the make this figure clearer, values of RMSEZ greater than 0.10 m have not
12 combinations of GCPs and the five distribution types. In order to been plotted. Furthermore, the horizontal line represents RMSEZ for the
make this figure clearer, values of RMSEXY greater than 0.10 m have not RP (RMSEZ = 0.031 m). Similarly to RMSEXY, for all types of GCP dis-
been plotted. Furthermore, the horizontal line represents RMSEXY for tributions, higher and lower RMSEZ values were found for four and 36
the RP (RMSEXY = 0.032 m). For all types of GCP distributions, the GCPs, respectively, ranging from 0.954 to 0.156 m for the corner dis-
highest and lowest values were found for 4 and 36 GCPs, respectively. tribution type, from 0.365 to 0.094 m for the central distribution, from
RMSEXY values ranged from to 0.610 to 0.102 m for the corner dis- 0.307 to 0.057 m for the random distribution, from 0.308 to 0.043 m for
tribution type, from 0.203 to 0.067 m for the central distribution, from the stratified distribution, and from 0.474 to 0.048 m for the edge
0.142 to 0.046 m for the random distribution, from 0.094 to 0043 m for distribution.
stratified distribution, and from 0.067 to 0.035 m for edge distribution. In this case, the results showed that the stratified distribution ob-
In view of these values plotted in Fig. 7, one can deduce that both the tained the best vertical accuracy. For 20 and 24 GCPs (approximately
number and the distribution of GCPs have an important effect on the 1.1 and 1.3 GCPs × ha−1, respectively), this distribution of GCPs
accuracy of the DSMs and orthophotos derived from the UAV-photo- yielded RMSEZ equal to 0.047 m, and from 30 and 36 GCPs (1.7 and 2
grammetric projects. The analysis of the results shows that the edge GCPs × ha−1, respectively), this value was practically constant and was
distribution obtained the best horizontal accuracy. From eight GCPs equal to 0.043 m.
(GCPs separated by 210 m), this distribution of GCPs yielded RMSEXY The edge distribution yielded maximum vertical accuracies of
under 0.05 m, and from 20 to 36 GCPs (GCPs separated by 84 m and around 0.05 m when 30 or more GCPs were used. For 20 and 24 GCPs,
GCPs separated by 46.7 m) this value was practically constant and was the accuracy was around 0.06 m. With the random distribution of GCPs,
equal to 0.035 m, which was practically equal to that found for the RP, the RMSEZ values were erratic. From four to 12 GCPs, the RMSEZ de-
which was obtained from 72 GCPs arranged on the whole of the surface. creased, but from 16 to 36 GCPs there was no clear trend, although
Of all these 72 GCPs used in the RP, only 36 were in the edge of the values were between 0.047 and 0.057 m. Central and corner distribu-
studied surface, so the internal GCPs added to these 36 did not improve tions showed the worst RMSEZ values. They ranged from 0.378 to
the horizontal accuracy. No other distribution type reached accuracies 0.094 m for the central distribution and from 0.954 to 0.116 m for the
under 0.04 m. RMSEXY for the stratified distribution was under 0.05 m corner distribution.
when the number of GCPs was 16 (approximately 1 GCP × ha−1) and it In view of these results, if the optimal combination for minimizing
decreased to reach a value close to but greater than 0.04 m when 36 the horizontal component (0.035 m) is selected (edge distribution, 20
GCPs where used (approximately 2 GCP × ha−1). The random dis- GCPs or one GCP every 84 m), the vertical accuracy is 0.062 m. In the
tribution had to use 24 or more GCPs to achieve an RMSEXY value under same way, if the optimal combination for minimizing the vertical
0.05 m. component (0.047 m) is selected (stratified distribution, 30 GCPs, ap-
If we consider the standard recommendations for GCPs placement in proximately 1.7 GCPs × ha−1), the horizontal accuracy is 0.045 m,
conventional photogrammetry, placing GCPs around the edge but also which is similar to the vertical accuracy. When 36 GCPs were used in

6
P. Martínez-Carricondo et al. Int J Appl  Earth Obs Geoinformation 72 (2018) 1–10

Fig. 8. RMSEz obtained for each GCP distribution type: edge (1), central (2), corner (3), stratified (4), and random (5). The horizontal line represents the RMSEz for
the RP. Only data below 0.10 m have been plotted.

the edge, the horizontal accuracy continued to be 0.035 m and the better when the number of GCPs increased (from 0.490 to 0.46 m for
vertical one was reduced to 0.048 m. Therefore, the maximum vertical the horizontal component and from 0.830 to 0.780 m for the vertical
accuracy could be reached by increasing the number GCPs in the edge component). The densities of GCPs used were very low in comparison to
distribution, which is important when only the edge of the study site is those used in the present work and this could be the reason why Tahar’s
accessible. results were worse than those obtained in the present work. Carvajal-
If we consider the standard recommendations for GCPs placement in Ramírez et al. (2016) proposed different photogrammetric projects to
conventional photogrammetry, placing GCPs around the edge but also establish the ideal distribution of GCPs in a study area of a road land-
adding some GCPs in the centre, taking for example 20 GCPs around the slide. They only used three GCPs to cover an area of less than 1 ha and
edge and two GCPs in the centre, average RMSEZ was equal to 0.054 m, all combinations of GCPs were in the edge of the studied site. They
better than the results obtained when only 20 GCPs were used on the deduced that the best distribution for minimizing both horizontal and
edge (0.062 m). However, the results did not improve those obtained in vertical error was to distribute the three GCPs regularly along the edge.
the stratified distribution (0.047 m). With this distribution, the GCPs were separated by 180 m and they
Fig. 9 shows two examples of comparisons between the RP cloud obtained RMSEXY equal to 0.058 m and RMSEZ equal to 0.100 m. In this
and the point cloud of the photogrammetric project. In both cases, the work, the equivalent distance between consecutive GCPs for the edge
areas with the lowest differences with respect to the RP corresponded to distribution yielded RMSEXY equal to 0.047 and RMSEZ equal to
the areas where the GCPs were located. For the two examples of dis- 0.100 m. Reshetyuk and Mårtensson (2016) investigated the influence
tributions, at the outer limits of the location of the GCPs, the point on the accuracy of the products of UAV photogrammetry projects of
clouds tend to be below the RP point cloud. However, for the edge different variations of GCPs arranged on an area of 2.73 ha A total of
distribution, as we move away from this zone, the differences increase five GCPs were uniformly distributed on the whole surface, which is
proportionally to the distance, and the point cloud tends to be above the equivalent to approximately to 1.8 GCPs × ha−1. Furthermore, two
RP point cloud. For the stratified distribution, the central zone showed flight altitudes were considered: 81 and 163 m. Reported values of
little difference from the RP. In all configurations, the greatest differ- RMSET (horizontal and vertical components) were around 0.030 m for
ences were found in the southeast zone, corresponding to the wooded the flight altitude of 81 m and 0.080 m for the flight altitude of 163 m.
zones. The distributions of both configurations were assimilated to a With the same GCP distribution, the equivalent accuracy reached in this
Gauss distribution with the following values. For the edge distribution, work with a flight altitude of 120 m had an intermediate value
the mean value was 0.024 m with a standard deviation of 0.036 m. For (0.058 m) between those reported by Reshetyuk and Mårtensson
the stratified distribution, the mean value was –0.006 m with a standard (2016). Taking into account that as the flight height increases the ac-
deviation of 0.024 m. In addition, the histograms show the distribution curacy gets worse (Agüera-Vega et al., 2016), the intermediate accuracy
values below 0.20 m. value reached in this work is coherent with the intermediate flight al-
titude. Tonkin and Midgley (2016) produced 16 DSMs from a UAV
4. Discussion survey using a varying number of GCPs (three to 101). These DSMs
were compared to 530 GNSS spot heights to calculate vertical error. All
In the scientific literature, there is very little research that focuses DSMs produced reasonable surface reconstructions (RMSEZ < 0.2 m);
on studying the effect of the number and distribution of GCPs on DSMs however, an improvement in DSM quality was found where four or
and orthophotos derived from UAV photogrammetry projects. Tahar more GCPs (up to 101 GCPs) were applied, with errors falling within
(2013) studied this effect on a surface of 150 ha and observed that for the suggested point quality range of the survey equipment used for GCP
any number and distributions of GCPs studied, as in this work, the acquisition (e.g., vertical RMSE of < 0.09 m). They also studied the
horizontal accuracy was better than the vertical one, and both were influence of a poor GCP distribution by producing a DSM using an

7
P. Martínez-Carricondo et al. Int J Appl  Earth Obs Geoinformation 72 (2018) 1–10

Fig. 9. Two examples of comparisons between the RP cloud and the point cloud of the photogrammetric project. The black spots represent the locations of the GCPs
used in the bundle adjustment. Statistical data comparing the RP cloud with that of the photogrammetric project are shown. In addition, the histograms are also
displayed limited to values less than 0.20 m. (a) Edge distribution and 20 GCPs. (b) Stratified distribution and 30 GCPs.

8
P. Martínez-Carricondo et al. Int J Appl  Earth Obs Geoinformation 72 (2018) 1–10

evenly distributed network of GCPs and comparing it to a DSM pro- Trans. Pattern Anal. Mach. Intell. 32, 1362–1376. http://dx.doi.org/10.1109/TPAMI.
duced using a clustered network of GCPs. These results accord with 2009.161.
Girardeau-Montaut, D., 2017. CloudCompare.
those obtained in this work, where vertical error increases with distance Hartley, R., Zisserman, A., 2003. Multiple View Geometry in Computer Vision.2nd.
from the area where the GCPs are located. Cambridge University Presshttp://dx.doi.org/10.1017/CBO9781107415324.004.
We have not carried out a more detailed study of the distribution of Hugenholtz, C.H., Whitehead, K., Brown, O.W., Barchyn, T.E., Moorman, B.J., LeClair, A.,
Riddell, K., Hamilton, T., 2013. Geomorphological mapping with a small unmanned
GCPs proposed by conventional photogrammetry, since if you have aircraft system (sUAS): feature detection and accuracy assessment of a photo-
access to the interior of the study area, you will obtain better results grammetrically-derived digital terrain model. Geomorphology 194, 16–24. http://dx.
with the stratified distribution. doi.org/10.1016/j.geomorph.2013.03.023.
Instituto Geográfico Nacional [WWW Document], 2017. URL http://www.ign.es/web/
ign/portal (Accessed 9.22.17).
5. Conclusions James, M.R., Robson, S., d’Oleire-Oltmanns, S., Niethammer, U., 2017. Optimising UAV
topographic surveys processed with structure-from-motion: Ground control quality,
quantity and bundle adjustment. Geomorphology. http://dx.doi.org/10.1016/j.
As a result of the above analysis, it is evident that it is necessary to
geomorph.2016.11.021.
make a detailed study of the locations of GCPs in order to maximize the Jaud, M., Passot, S., Le Bivic, R., Delacourt, C., Grandjean, P., Le Dantec, N., 2016.
accuracy obtained in photogrammetric projects. In order to achieve Assessing the accuracy of high resolution digital surface models computed by
optimum results in planimetry, we must place the GCPs on the edge of PhotoScan® and MicMac® in sub-optimal survey conditions. Remote Sens. 8. http://
dx.doi.org/10.3390/rs8060465.
the study area. However, this configuration does not optimize the re- Javernick, L., Brasington, J., Caruso, B., 2014. Modeling the topography of shallow
sults in altimetry. This means that GCPs must be placed inside the study braided rivers using structure-from-motion photogrammetry. Geomorphology 213,
area with a stratified distribution. As the density of GCPs increases, the 166–182. http://dx.doi.org/10.1016/j.geomorph.2014.01.006.
Juan, L., Gwun, O., 2009. A comparison of sift, pca-sift and surf. Int. J. Image Process. 3,
accuracy will improve until such a time as the results are not improved 143–152. http://dx.doi.org/10.1007/s11270-006-2859-8.
So, if the optimal combination for optimizing horizontal accuracy Kamal, W.A., Samar, R., 2008. A mission planning approach for UAV applications.
(0.035 m) is selected (edge distribution, GCPs separated by 84 m), the Proceedings of the IEEE Conference on Decision and Control 3101–3106. http://dx.
doi.org/10.1109/CDC.2008.4739187.
vertical accuracy increases to 0.062 m. If the optimal combination for Lague, D., Brodu, N., Leroux, J., 2013. Accurate 3D comparison of complex topography
optimizing vertical accuracy (0.047 m) is selected (stratified distribu- with terrestrial laser scanner: application to the Rangitikei canyon (N-Z). ISPRS J.
tion, approximately 1.7 GCPs × ha−1), the horizontal accuracy is Photogramm. Remote Sens. 82, 10–26. http://dx.doi.org/10.1016/j.isprsjprs.2013.
04.009.
0.045 m, which is similar to the vertical accuracy. When the GCPs are in Lowe, D.G., 2004. Distinctive image features from scale-invariant keypoints. Int. J.
the edge, separated by 46.7 m, the horizontal accuracy is 0.035 m and Comput. Vis. 60, 91–110. http://dx.doi.org/10.1023/B:VISI.0000029664.99615.94.
the vertical one is 0.048 m. When making a UAV photogrammetric Mancini, F., Dubbini, M., Gattelli, M., Stecchi, F., Fabbri, S., Gabbianelli, G., 2013. Using
unmanned aerial vehicles (UAV) for high-resolution reconstruction of topography:
survey, one of the most time consuming and limiting tasks in terms of
the structure from motion approach on coastal environments. Remote Sens. 5,
access to the area is the positioning of the GCPs. It is thus a great help 6880–6898. http://dx.doi.org/10.3390/rs5126880.
when planning a UAV photogrammetry project to have the magnitude Mesas-Carrascosa, F.J., Notario-García, M.D., de Larriva, M.D.N.G., de la Orden, M.S.,
of the errors in relation to the GCPS distribution chosen in advance. Porras, A.G.F., 2014. Validation of measurements of land plot area using UAV ima-
gery. Int. J. Appl. Earth Obs. Geoinf. 33, 270–279. http://dx.doi.org/10.1016/j.jag.
Therefore, best accuracies are achieved placing GCPs around the edge 2014.06.009.
of the study area, but it is also essential to place GCPs inside the area Mesas-Carrascosa, F.J., García, M.D.N., De Larriva, J.E.M., García-Ferrer, A., 2016. An
with stratified distribution to optimize vertical accuracy. In this way, analysis of the influence of flight parameters in the generation of unmanned aerial
vehicle (UAV) orthomosaicks to survey archaeological areas. Sensors (Switz.) 16.
the total error obtained is minimized. http://dx.doi.org/10.3390/s16111838.
Murtiyoso, A., Grussenmeyer, P., 2017. Documentation of heritage buildings using close-
References range UAV images: dense matching issues, comparison and case studies.
Photogramm. Rec. 32, 206–229. http://dx.doi.org/10.1111/phor.12197.
Nelson, A., Reuter, H.I., Gessler, P., 2009. Dem production methods and sources. Dev.
Aber, J.S., Marzolff, I., Ries, J.B., Photography, S.A., 2010. Small-Format Aerial Soil. Sci. http://dx.doi.org/10.1016/S0166-2481(08)00003-2.
Photography, Small-Format Aerial Photography. http://dx.doi.org/10.1016/B978-0- Remondino, F., El-hakim, S., 2006. Image-based 3D modelling: a review. Photogramm.
444-53260-2.10008-0. Rec. http://dx.doi.org/10.1111/j.1477-9730.2006.00383.x.
Agisoft PhotoScan, 2015. Reshetyuk, Y., Mårtensson, S.-G., 2016. Generation of highly accurate digital elevation
Agüera-Vega, F., Carvajal-Ramírez, F., Martínez-Carricondo, P., 2016. Accuracy of digital models with unmanned aerial vehicles. Photogramm. Rec. 31, 143–165. http://dx.
surface models and orthophotos derived from unmanned aerial vehicle photo- doi.org/10.1111/phor.12143.
grammetry. J. Surv. Eng. 4016025. http://dx.doi.org/10.1061/(ASCE)SU.1943- Rosnell, T., Honkavaara, E., 2012. Point cloud generation from aerial image data acquired
5428.0000206. by a quadrocopter type micro unmanned aerial vehicle and a digital still camera.
Agüera-Vega, F., Carvajal-Ramírez, F., Martínez-Carricondo, P., 2017. Assessment of Sensors 12, 453–480. http://dx.doi.org/10.3390/s120100453.
photogrammetric mapping accuracy based on variation ground control points Ruzgienė, B., Berteška, T., Gečyte, S., Jakubauskienė, E., Aksamitauskas, V.Č., 2015. The
number using unmanned aerial vehicle. Measurement 98, 221–227. http://dx.doi. surface modelling based on UAV photogrammetry and qualitative estimation.
org/10.1016/j.measurement.2016.12.002. Measurement 73, 619–627. http://dx.doi.org/10.1016/j.measurement.2015.04.018.
Amrullah, C., Suwardhi, D., Meilano, I., 2016. Product accuracy effect of oblique and Sallenger, A.H., Krabill, W.B., Swift, R.N., Brock, J.C., List, J.H., Hansen, M., Holman,
vertical non-metric digital camera utilization in Uav-photogrammetry to determine R.A., Manizade, S., Sontag, J., Meredith, A., Morgan, K., Yunkel, J.K., Frederick, E.B.,
fault plane. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. III-6, 41–48. http:// Stockdon, H.F., 2003. Evaluation of airborne topographic lidar for quantifying beach
dx.doi.org/10.5194/isprsannals-III-6-41-2016. changes. J. Coast. Res. 19, 125–133. http://dx.doi.org/10.2307/4299152.
Atkinson, K.B., 2001. Close Range Photogrammetry and Machine Vision. Whittles Snavely, N., Seitz, S.M., Szeliski, R., 2008. Modeling the world from internet photo col-
Publishing. lections. Int. J. Comput. Vis. 80, 189–210. http://dx.doi.org/10.1007/s11263-007-
Carvajal-Ramírez, F., Agüera-Vega, F., Martínez-Carricondo, P.J., 2016. Effects of image 0107-3.
orientation and ground control points distribution on unmanned aerial vehicle Sona, Siovanna., 2014. Experimental Analysis of Different Software Packages for
photogrammetry projects on a road cut slope. J. Appl. Remote Sens. 10, 34004. Orientation and Digital Surface Modelling from UAV Images. Earth Sci. Info. 7 (2),
http://dx.doi.org/10.1117/1.JRS.10.034004. 97–107.
Dandois, J.P., Olano, M., Ellis, E.C., 2015. Optimal altitude, overlap, and weather con- Tahar, K.N., 2013. An Evaluation On Different Number Of Ground Control Points In
ditions for computer vision uav estimates of forest structure. Remote Sens. 7, Unmanned Aerial Vehicle Photogrammetric Block XL. pp. 27–29.
13895–13920. http://dx.doi.org/10.3390/rs71013895. Tahar, K., Ahmad, A., 2013. An evaluation on fixed wing and multi-rotor UAV images
Eisenbeiss, H., Sauerbier, M., 2011. Investigation of UAV systems and flight modes for using photogrammetric image processing. Int. J. Comput. Electr. Autom. Control Inf.
photogrammetric applications. Photogramm. Rec. 26, 400–421. http://dx.doi.org/ Eng. 7, 48–52.
10.1111/j.1477-9730.2011.00657.x. Tonkin, T.N., Midgley, N.G., 2016. Ground-control networks for image based surface
Fernández-Hernandez, J., González-Aguilera, D., Rodríguez-Gonzálvez, P., Mancera- reconstruction : an investigation of optimum survey designs using UAV derived
Taboada, J., 2015. Image-based modelling from unmanned aerial vehicle (UAV) imagery and. Remote Sens. 8, 16–19. http://dx.doi.org/10.3390/rs8090786.
photogrammetry: an effective, Low-cost tool for archaeological applications. Vasuki, Y., Holden, E.J., Kovesi, P., Micklethwaite, S., 2014. Semi-automatic mapping of
Archaeometry 57, 128–145. http://dx.doi.org/10.1111/arcm.12078. geological structures using UAV-based photogrammetric data: an image analysis
Fonstad, M.A., Dietrich, J.T., Courville, B.C., Jensen, J.L., Carbonneau, P.E., 2013. approach. Comput. Geosci. 69, 22–32. http://dx.doi.org/10.1016/j.cageo.2014.04.
Topographic structure from motion: a new development in photogrammetric mea- 012.
surement. Earth Surf. Process. Landforms. http://dx.doi.org/10.1002/esp.3366. Vautherin, J., Rutishauser, S., Schneider-Zapp, K., Choi, H.F., Chovancova, V., Glass, A.,
Furukawa, Y., Ponce, J., 2010. Accurate, dense, and robust multiview stereopsis. IEEE Strecha, C., 2016. Photogrammetric accuracy and modeling of rolling shutter

9
P. Martínez-Carricondo et al. Int J Appl  Earth Obs Geoinformation 72 (2018) 1–10

cameras. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. III-3, 139–146. http:// applications. Geomorphology 179, 300–314. http://dx.doi.org/10.1016/j.geomorph.
dx.doi.org/10.5194/isprsannals-III-3-139-2016. 2012.08.021.
Verhoeven, G., 2011. Taking computer vision aloft - archaeological three-dimensional Wierzbicki, D., Kedzierski, M., Fryskowska, A., 2015. Assesment of the influence of UAV
reconstructions from aerial photographs with photoscan. Archaeol. Prospect. http:// image quality on the orthophoto production, in: International archives of the pho-
dx.doi.org/10.1002/arp.399. togrammetry. Remote Sensing and Spatial Information Sciences - ISPRS Archives. pp.
Westoby, M.J., Brasington, J., Glasser, N.F., Hambrey, M.J., Reynolds, J.M., 2012. 1–8. http://dx.doi.org/10.5194/isprsarchives-XL-1-W4-1-2015.
“Structure-from-motion” photogrammetry: a low-cost, effective tool for geoscience

10