Science of the Total Environment 589 (2017) 250–267

Contents lists available at ScienceDirect

Science of the Total Environment

journal homepage: www.elsevier.com/locate/scitotenv

Mapping landslide susceptibility using data-driven methods

J.L. Zêzere ⁎, S. Pereira, R. Melo, S.C. Oliveira, R.A.C. Garcia
Institute of Geography and Spatial Planning, Universidade de Lisboa, Portugal

H I G H L I G H T S G R A P H I C A L A B S T R A C T

• Multivariate statistical methods per-

form better when computed over het-
erogeneous terrain mapping units.
• Landslide susceptibility should be
assessed over grid cells when the spa-
tial accuracy of landslide inventory is
good.
• A single point per landslide proved to
be efficient to generate accurate land-
slide susceptibility maps
• The model with the highest AUC ROC is
not necessarily the best landslide sus-
ceptibility model.

a r t i c l e i n f o a b s t r a c t

Article history: Most epistemic uncertainty within data-driven landslide susceptibility assessment results from errors in land-
Received 7 November 2016 slide inventories, difficulty in identifying and mapping landslide causes and decisions related with the modelling
Received in revised form 20 February 2017 procedure. In this work we evaluate and discuss differences observed on landslide susceptibility maps resulting
Accepted 22 February 2017
from: (i) the selection of the statistical method; (ii) the selection of the terrain mapping unit; and (iii) the selec-
Available online 3 March 2017
tion of the feature type to represent landslides in the model (polygon versus point). The work is performed in a
Editor: D Barcelo single study area (Silveira Basin - 18.2 km2 - Lisbon Region, Portugal) using a unique database of geo-environ-
mental landslide predisposing factors and an inventory of 82 shallow translational slides.
Keywords: The logistic regression, the discriminant analysis and two versions of the information value were used and we
Landslide susceptibility conclude that multivariate statistical methods perform better when computed over heterogeneous terrain
Data-driven methods units and should be selected to assess landslide susceptibility based on slope terrain units, geo-hydrological ter-
Terrain mapping units rain units or census terrain units. However, evidence was found that the chosen terrain mapping unit can pro-
Validation duce greater differences on final susceptibility results than those resulting from the chosen statistical method
Uncertainty
for modelling.
The landslide susceptibility should be assessed over grid cell terrain units whenever the spatial accuracy of
landslide inventory is good. In addition, a single point per landslide proved to be efficient to generate accurate
landslide susceptibility maps, providing the landslides are of small size, thus minimizing the possible existence
of heterogeneities of predisposing factors within the landslide boundary.
Although during last years the ROC curves have been preferred to evaluate the susceptibility model's perfor-
mance, evidence was found that the model with the highest AUC ROC is not necessarily the best landslide suscep-
tibility model, namely when terrain mapping units are heterogeneous in size and reduced in number.
© 2017 Elsevier B.V. All rights reserved.

⁎ Corresponding author.
E-mail address: [email protected] (J.L. Zêzere).

http://dx.doi.org/10.1016/j.scitotenv.2017.02.188
0048-9697/© 2017 Elsevier B.V. All rights reserved.
 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267 251

1. Introduction angle, slope shape, lithology, land-cover) it is possible to quantitatively

predict the likelihood of future landslide occurrence even in non-land-
Landslides are geomorphic processes that generate a substantial slide-affected areas (Corominas et al., 2014). In this process, data from
death toll (Petley, 2012) and significant economic losses. The mitigation past and present landslides are used to evaluate the relative importance
of landslide consequences implies the likelihood of future landslide oc- of the predisposing factors and respective classes.
currence to be known, and landslide susceptibility has been mapped Data-driven landslide susceptibility models are sustained by three
worldwide since the pioneer work of Brabb et al. (1972) for the San major assumptions (Carrara et al., 1991; Guzzetti et al., 1999;
Mateo County, California. Also in 1972, the landslide susceptibility Hutchinson, 1995; Varnes, 1984): (i) landslides can be recognized, clas-
evaluation started in France through the ZERMOS plan (Chazan, sified and mapped; (ii) landslide predisposing factors can be identified,
1974). Presently, the assessment and mapping of the landslide suscep- registered and used to build landslide predictive models; and (iii) future
tibility is a critical step within the landslide risk analysis at the basin landslide occurrence can be spatially inferred. In other words, following
scale (e.g. Corominas et al., 2014; van Westen et al., 2008). the Uniformitarianism principle, the past and the present are consid-
The concept of landslide susceptibility expresses the likelihood of ered keys to the future, hence it is assumed that future landslides are
landslide occurrence in an area based on the local terrain conditions more likely to occur under the same geologic and geomorphologic con-
(landslide predisposing factors), not including the return period nor ditions that led to past slope instability (Guzzetti et al., 2005; Varnes,
the probability of occurrence of the instability process (Brabb et al., 1984).
1972; Corominas et al., 2014; Fell et al., 2008; Guzzetti et al., 1999; As a rule, any data-driven landslide susceptibility assessment (LSA)
Soeters and van Westen, 1996). is guided by the following methodological cascade: (i) the inventorying
Landslide susceptibility mapping can be achieved using qualitative of landslides; (ii) the identification and classification of (mappable)
methods (inventory-based and knowledge-driven methods) and quan- landslide predisposing factors; (iii) the weighting of classes whitin
titative methods (data-driven methods and physically-based models) each landslide predisposing factor using a bivariate or multivariate sta-
(Corominas et al., 2014; Fell et al., 2008). Data-driven methods statisti- tistical method; (iv) the construction of the landslide predictive model
cally evaluate the combination of the predisposing factors which are and the classification of susceptibility; and (v) the evaluation of predic-
better related to the spatial distribution of past and present landslides. tion model performance. In addition, in any study area the landslide sus-
Therefore, according to the existing terrain conditions (e.g. slope ceptibility must be assessed individualy for each type of landslide

Fig. 1. Location of the Silveira basin and distribution of landslides.

252 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267

because different types of landslides have different spatial incidence re- Table 1
lated to distinct thresholds conditions concerning landslide predispos- Landslide subgroups number and area.

ing factors (Zêzere, 2002). Landslide training Landslide test Total landslide
Landslide inventory maps are used as the dependent variable for group group inventory
data-driven landslide susceptibility methods and are also crucial to val- Number of landslides 41 41 82
idate any landslide susceptibility map, regardless the method used to Area (m2)
construct it (Corominas et al., 2014). Landslide inventory mapping can Minimum 127 68 68
Maximum 8566 2328 8566
be carried out using a variety of methods that were updated and sum-
Average 1249 852 1050
marized by Guzzetti et al. (2012). Landslide inventories can be ar- Standard deviation 1499 626 1166
chive-based, supported by documental sources (Carrara et al., 2003; Total 51,189 34,922 86,111
Galli et al., 2008; Pereira et al., 2014) or geomorphological-based
(Guzzetti et al., 2012), supported by identification of landslide features
using field work or image interpretation techniques. The latter category
can be divided into historical, event-based, seasonal and multi-temporal models do not take into account the interdependence of predictive var-
inventories (Fiorucci et al., 2011; Galli et al., 2008; Mondini et al., 2014). iables (Corominas et al., 2014; Pereira et al., 2012), and this is a major
Archive-based landslide inventories are usually represented as point drawback of the method. The multivariate statistical analysis evaluates
features, whereas geomorphological-based landslide inventories are the combined relationship between the dependent variable (landslide
commonly represented with polygon features used to outline the entire distribution) and a set of independent variables (landslide predisposing
landslide area (e.g. Zêzere et al., 2004a, 2004b) or only the landslide factors). The most popular methods to analyse the resulting matrix in-
rupture zone (e.g. Blahut et al., 2010; Pereira et al., 2012). For landslide clude the logistic regression (Chen and Wang, 2007; Felicísimo et al.,
susceptibility modelling purposes landslide inventories can be repre- 2013; Gorsevski et al., 2006; Guzzetti et al., 1999; Suzen and Doyuran,
sented as polygons or points, the latter corresponding to the centroid 2004), discriminant analysis (Guzzetti et al., 2006; Lee et al., 2008),
of the complete landslide area (e.g. Wang et al., 2014) or to the centroid and random forest (Catani et al., 2013; Goetz et al., 2015; Vorpahl et
of the landslide rupture zone (e.g. Sterlacchini et al., 2011). The choice al., 2012).
between point or polygons to represent landslides is usually guided by
the following conditions: (i) the landslide inventory characteristics,
namely the positional accuracy; (ii) the scale of the work; (iii) the objec-
tive of the landslide susceptibility map; and (iv) the limitations of soft- Table 2
ware and hardware (Oliveira et al., 2015b). According to Oliveira et al. Landslide predisposing factors and variable classes considered for landslide susceptibility
assessment.
(2015b) when landslides are large in size, landslide susceptibility as-
sessment using landslide areas instead of landslide points generate bet- ID Area (m2) Landslide Landslide
ter prediction results. Some authors have explored the influence of area (m2) density
particular positional errors of the landslide features (polygons and Slope (degrees)
points) and how it can propagate into the final landslide susceptibility 1 0–5 2,069400 275 0.00013
maps computed with data-driven methods (e.g. Ardizzone et al., 2 5–10 6,413,175 4700 0.00073
3 10–15 4,919,600 11,525 0.00234
2002; Steger et al., 2016). According to Ardizzone et al. (2002), despite 4 15–20 2,228,625 18,375 0.00825
the positional mismatches of the landslide area, results obtained with 5 20–25 1,200,675 19,700 0.01641
discriminant analysis are similar and minimize the impact of input 6 25–30 709,900 14,375 0.02025
data errors. Zêzere et al. (2009) showed that landslide susceptibility 7 N 30 608,175 17,225 0.02832
models built with logistic regression and using different landslide in- Aspect
ventories for a single study area generated similar predictive results 1 Flat 52,975 0 0
and landslide susceptibility maps. Recently, Santangelo et al. (2015) 2 N 1,457,850 27,700 0.01900
3 NE 2,381,225 23,400 0.00983
proposed a GIS-based semi-automatic procedure of aerial-photographs
4 E 2,877,800 7600 0.00264
interpretation in order to reduce landslide positional errors. 5 SE 2,697,625 3025 0.00112
The causes that can explain the spatial distribution of landslides 6 S 3,273,400 1425 0.00044
have been summarized in literature. Extensive explanation of landslide 7 SW 2,916,175 8925 0.00306
predisposing factors can be found in van Westen et al. (2008) and 8 W 1,586,300 9400 0.00593
9 NW 906,200 4700 0.00519
Corominas et al. (2014). The former discusses the suitability of geo-en-
vironmental predisposing factors according to scale (van Westen et al., Slope/area ratio
2008, page 120, table 4), whereas the latter lists the relevance of land- 1 0 192,925 25 0.00013
2 0–0.0001 680,800 500 0.00073
slide predisposing factors for different types of landslides (Corominas 3 0.0001–0.001 3,706,650 6300 0.00170
et al., 2014, pages 216–217, table 2). The selection of the landslide pre- 4 0.001–0.01 10,947,850 60,850 0.00556
disposing factors must take into account: (i) the scale of the analysis, (ii) 5 0.01–0.1 2,605,950 18,500 0.00710
the characteristics of the study area, (iii) the landslide type, and (iv) the 6 N 0.1 15,375 0 0
availability of data at the appropriate scale (Glade and Crozier, 2005; Lithology
van Westen et al., 2008). Ideally, the input data must have identical 1 Limestone and marl (Tithonian) 5,927,975 26,450 0.00446
quality and resolution. 2 Sandstone, marl and limestone (Upper 2,182,850 21,500 0.00985
Kimmeridgian – Tithonian)
Data-driven methods can be grouped in bivariate statistical analysis
3 Limestone (Upper Kimmeridgian) 3,608,800 13,725 0.00380
and multivariate statistical analysis. The bivariate statistical analysis 4 Marl (Upper Kimmeridgian) 1,101,650 3925 0.00356
compares independently each predisposing factor with the landslide 5 Marl (Lower-Middle Kimmeridgian) 5,328,275 20,575 0.00386
distribution. Weights of the landslide predisposing factors are assigned
Land use/land cover
based on landslide density using different methods such as favourability 1 Urban area and motorway 1,063,600 1475 0.00014
functions (Chung and Fabbri, 1993; Zêzere et al., 2004a, 2004b), infor- 2 Quarry 1,809,650 0 0
mation value (Guillard and Zêzere, 2012; Yin and Yan, 1988), and 3 Crop area 5,416,475 5000 0.00090
weights of evidence (Blahut et al., 2010; Lee and Choi, 2004; 4 Forest 1,153,000 14,150 0.01227
5 Open forest and shrub 8,706,825 65,550 0.00750
Sterlacchini et al., 2011; Suzen and Doyuran, 2004). Bivariate statistical
 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267 253

Comparison between methods used to assess landslide susceptibili- susceptibility maps statistically derived presented the most satisfying
ty at the basin scale has been extensively performed worldwide. Expert- results. Data-driven methods minimize the subjectivity in the predis-
based (qualitative methods) and bivariate statistical methods were posing factors weights assignment and produce more objective and re-
compared to assess landslide susceptibility for basins (Magliulo et al., producible results (Kanungo et al., 2009) in comparison with
2009) and regions over 1000 km2 (Den Eeckhaut et al., 2010). Landslide knowledge-data based methods. Comparison between landslide

Fig. 2. Landslide predisposing factors used to model landslide susceptibility; slope angle (a), slope aspect (b), slope over area ratio (c), lithology (d), and land use/land cover (e). See Table 2
for classes' description.
254 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267

susceptibility assessment using different bivariate and multivariate database of geo-environmental landslide predisposing factors. The dis-
data-driven methods has been made (Felicísimo et al., 2013; Goetz et cussion on the relevance of landslide predisposing factors within land-
al., 2015; Suzen and Doyuran, 2004) showing typically a better perfor- slide susceptibility models is beyond the scope of this work.
mance of multivariate methods. Recent comparisons made between
‘traditional’ statistical and novel machine learning models demonstrate 2. Data and methods
a little differentiation in the prediction performance between these
landslide susceptibility modelling techniques (Goetz et al., 2015). 2.1. Study area
Data-driven methods have been compared with physically-based
methods, typically for shallow landslides and adopting the infinite The study area selected for this study is the Silveira basin (18.2 km2)
slope model (Carrara et al., 2008; Cervi et al., 2010; Yilmaz, 2009). As located north of Lisbon, Portugal (Fig. 1). The elevation ranges from 27
a rule, the physically-based methods generated slightly better suscepti- to 377 m and the lithology is dominated by sedimentary rock (alternat-
bility results, although they are more demanding concerning geotechni- ing marl, limestone and sandstone) of upper Jurassic age. The geological
cal and hydrological input data. structure is characterized by the monocline dipping of the layers to-
Selection of the suitable terrain mapping unit is important to evalu- wards south and southeast, with dip ranging from 8 to 12 degrees
ate the likelihood of a limited portion of the land surface to contain a set (Zêzere, 2002).
of conditions that differ from the surrounding units, ensuring internal
homogeneity and heterogeneity between terrain units (Guzzetti et al.,
1999; Guzzetti, 2005). Data-driven methods for landslide susceptibility 2.2. The landslide inventory
assessment generally use the following types of terrain mapping units:
(i) grid cells; (ii) slope units; (iii) geo-hydrological units; and (iv) ad- Landslides were inventoried in the study area based on interpreta-
ministrative units. The choice of the most adjusted terrain mapping tion of aerial photos and orthophotomaps and extensive field work. In
unit for modelling depends on the scale of the work, the quality and res- this work, we only consider shallow translational slides that are thin
olution of the available thematic (mapping) information, and the type slope movements occurring over planar slip surfaces, which depth is
and size of landslides (Guzzetti et al., 1999; Guzzetti, 2005). typically b 1.5 m. At total, 82 shallow translational slides were identified
The type and size of the terrain mapping unit may exert a significant (Fig. 1), which corresponds to a density of 4.5 landslides per km2,
influence on the final results of the susceptibility assessment and this whereas the ratio of unstable area over the total study area is 0.0047.
topic has been less addressed in literature. Guzzetti et al. (2000) com- These landslides are typically of small size, (minimum area = 68 m2;
pared grid-based (25 m2) and slope-unit-based landslide density maximum area = 8566 m2; average area = 1050 m2) (Table 1). For
maps. Comparison of landslide susceptibility models based on grid modelling purposes, the landslide inventory was randomly divided in
cells and slope units was made using discriminant analysis (Baeza et two subsets with equal number (41 cases) of landslides (Fig. 1): the
al., 2010; Carrara et al., 2008; Van Den Eeckhaut et al., 2009) and logistic landslide training group and the landslide test group. The landslide
regression (Erener and Düzgün, 2012). In general authors concluded training group contains 59% of the total landslide affected area, which
that landslide susceptibility models performance increased when reflects the general larger size of landslides belonging to this group, in
slope units were used. comparison with landslides included in the test group (Table 1).
The major epistemic uncertainty within data-driven landslide sus-
ceptibility assessment results from: (i) errors in landslide inventories; 2.3. The landslide predisposing factors
(ii) difficulty in identifying and mapping landslide causes; and (iii) de-
cisions related with the modelling procedure. Uncertainty related with As the aim of this work is to evaluate the impact of statistical
landslide inventories, predisposing factors and analytical tools have methods, terrain mapping units and landslide representation features
been addressed in literature but little attention has been given to the ef- on data-driven landslide susceptibility maps, we decided to not consid-
fects on landslide susceptibility maps resulting from the selection of the er a large number of landslide predisposing factors, in order to constrain
terrain mapping unit and of the feature type to represent landslides. The the amount of possible combinations resulting from the factors overlay.
main objective of this study is to evaluate and discuss differences ob- Additionally, the set of landslide geo-environmental predisposing fac-
served on landslide susceptibility maps resulting from three sources of tors and the corresponding classes were forced to be exactly the same
uncertainty associated to the modelling procedure within data-driven in all models to assure that results are not biased by this source of uncer-
models: (i) the selection of the statistical method; (ii) the selection of tainty, which is not addressed in this work. Five landslide predisposing
the terrain mapping unit; and (iii) the selection of the feature type to factors were considered: slope angle, slope aspect, slope over area ratio,
represent landslides in the model (polygon versus point). The work is lithology and land use/land cover. These factors have been used to as-
performed in a single study area (Silveira Basin, Lisbon Region, Portu- sess landslide susceptibility with good results in areas similar to the
gal) using an inventory of 82 shallow translational slides and a unique test site (Guillard and Zêzere, 2012; Zêzere et al., 2004a, 2004b).

Table 3
Characteristics of landslide susceptibility models performed for the study area.

Landslide susceptibility model Statistical method Terrain unit Landslide inventory Landslide feature

Model A Logistic regression Slope terrain unit All landslides Polygon

Model B Discriminant analysis Slope terrain unit All landslides Polygon
Model C Information value Slope terrain unit All landslides Polygon
Model D Information value (positive scores) Slope terrain unit All landslides Polygon
Model E Logistic regression Geo-hydrological terrain unit All landslides Polygon
Model F Logistic regression Census terrain unit All landslides Polygon
Model G Logistic regression Grid cell terrain unit (5 m) All landslides Polygon
Model H Logistic regression Slope terrain unit Training group Polygon
Model I Logistic regression Geo-hydrological terrain unit Training group Polygon
Model J Logistic regression Census terrain unit Training group Polygon
Model K Logistic regression Grid cell terrain unit Training group Polygon
Model L Logistic regression Grid cell terrain unit Training group Point (landslide centroid)
Model M Logistic regression Grid cell terrain unit Training group Point (centroid of landslide rupture zone)
 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267 255

Table 4
General characteristics of considered terrain units.

Terrain units # Average area (m2) Minimum area (m2) Maximum area (m2) STD # unstable units % unstable units

Slope TU 275 6,6003 3072 318,548 58,974 54 19.6

Geo-hydrological TU 684 26,536 112 296,889 33,648 90 13.2
Census TU 98 187,120 1616 2,625,674 387,102 23 23.5
Grid cell TU 725,982 25 25 25 0 3447 0.5

The slope angle is a very good proxy of the shear stress that pro- directly the state of stability of slopes. Finally, the use of the land use/
motes instability on slopes. Theoretically, the propensity to landslide oc- land cover as a landslide predisposing factor is twofold. On the one
currence tends to increase with the increase of slope angle. The slope hand, land use reflects anthropogenic actions that can increase landslide
aspect is a good proxy of the morpho-structural setting within the activity (e.g. van Westen et al., 2008). On the other hand, land cover in-
study area, as the geologic structure is regularly monocline, the layers fluences landslide occurrence, namely shallow movements that are less
dipping south and southeast. Therefore, slopes facing south and south- frequent along slopes with dense and deeply rooted vegetation
east follow the general dip of the layers, whereas slopes facing north (Guillard and Zêzere, 2012).
and northwest are anti-dip slopes, which correspond typically to a A digital elevation model (DEM) was built based on a digital topo-
higher slope angle. The slope over the specific catchment area ratio is graphic map at the 1:10,000 scale containing contour lines equidistant
a proxy variable that reflects the moisture retention, the soil water con- each 5 m. A pixel size of 5 m (25 m2) was adopted to the DEM layer
tent and the surface saturation zones. The lithology is a proxy of the that was the source to derive three maps used as landslide predisposing
shear strength of the materials constituting slopes, thus controlling factors: the slope angle, the slope aspect, and the slope over area ratio.

Fig. 3. Landslide susceptibility maps based on slope terrain units (STU) considering the complete landslide inventory and using logistic regression – Model A (a), discriminant analysis –
Model B (b), information value – Model C (c), and information value (positive scores) – Model D (d).
256 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267

The lithology was obtained from the official geological map of the region constraint. In addition, with this option we guarantee a more straight-
(1:25,000 scale, provided by the Portuguese Geological Survey), where- forward comparison among statistical models, as the model input is ex-
as the lithological units boundaries were checked and rectified during actly the same.
field work. The land use/land cover information was reclassified starting In total, 13 landslide susceptibility maps were constructed and clas-
from the official map in vector format that was generated through the sified in 10 classes containing equal number of terrain units to facilitate
interpretation of digital ortho-rectified aerial photos obtained in 2007. the visual comparison. The evaluation of the prediction skills of suscep-
The minimum cartographic unit of this map is 1 ha. tibility models was made using receiver operating characteristics (ROC)
The complete landslide predisposing factors were represented in curves and computing the following ROC metrics (Bradley, 1997;
raster with a 5 m (25 m2) pixel size, which is coherent with the detail Fawcett, 2006; Tien Bui et al., 2016a): positive predictive value (proba-
of the cartographic sources. Table 2 summarizes the variable classes of bility of terrain units that are correctly classified as containing land-
each predisposing factor considered for landslide susceptibility assess- slides); negative predictive value (probability of terrain units that are
ment in the study area. At total, 38 categorical classes were defined. correctly classified as not containing landslides); sensitivity (% of terrain
The spatial distribution of categorical classes within each landslide pre- units containing landslides that are correctly classified); specificity (% of
disposing factor is represented in Fig. 2. terrain units not containing landslides that are correctly classified); ac-
curacy (proportion of terrain units containing and not containing land-
2.4. The modelling strategy slides that are correctly classified); Matthews correlation coefficient
(MCC; Matthews, 1975), and the ROC AUC (Area Under the Curve).
In this work we aimed to construct and critically evaluate several MCC ranges from −1 to +1, where −1 (+1) indicates complete dis-
landslide susceptibility maps built with data-driven methods and agreement (agreement) between predicted and observed, and 0 indi-
using different statistical methods, different terrain mapping units and cates prediction equivalent to random. The ROC AUC evaluates the
different features to represent landslides within the predictive models. overall performance of the landslide susceptibility models (e.g.
The adopted modelling strategy includes 4 steps. Gorsevski et al., 2006; Pereira et al., 2012). As a rule, the closer the
In the step one we evaluate the performance of four statistical ROC AUC value to 1, the better is the landslide model performance.
methods that were recently recommended for landslide susceptibility With the single exception of the ROC AUC, the abovementioned met-
analysis (Corominas et al., 2014): logistic regression, discriminant anal- rics to measure models prediction skills are valid only for one particular
ysis, information value and information value (positive scores). These operation point within the ROC space (Bradley, 1997). In this study, the
methods are described in Section 2.4.1. Therefore, four landslide suscep- optimal operation point of each model was defined by the Youden index
tibility models were computed (Models A, B, C and D, Table 3) using the (J) (Perkins and Schisterman, 2005):
same terrain mapping unit (slope terrain units) and the complete land-
slide inventory represented with polygon features. J ¼ Maximum fsensitivity ðcÞ þ specificity ðcÞ−1g ð1Þ
In the step two we selected the statistical method that showed the
best performance in step one (logistic regression) in order to evaluate Within the ROC space, this point maximizes the accuracy as well as
the performance of four different terrain mapping units: slope terrain the vertical distance between the ROC curve and the diagonal of no dis-
units (STU), geo-hydrological terrain units (GHTU), census terrain crimination (Jiménez-Valverde, 2012; Tien Bui et al., 2016b).
units (CTU) and grid cell terrain units (GCTU). These terrain units are Any landslide spatial prediction can present errors of omission (false
described in Section 2.4.2. Four landslide susceptibility models were negatives, i.e. landslide terrain units that were not correctly predicted)
computed (Models A, E, F and G, Table 3) using the same statistical and errors of commission (false positives, i.e. no landslide terrain units
method (logistic regression) and the complete landslide inventory and that were not correctly predicted). The cost of both types of errors is
considering the total landslide area. not the same within prediction models: omission errors are of much
The modelling strategy used in steps one and two does not allow the more concern than commission errors, which is not considered by the
evaluation of the predictive capacity of susceptibility models because all
landslides were used to weight predictive variables. Therefore, in step
three the modelling procedure in step two was repeated using the land-
slide training group (Models H, I, J and K, Table 3). In these cases, valida-
tion was performed using independently the landslide training group
and the landslide test group.
In the step four we evaluate the influence of the feature type used to
represent landslides. Two additional landslide susceptibility models
were computed using the logistic regression method over GCTU, and
using the landslide training group represented as point features corre-
sponding to the centroid of landslide and the centroid of landslide rup-
ture zone (respectively, Models L and M, Table 3).
By default, the complete non-landslide terrain units are used to
weight landslide predisposing factor's classes in any bivariate statistical
method (e.g. information value and information value - positive scores).
In this work we decided to consider the complete non-landslide terrain
units in the complete set of landslide susceptibility models, even in
models built with logistic regression and discriminant analysis. Howev-
er, as a rule, the ratio between presence/absence of landslides is very
low. Therefore, the obtained results with logistic regression and dis-
criminant analysis are typically low and cannot be interpreted as prob-
abilities, because we did not perform any sampling of terrain units with
landslide absence for modelling. The aim of our work was not to obtain
probabilities of future landslide occurrence, but only to rank the terrain
units in terms of landslide susceptibility, which means that the issue of Fig. 4. ROC curves of landslide susceptibility models based on slope terrain units (STU) and
sample proportionality (landslide presence and absence) was not a considering the complete landslide inventory.
 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267 257

Table 5 the success/prediction-rate curves is interpreted with the same rules

Prediction skills of landslide susceptibility models A, B, C and D. as for the ROC AUC.
ROC metric Model A Model B Model C Model D Finally, similarities of the spatial patterns in four classes of the land-
Independent validation No No No No slide susceptibility maps computed with the logistic regression over
True positive 31 32 29 38 grid cells were evaluated with the Kappa statistics, using the Map Com-
True negative 211 203 185 133 parison Kit (Visser and De Nijs, 2006). The Kappa statistic (Cohen, 1960)
False positive 10 18 36 88 is a measure of similarity between a pair of maps that is based on the
False negative 23 22 25 16
percentage of agreement corrected for the fraction of agreement that
Positive predictive value (%) 75.61 64.00 44.62 30.16
Negative predictive value (%) 90.17 90.22 88.10 89.26 can result by chance (Visser and De Nijs, 2006). Three Kappa statistics
Sensitivity (%) 57.41 59.26 53.70 70.37 were applied: Kappa location, Kappa histogram and Kappa index.
Specificity (%) 95.48 91.86 83.71 60.18 Kappa location measures similarity of location regarding the spatial dis-
Accuracy (%) 88.00 85.45 77.82 62.18 tribution of categories on the maps, whereas Kappa histogram measures
Matthews correl. coef. (MCC) 0.590 0.526 0.350 0.244
ROC AUC 0.858 0.846 0.712 0.669
similarity of frequency of categories (i.e. susceptibility classes) in both
maps. The Kappa index is computed as the product between Kappa lo-
cation and Kappa histogram (Visser and De Nijs, 2006).
ROC AUC metrics (Jiménez-Valverde, 2012; Lobo et al., 2008; Peterson The values of Kappa statistics range between 0 and 1. Value of 0 indi-
et al., 2008). As a consequence, the assessment of performance of land- cates no spatial agreement between maps, whereas value of 1 indicates
slide susceptibility models based only on the ROC curve may be inade- that maps are identical. Landis and Koch (1977) proposed the following
quate (Tien Bui et al., 2016a). classes of spatial agreement between two maps according to Kappa
For comparison with the ROC, the success/prediction-rate curve index that were used in this study: slight (0–0.2), fair (0.2–0.4), moderate
method (Chung and Fabbri, 2003) was also used in this study. The suc- (0.4–0.6), substantial (0.6–0.8), and almost perfect (0.8–1.0). The signifi-
cess (prediction) rate curve is obtained by plotting the cumulative per- cance, the asymptotic standard error and the confidence intervals of
centage of the training (test) landslide terrain units against the Kappa scores were measured using the IBM SPSS Statistics 24 software.
cumulative percentage of the study area classified as landslide suscepti-
ble in descending order. The success/prediction-rate graph is similar to a 2.4.1. The statistical methods
traditional ROC graph, except that the x-axis is measured as proportion The logistic regression (Cox, 1958) is a multivariate statistic tech-
of the study area predicted as ‘present’ instead of being measured as nique that has been widely used to assess landslide susceptibility (e.g.
success in predicting ‘absence’ (Peterson et al., 2008). The area under Chen and Wang, 2007; Gorsevski et al., 2006; Heckmann et al., 2014;

Fig. 5. Overlapping between four landslide susceptibility models built over slope terrain units (STU) considering the 30% more susceptible terrain units. The logistic regression model
(Model A) is used as reference.
258 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267

Jaiswal et al., 2010; Suzen and Doyuran, 2004; Van Den Eeckhaut et al., The logistic regression was computed with the Statistica 7 software
2006). Logistic regression is used to determine the likelihood of future for STU, GHTU and CTU terrain mapping units, and the results were fur-
landslide occurrence based on a dependent binary variable (presence ther transferred to the ArcGIS 10.4 software. In the case of GCTU, the lo-
or absence of landslide), whereas the explanatory (independent) vari- gistic regression was computed with the Arc SDM toolbox (Sawatzky et
ables can be categorical, numerical or both (Hosmer et al., 2013; Van al., 2009).
Den Eeckhaut et al., 2006). Logistic regression fits a special s-shaped The discriminant analysis (Fisher, 1936) is another multivariate sta-
curve by taking a linear regression, that may produce any y-value be- tistic technique that has been used for landslide susceptibility assess-
tween −∞ and +∞, and transforming it with the function that produces ment (e.g. Baeza et al., 2010; Guzzetti et al., 2005; Lee et al., 2008;
a probability (p-probability) between 0 (as y approaches minus infinity) Pardeshi et al., 2015; Van Den Eeckhaut et al., 2009). Discriminant anal-
and 1 (as y approaches plus infinity) (Gorsevski et al., 2006). ysis classifies terrain units into one of two mutually exclusive groups
The logistic regression method has the form (e.g. Budimir et al., (mapping units free of landslides, and mapping units having landslides)
2015): based on their values for a set of predictor variables (Guzzetti et al.,
2006). The assumption made is that the two groups are distinct and nor-
logit ðyÞ ¼ β0 þ β1 x1 þ β2 x2 þ ⋯ þ βi xi þ e ð2Þ mally distributed. The scope of the analysis is to determine the group
membership of a mapping unit by finding a linear combination of the
where y is the dependent variable, xi is the i-th is the explanatory vari- environmental (independent) variables which maximizes the differ-
able, β0 is a constant, β1 is the i-th regression coefficient and e is the ences between the populations of stable and unstable terrain units
error. The probability (p) of occurrence of y is: (Guzzetti et al., 2006; Lee et al., 2008; Van Den Eeckhaut et al., 2009).
The discriminant analysis model has the form of a linear combina-
tion of the input variables:
expðβ0þβ1x1þβ2x2þ…þβixiÞ
p¼ ð3Þ
1 þ expðβ0þβ1x1þβ2x2þ…þβixiÞ Z ¼ β1ν1 þ β2ν2 þ … þ βmνm ð4Þ

Fig. 6. Landslide susceptibility maps obtained with logistic regression and considering the complete landslide inventory, based on slope terrain units (STU, Model A) (a), geo-hydrological
terrain units (GHTU, Model E) (b), census terrain units (CTU, Model F) (c), and grid cell (5 m) terrain units (GCTU, Model G) (d).
 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267 259

where ν1, ν2, …, νm are the explanatory variables, and β1, β2, …, βm Table 6
are standardized coefficients that aim to maximize differences between Prediction skills of landslide susceptibility Models A, E, F and G.

groups and to minimize differences within each group. ROC metric Model A Model E Model F Model G
In this work discriminant analysis was computed with the Statistica Independent validation No No No No
7 software for STU terrain mapping units and the results were further True positive 31 68 20 3007
exported to the ArcGIS 10.4 software. True negative 211 462 73 546,534
The information value method (Yan, 1988; Yin and Yan, 1988) is a False positive 10 132 1 176,001
False negative 23 22 3 440
bivariate statistical method that has been used with good results for
Positive predictive value (%) 75.61 34.00 95.24 1.68
landslide susceptibility assessment (e.g. Conforti et al., 2011; Guillard Negative predictive value (%) 90.17 95.45 96.05 99.92
and Zêzere, 2012; Oliveira et al., 2015a, 2015b). The information value Sensitivity (%) 57.41 75.56 86.96 87.24
of each class within each explanatory variable is given by Eq. (4) (Yan, Specificity (%) 95.48 77.78 98.65 75.64
1988; Yin and Yan, 1988): Accuracy (%) 88.00 77.49 95.88 75.70
Matthews correl. coef. (MCC) 0.590 0.396 0.884 0.100
ROC AUC 0.858 0.816 0.947 0.883
Si=Ni
IVXi ¼ ln ð5Þ
S=N
physical meaning of terrain units boundaries, CTU allows for the easy
where IVXi is the information value of variable Xi; Si is the number of comparison between landslide susceptibility maps and census data on
terrain units with landslides and the presence of variable Xi; Ni is the exposed elements (e.g. population and the building environment),
number of terrain units with variable Xi; S is the total number of terrain which is interesting for politicians and decision makers (Guzzetti and
units with landslides; and N is the total number of terrain units. Tonelli, 2004). Grid cell terrain units divide the territory into regular
The relevance of any independent variable to discriminate stable and areas with a pre-defined size and shape (usually squares) for which a
unstable areas is as greater as its distance from the 0 value of IV. When value is assigned for each landslide predisposing factor. Grid cells are
the score is negative it means that the presence of the variable Xi is preferred by raster-based GIS users and have been used worldwide to
favourable to slope stability. Positive scores mean a positive relationship model landslide susceptibility (e.g. Baeza et al., 2010; Erener and
between the presence of the variable and the landslide occurrence, as Düzgün, 2012; Günther et al., 2013; Pereira et al., 2016; Van Den
high as the higher the score. Information values equal to zero means Eeckhaut et al., 2009; Zêzere et al., 2004a, 2004b).
no clear relationship between the variable and the landslide occurrence. In this study STU were identified manually using the official digital
The information value was applied using STU terrain mapping units. topographic map (1:10,000 scale, provided by the Portuguese Director-
The Eq. (5) was used to weight any class of explanatory variables by tab- ate-General for Territory), including the drainage network and accurate
ulating each predisposing factor with the complete landslide inventory. elevation data (contour lines with 5 m interval and elevation points).
Then, for each terrain unit a single score was computed for each predis- Channels were considered until the 1st order (Strahler) (minimum
posing factor, expressing the weighted average IV, which accounts the channel length = 57 m) and the original map was zoomed until the
area covered by each class within the terrain unit. The final information (maximum) 1:2000 scale to draw divide boundaries. At total, 275 STU
value of each STU is given by the Sum of the corresponding weighted av- were delineated ranging in size from 3072 m2 to 318,548 m2 (average
erage IV of each landslide predisposing factor. area = 66.003 m2) (Table 4). The number of unstable terrain units is
The results of information value over STU may be biased due to the 54 (19.6% of total STU), considering the complete landslide inventory.
existence of IV with different signal (positive, negative) for the same GHTU were delimited by crossing the previous mentioned STU with
predisposing factor within the same terrain unit. In order to eliminate the lithological units described in Section 2.3 that were drawn at the
this possible bias, the last landslide susceptibility model used again 1:25,000 scale. The GHTU map (not showed) includes 684 terrain
the information value rules to weight variable classes, but only the pos- units, which dimension ranges from 112 m2 to 296,889 m2 (average
itive IV scores were selected to compute the final information value of
each STU.

2.4.2. The terrain mapping units

Landslide susceptibility maps were made along this work over the
most usual terrain mapping units referred in literature: STU, GHTU,
CTU and GCTU. Slope terrain units correspond to the partition of the ter-
ritory into hydrological regions bounded by drainage and divide lines
(Carrara et al., 2008; Galli et al., 2008; Guzzetti et al., 1999). STU have
been extensively used in landslide susceptibility assessment, namely
in Italy (e.g. Alvioli et al., 2015; Galli et al., 2008; Guzzetti et al., 2006).
Geo-hydrological terrain units are delimited by further partitioning
the STU based on the main lithological types of the study area that dif-
ferentiate susceptibility conditions within the same slope (Ardizzone
et al., 2002; Guzzetti et al., 2005). Therefore, each GHTU have a unique
lithologic condition in the same terrain unit. Example of GHTU can be
found in the work of Ardizzone et al. (2002). Census terrain units, as
any administrative terrain unit (e.g. parish, municipality), do not reflect
any morphological, hydrological or lithological boundary, which is a
major limitation to be used to assess landslide susceptibility. However,
administrative terrain units have been used in literature (e.g. Garcia et
al., 2016; Günther et al., 2013), namely when landslide inventories pro-
duced by official entities (e.g. the civil protection) only provide the
name or the centroid of the administrative unit where landslides are lo-
Fig. 7. ROC Curves of landslide susceptibility models obtained with logistic regression and
cated. In such case, the use of administrative terrain units is the single the complete landslide inventory, using four different types of terrain units (STU, GHTU,
option to perform a landside susceptibility model. Despite the lack of CTU and GCTU).
260 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267

area = 26.536 m2) (Table 4). In comparison with STU, the number of regression (a), discriminant analysis (b), information value (c) and in-
unstable GHTU is higher (90), although the percentage of unstable ter- formation value (positive scores) (d). The same terrain mapping units
rain units decreases (13.2%). The CTU were directly provided by the Na- (STU) and the complete polygons representing all inventoried land-
tional Institute of Statistics. At total, 98 CTU are present in the study area slides were used. The landslide susceptibility maps include 10 suscepti-
with an average size of 187,120 m2 (Table 4). These terrain units have a bility classes containing equal number of terrain units to facilitate maps
very large variation in size (from 1616 m2 to 2,925,674 m2) which is comparison. Although, the four landslide susceptibility maps look simi-
confirmed by the very high standard deviation (STD). The number of lar, the multivariate statistical methods proved to be more efficient to
unstable terrain units is 23 (23.5% of total CTU). Finally, the GCTU corre- assess susceptibility, as it is showed by the ROC curves of predictive
spond to a 5 × 5 m grid, which is in accordance with the DEM resolution. models (Fig. 4, Table 5). The logistic regression (Model A) is the best
The number of GCTU is much higher compared to the other three model, which is confirmed by the highest values for the positive predic-
methods (725,982) as well as the number of unstable units (3447), al- tive value (75.61%), specificity (95.48%), accuracy (88.00%), the MCC
though the percentage of unstable units is much lower (0.5%) when (0.590) and the AUC ROC (0.858). The discriminant analysis (Model B)
compared with the other terrain mapping units (Table 4). is the second best model according to the ROC metrics and registered
STU, GHTU and CTU were cross tabulated with the classified land- the highest negative predictive value (90.22%). In comparison, the ca-
slide predisposing factors described in Section 2.3 to obtain the final ma- pacity of both approaches associated with the information value meth-
trices used to assess landslide susceptibility. od (Model C and Model D) to correctly predict landslide occurrence in
space is lower, which is confirmed by the lower values of the ROC met-
3. Results rics. For example, the AUC ROC (0.712 and 0.669, respectively for the in-
formation value model and the information value (positive scores)
The first modelling approach aimed to assess differences on final model) is below the AUC ROC of Models A and B from 0.134 to 0.189.
landslide susceptibility results produced by distinct statistical methods Despite the abovementioned differences, the spatial agreement be-
(cf. Section 2.4, step one). Fig. 3 shows the results obtained with logistic tween models is evident when the attention is focused over the areas

Fig. 8. Landslide susceptibility maps obtained with logistic regression and considering the landslide training group, based on STU (Model H) (a), GHTU (Model I) (b). CTU (Model J) (c) and
GCTU (Model K) (d).
 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267 261

classified as more susceptible. Fig. 5 shows the overlapping between obtained maps are visually very similar, and are also comparable with
models considering only the 30% terrain units classified as more suscep- the map corresponding to Model K that was built with the total area
tible to landslide occurrence, and using the logistic regression model as of the landside training group, using the same statistical method and
reference. The overlap between the four models extends to 44.4% of the the same terrain units. The ROC curves of Models L and M are shown
considered terrain units, whereas only 3.7% of terrain units are classified in Fig. 11, which also depict ROC curves of Model K, for comparison.
by just one model. 51.9% of terrain units are classified by three or two The ROC metrics are summarized in Table 8. In all cases, the validation
models, attesting a meaningful agreement between statistical methods. of models was made considering the total landslide areas (and not
As the logistic regression produced the best results to assess land- only the landslide centroids). It is noteworthy that, unlike Model K,
slide susceptibility in the first step of the modelling, this method was the validation results of Models L and M are very similar for both the
chosen to generate additional landslide susceptibility models using dif- landslide training group and the landslide test group. The positive pre-
ferent terrain mapping units and the complete polygons representing diction value and sensitivity are higher for landslide training group,
all inventoried landslides (cf. Section 2.4, step two). Fig. 6 shows the whereas specificity and accuracy are higher for landslide test group.
landslide susceptibility maps obtained with STU (a), GHTU (b), CTU The MCC is typically very low reflecting the strong contrast between
(c), and GCTU (d). As in the step one of modelling, the landslide suscep- positive and negative occurrences. The AUC ROC is virtual the same
tibility was classified in 10 classes containing equal number of terrain for training and test groups (Table 8), which indicates a remarkable sta-
units. bility of the landslide susceptibility models built with points, regardless
The number and shape of considered terrain mapping units are very their position in the centroid of the landslides or the landslide rupture
diverse, which strongly influence the delineation of landslide suscepti- zone.
bility classes and limit the comparisons that can be made between The results of the Kappa statistics computed for the landslide suscep-
models. This is the reason why we did not perform any evaluation of tibility maps obtained with logistic regression over grid cells (Models G,
the spatial agreement between landslide susceptibility maps shown in K, L and M) are summarized in Table 9. These maps were reclassified in
Fig. 6. Nevertheless, the prediction skills of the four landslide suscepti- 4 classes following the hierarchy of landslide susceptibility (very high,
bility maps are comparable by the ROC metrics (Table 6, Fig. 7). The high, moderate, low and very low), where very high class corresponds
quality of prediction models is generally high, which is demonstrated, to the top 10% (90–100%) most susceptible area, the high and moderate
for example, by the accuracy (ranging from 75.7% to 95.9%) and the classes include the area in the range 80–90% and 70–80%, respectively,
AUC ROC (ranging from 0.82 to 0.95). and the low and very low class includes the 70% less susceptible zones
The model built with CTU (Model F) is almost perfect, which is con- of the study area (0–70%). The number and boundaries of landslide sus-
firmed by the highest values for the positive predictive value (95.24%), ceptibility classes were defined to maximize the Kappa histogram (i.e.
sensitivity (86.96%), specificity (98.65%), accuracy (95.88%), the MCC equivalent landslide susceptibility classes should be similar in size for
(0.884) and the AUC ROC (0.947). The model built with GCTU (Model the pair of maps). With this option, Kappa index approximates Kappa
G) is the second best model according to the AUC ROC (0.883) and reg- location, which measures the spatial agreement between equivalent
ister the highest negative predictive value (99.92%), which is strongly landslide susceptibility classes in two maps.
influenced by the very large number of terrain units. Due to the same The asymptotic standard error is of 0.001 in all measures of agree-
reason, the MCC of Model G is much lower compared to the other ment, which means that the 95% confidence intervals of Kappa values
three models. Lastly, the performance of the model built with GHTU reported in Table 9 vary in the range − 0.002 to + 0.002. In addition,
(Model E) is generally lower than Model A, as attested by the corre- p b 0.0005 was obtained in all measures of agreement, which means
sponding specificity, accuracy, the MCC and the AUC ROC. that Kappa scores are statistically significantly different from zero. For
Landslide susceptibility models A, F, G and H were constructed with the complete maps, the agreement is substantial (0.6 to 0.8) between
the complete landslide inventory, which prevents their independent
validation. To overcome this limitation, the logistic regression was ap-
plied again over STU, GHTU, CTU and GCTU terrain units, but using the
landslide training group as dependent variable (cf. Section 2.4, step
three). The obtained landslide susceptibility maps (Fig. 8) are virtually
similar with the equivalent maps produced with the complete landslide
inventory. However, the model's prediction skills differ to some extent
(Fig. 9, Table 7). Although the AUC ROC of the landslide subgroup used
to weight variable classes (landslide training group) increase from
0.01 to 0.08 in comparison with the corresponding models produced
with total landslides, the accuracy and the MCC are lower for models
built over GHTU (Model I) and CTU (Model H).
As it was expected, the ROC metrics decrease substantially in any
model when the landslide test group is considered (Fig. 9, Table 7), as
this implies the independent validation of the predictive models. The
AUC ROC decreases from 0.07 to 0.19, in comparison with features ob-
tained with the landside testing group, being highest for the STU
model. The decrease is also apparent for the other ROC metrics, with
the single exception of model built with GCTU (Model K) that registers
a slight increase in specificity and accuracy (Table 7).
The last modelling step aimed to evaluate the influence of the land-
slide feature type on the susceptibility results (cf. Section 2.4, step four).
Fig. 10 shows the landslide susceptibility maps computed with logistic
regression over GCTU terrain units and using point features to represent
the dependent variable (landslide training group). The centroid of each
Fig. 9. ROC Curves of landslide susceptibility models obtained with logistic regression and
landslide area (Model L) and the centroid of each landslide rupture zone
the landslide training group, using four different types of terrain units (STU, GHTU, CTU
(Model M) are represented with a point feature. As in the previous and GCTU). The validation is made with the landslide training group (solid line) and the
cases, the maps were classified with 10 classes of equal size. The landslide test group (dashed line).
262 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267

Table 7
Prediction skills of landslide susceptibility Models H, I, J and K.

Model H Model I Model J Model K

ROC metric Training Test Training Test Training Test Training Test
Independent validation No Yes No Yes No Yes No Yes

True positive 29 22 49 40 18 15 1871 1003

True negative 227 213 471 466 69 69 558,787 575,267
False positive 12 25 155 164 9 9 165,141 149,322
False negative 7 15 9 14 1 4 183 390
Positive predictive value (%) 70.73 46.81 24.02 19.61 66.67 62.50 1.12 0.67
Negative predictive value (%) 97.01 93.42 98.13 97.08 98.57 94.52 99.97 99.93
Sensitivity (%) 80.56 59.46 84.48 74.07 94.74 78.95 91.09 72.00
Specificity (%) 94.98 89.50 75.24 73.97 88.46 88.46 77.19 79.39
Accuracy (%) 93.09 85.45 76.02 73.98 89.69 86.60 77.23 79.38
Matthews correl. coef. (MCC) 0.715 0.444 0.364 0.283 0.737 0.620 0.086 0.056
ROC AUC 0.933 0.743 0.840 0.767 0.966 0.890 0.897 0.831

models that are conceptually similar (i.e., models built with landslide sustain data-driven methods and the procedures to validate predictive
polygons – Model G versus Model K – and models built with a single models, which are crucial to correctly interpret any landslide suscepti-
point representing each landslide – Model L versus Model M), whereas bility map.
for models conceptually different (Model G/K versus Model L/M) the
agreement is only moderate (0.4 to 0.6). 4.1. Uncertainty within assumptions that sustain data-driven landslide sus-
The Kappa index increase substantially when only the very high sus- ceptibility assessment
ceptibility class is considered. The agreement is almost perfect between
Models K and G (Kappa index = 0.81) and Models L and M (Kappa As it was highlighted in the Introduction section, when applied to
index = 0.957). In addition, models produced with centroids of the landslide susceptibility assessment, data-driven methods are supported
landslide training group (Models L and M) have a higher agreement by the assumption that location of future landslides can be predicted
with Model G that was computed with the landslide area of the com- based on recognition of existing landslides and landslide predisposing
plete landslide inventory (Kappa index ranging from 0.753 to 0.796), factors, which are used to compute predictive models. It is widely ac-
whereas the agreement is lower with Model K that was computed cepted that the quality of data-driven landslide susceptibility models
with the total area of the same landslide training group used to comput- depends mostly on the quality of the landslide inventory as well as on
ed Models L and M (Kappa index ranging from 0.599 to 0.617). the relevance and reliability of the considered landslide predisposing
factors. However, landslides are discontinuous both in space and in
4. Discussion time, and the complete, coherent historical data concerning landslide
frequency is usually lacking in any study area. In addition, frequently
There are multiple sources of uncertainty within data-driven land- there are errors in landslide identification and mapping, namely for
slide susceptibility assessment that needs to be accounted and moni- old-poorly preserved landslide cases, and these errors propagate along
tored. Along this work, besides the influence of the statistical method, the methodological cascade of landslide susceptibility assessment.
we focused on topics that have received less attention by researchers, On the other hand, the relationship between landslide predisposing
as the influence of the chosen terrain mapping unit and the feature factors and landslide distribution is not always straightforward because
type to represent landslides. The discussion in the following sections the available thematic layers used as proxy to reproduce the slope insta-
covers these topics, and it is expanded to assumptions that conceptually bility system are heterogeneous regarding quality, precision and

Fig. 10. Landslide susceptibility maps obtained with logistic regression based on grid cells (5 m) terrain units (GCTU) and considering the landslide training group represented by the
centroid of each landslide polygon (Model L) (a), and the centroid of each landslide rupture zone polygon (Model M) (b).
 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267 263

resolution. Moreover, meaningful mappable landslide predisposing fac- Table 8

tors are dependent on scale (e.g. van Westen et al., 2008) and their Prediction skills of landslide susceptibility Models L and M.

physical significance is often poorly understood. On this subject, we Model L Model M

argue that a good spatial correlation among variables does not mean ROC metric Training Test Training Test
necessarily a cause–effect relationship. In addition to the previous Independent validation No Yes No Yes
shortcomings, the prediction of future landslide occurrence based on
True positive 1790 1093 1695 1077
landslide predisposing factors can be biased by landscape changes in True negative 534,324 551,980 547,043 552,326
unstable slopes, namely concerning the slope geometry and the land False positive 189,604 172,609 176,885 172,263
use. The evaluation of landslide susceptibility may be further biased be- False negative 264 300 359 316
cause different types of slope movements may correlate different with Positive predictive value (%) 0.94 0.63 0.95 0.62
Negative predictive value (%) 99.95 99.95 99.93 99.94
the same set of landslide predisposing factors in any study area, which Sensitivity (%) 87.15 78.46 82.52 77.32
can be solved by assessing landslide susceptibility independently for Specificity (%) 73.81 76.18 75.57 76.23
each existing landslide type. Accuracy (%) 73.85 76.18 75.59 76.23
Lastly, any data-driven landslide susceptibility method is conceptu- Matthews correl. coef. (MCC) 0.073 0.056 0.072 0.055
ROC AUC 0.854 0.856 0.849 0.848
ally supported on the idea that ‘past and present are keys to future’.
However, there is an increasing uncertainty regarding future relation-
ships between landslides and landslide factors in the context of climate
change and global change (e.g. Bruschi et al., 2013; Gariano and weights of evidence, likelihood ratio) to assess landslide susceptibility
Guzzetti, 2016), namely concerning probable changes in the return pe- over heterogeneous terrain units as STU, GHTU and CTU.
riod and in the magnitude-frequency relationships of future landslide However previous works (Garcia et al., 2009; Marques et al., 2009)
events. have shown that differences on prediction results decrease substantially
when multivariate and bivariate methods are performed over GCTU. In
4.2. The selection of statistical method addition, bivariate methods do not imply any sampling within the land-
slide-free areas to weight predisposing factors classes. The consider-
Usually, different statistical methods do not generate large differ- ation of data of the total study area may be considered as an
ences in landslide susceptibility assessment. In the present case study, advantage of bivariate methods in comparison with multivariate
multivariate methods (logistic regression and discriminant analysis) methods. Therefore, we consider that bivariate methods are as effective
generated better results than bivariate methods (information value) as multivariate methods, to model landslide susceptibility using grid cell
when using STU. cartographic databases.
The modest performance of the information value to model land-
slide susceptibility over STU is related with the simultaneous presence
in the same STU of variable classes with contradictory signal (positive
IV versus negative IV) that mutually nullify when the weighted average Table 9
Kappa values resulting from comparison of classified landslide susceptibility maps built
is computed to estimate the final IV of the STU. This drawback was not with logistic regression and using grid cells terrain units (Models G, K, L and M).
solved considering in the predictive model only the variable classes re-
cording positive scores. Therefore, we do not recommend the use of the Landslide Landslide susceptibility classes
susceptibility
information value, or other equivalent bivariate statistical method (e.g. Low and very Moderate High Very high Complete
maps
low (0–70%) (70–80%) (80–90%) (90–100%) maps

Model G versus Model K

Kappa index 0.854 0.480 0.609 0.810 0.730
Kappa location 0.860 0.489 0.610 0.810 0.735
Kappa 0.993 0.981 0.998 1.000 0.993
histogram

Model G versus Model L

Kappa index 0.656 0.109 0.197 0.769 0.491
Kappa location 0.689 0.115 0.245 0.775 0.525
Kappa 0.953 0.948 0.804 0.993 0.935
histogram

Model G versus Model M

Kappa index 0.585 0.072 0.377 0.753 0.502
Kappa location 0.640 0.178 0.459 0.757 0.597
Kappa 0.915 0.407 0.821 0.995 0.841
histogram

Model K versus Model L

Kappa index 0.632 0.138 0.119 0.617 0.445
Kappa location 0.658 0.148 0.149 0.621 0.476
Kappa 0.960 0.929 0.803 0.993 0.934
histogram

Model K versus Model M

Kappa index 0.565 0.018 0.179 0.599 0.411
Kappa location 0.612 0.043 0.218 0.602 0.485
Kappa 0.922 0.420 0.822 0.995 0.848
histogram
Fig. 11. ROC Curves of landslide susceptibility models obtained with logistic regression
Model L versus Model M
and GCTU. The landslide training group is represented by the centroid of each landslide
Kappa index 0.894 0.043 0.459 0.957 0.690
polygon (LI2) and the centroid of each landslide depletion zone polygon (LI3). The
Kappa location 0.930 0.115 0.720 0.959 0.847
validation is made with the landslide training group and the landslide test group. ROC
Kappa 0.962 0.375 0.638 0.998 0.815
curves of the model constructed using the complete area of landslide training group
histogram
(Model K) is also shown for comparison.
264 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267

Table 10 (evaluating the model's predictive power) we verified that the model
Area covered by the two highest landslide susceptibility classes within models A, E, F and made with CTU still generated the best ROC metrics (including the
G.
AUC ROC) for both validation procedures, which is interpreted as
Landslide susceptibility Terrain % number of terrain % of total study resulting from the overestimation of landslide susceptibility. The STU
model unit units area generates a very high AUC ROC for the training group (0.933) indicating
Model A STU 20 31 a very good fit of dependent variable, which contrast with the equiva-
Model E GHTU 20 20 lent AUC ROC obtained for the GHTU model (0.84). However, the corre-
Model F CTU 20 62
sponding AUC for the ROC curves fitting the landslide test group
Model G GCTU 20 20
indicate a better prediction skill for the GHTU model (0.767 against
0.743 for the STU model). The GCTU generated landslide susceptibility
4.3. The selection of terrain unit models with high AUC ROC for the training group (0.897) and lower
values, as expected, for the test group (0.831). However, the predictive
In the case study, when landslide susceptibility is modelled with the capacity of the model base on pixel is higher than those produced with
complete landslide inventory the CTU generate the highest AUC ROC STU and GHTU.
and other ROC metrics, but the variability of terrain unit size is very We argue that the precision of the spatial location of landslides is a
large, as it is attested by the standard deviation (Table 4) and landslides critical criterion for the selection of the most suitable terrain unit for
are mostly included in very large terrain units, which overestimate modelling. When the spatial accuracy of landslide inventory is good,
landslide susceptibility. The size effect of terrain mapping units is also the susceptibility should be assessed over grid cell terrain units. When
apparent in STU and justifies the higher AUC ROC for the corresponding the spatial accuracy is low the grid cell terrain units should not be
model (A) in comparison with the AUC ROC of models performed over used and may be substituted by a geo-hydrological terrain unit or, alter-
GHTU (E), which includes information about parent lithology, besides natively by a slope terrain unit or even an administrative terrain unit. In
the information of sub-basins already present in STU. The number of the latter cases, the use of a terrain unit with larger size than a grid cell
GHTU increases 2.5 times in comparison with STU, whereas the average helps to mitigate the consequences of existing positional errors within
size of UT decreases 2.5 times. The decreasing of AUC ROC from 0.858 the landslide inventory.
(Model A) to 0.816 (Model E) indicates that quality of Model A is prob-
ably overestimated. The GCUT generated, as expected, the major spatial 4.4. The validation of landslide susceptibility models
dispersion of the landslide susceptibility classes due to the much larger
number of terrain mapping units. The corresponding goodness of fit is Landslide susceptibility models need to be validated, which usually
high (AUC ROC = 0.883), only surpassed by the Model F performed implies the partitioning of the landslide inventory in subsets and the
with CTU. evaluation of the goodness of fit and the predictive capacity of the
The Table 10 clearly demonstrates that, although the higher AUC models. The partition of landslide inventories can be made based on
ROC and the other ROC metrics, susceptibility models built with CTU temporal, spatial or random criteria (Chung and Fabbri, 2003). The tem-
and STU (models F and A) are very conservative, as the percentage of poral partition of landslide databases is conceptually desirable because
the study area included in the two highest susceptibility classes is 62% it makes available a ‘future’ landslide group to independently validate
and 31%, respectively. In contrast, that feature is only 20% for models a susceptibility map that was generated based on the spatial distribu-
built with GHTU and GCTU (Models E and G). tion of ‘past’ landslides. The spatial partition of landslide databases is
When landslide susceptibility is modelled with half of the landslide the most adequate to evaluate the spatial transferability of a landslide
population (landslide training group) and validated with the training susceptibility model, whereas the random partition is the less exigent
group (evaluating the model's goodness of fit) and the test group strategy to generate landslide training and test groups, which usually
reflects in (artificial) better prediction results when compared with
temporal and spatial partitions (Pereira et al., 2012; Zêzere et al.,
2004a, 2004b). In the presented case study the random partition was
used due to existing constrains with the temporal and the spatial parti-
tion: the absolute and relative age of some landslides is unknown, and
the study area is too small to allow any reasonable spatial partition.
However, we recommend the use of temporal or spatial criteria to gen-
erate landslide training and test groups whenever possible.
The computation of the area under the curve (AUC) is the most pop-
ular metrics to estimate the model quality, which has been applied for
both ROC curves (e.g. Oliveira et al., 2015a; Pereira et al., 2012) and suc-
cess/prediction-rate curves (Guillard and Zêzere, 2012; Zêzere et al.,
2008). In recent years ROC curves have been preferred instead of suc-
cess/prediction curves because the former do not consider the same ter-
rain unit in both axes in the graph. However, the AUC ROC uses both the
landslide terrain units and the non-landslide terrain units to evaluate
model performance, and omission and commission errors are equally
considered, which can be a problem for predictive models. In contrast,
the success/prediction rate curves use only the landslide terrain units
for building curves.
We argue that the best landslide susceptibility model does not have,
necessarily, the best AUC ROC, as it has been shown in other scientific
fields (Lobo et al., 2008; Peterson et al., 2008). The Fig. 12 shows the per-
formance of landslide susceptibility models H, I, J and K expressed as
success-rate and prediction-rate curves and should be compared with
Fig. 12. Success rate and prediction rate curves of landslide susceptibility models H, I, J and Fig. 9. It is clear the ‘loss’ of quality of models performed over CTU,
K (to be compared with Fig. 9). STU and GHTU (Models J, H and I, respectively), not only concerning
 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267 265

the model's degree of fit (success rate, for the landslide training group) susceptibility assessment. The major advantage of slope terrain units
but also concerning the model's predictive power (prediction rate, for and geo-hydrological terrain units is the physical meaning of the terrain
the landslide test group). On contrary, the model built with GCTU unit, whereas the easy comparison of landslide susceptibility with data
(Model F) maintains a performance virtually equal the observed with on exposed elements (e.g. population and the building environment) is
ROC. a clear advantage of the census terrain unit. The easy computation pro-
We argue that differences between ROC curves and success/predic- cess in GIS and the stability of the modelling performance, including the
tion rate curves are minimal for landslide susceptibility models built validation procedure, are the major advantages of the grid cell terrain
over grid cell terrain units, namely if the total unstable are is only a unit. The lack of physical meaning of the terrain unit and the tricky pre-
very small fraction of the total study area (typically b2 to 3%). When cise location of the cell into the landscape are major drawbacks of grid
models are based in terrain units other than grid cell, the percentage cell terrain unit, whereas the uncertainty on the precise location of land-
of landslide affected units tends to increase following the reduction of slides inside the terrain unit is a major shortcoming for slope, geo-hy-
the total number of terrain units. As a consequence, results of landslide drological and census terrain units. The heterogeneity may be high
susceptibility models expressed by ROC curves may be optimistic in ex- inside these terrain units, but the classification of landslide susceptibil-
cess. In such cases, we consider more appropriated to evaluate the ity always considers each terrain unit as a whole. In such circumstances,
model performance by computing success and prediction rate curves. the landslide susceptibility tends to be overestimated, and the overesti-
In addition, the success/prediction rate curves have the advantage to mation tends to increase with the reduction of the total number of ter-
allow a predictive lecture (e.g. Zêzere et al., 2004a, 2004b), which is rain units within the study area.
not the case of the ROC curve. For instance, taking as an example the In this work we found evidence that the chosen terrain mapping unit
prediction-rate curve of the Model K (CGTU test, in Fig. 9) we can can produce greater differences on final susceptibility results than those
state that in a non-defined time span, 80% of future shallow translation- resulting from the chosen statistical method for modelling. In those
al slides should occur in the 30% of the study area classified as more sus- cases where the spatial accuracy of landslide inventory is good, we rec-
ceptible by the model. ommend to assess landslide susceptibility using grid cell terrain units.
Moreover, a single point per landslide proved to be efficient to generate
4.5. The selection of the feature type to represent landslides in the model accurate landslide susceptibility maps, even though the studied land-
slides are small sized shallow translational slides that present a remark-
Landslide susceptibility models made with GCTU using a single point able internal homogeneity regarding variables considered as landslide
to represent each landslide generated worst AUC ROC validating the predisposing factors (e.g. slope angle, lithology, land use). We recom-
landslide training group when compared with Model K. However, the mend that feasibility of landslide point features to correctly predict
ROC validating the (independent) landslide test group have equivalent landslide susceptibility should be verified in any case study, particularly
AUC (Model L = 0.856, Model M = 0.848, Model K = 0.831), which in- when the landscape is heterogeneous and the inventory includes large
dicates a similar predictive capacity of these models. Therefore we can size, deep-seated landslides. Whatever the model, the use of the land-
conclude that within a landscape with characteristics similar to the slide area for validation will generate a more robust result.
study area, a single point per landslide located in the centroid of the The validation of the landslide susceptibility map is a critical step in
landslide or in the centroid of the landslide rupture zone is enough to any landslide susceptibility assessment. Although during last years the
generate a reliable landslide susceptibility model, providing the land- ROC curves have been preferred to evaluate the susceptibility model's
slides are of small size, thus minimizing the possible existence of het- performance, we found evidence that the model with the highest AUC
erogeneities of predisposing factors within the landslide boundary. ROC is not necessarily the best landslide susceptibility model. Instead
However, we argue for the need to consider the landslide area in any of ROC curves, we recommend the use of success/prediction rate curves
landslide susceptibility validation, because landslides are not points to validate landslide susceptibility models when the ratio between un-
but polygon features within landscape. stable slope units and total slope units is typically high (higher than
0.1–0.15).
5. Concluding remarks Finally, we recommend maintaining all the effort to increase the
quality of landslide inventories and of landslide predisposing factors
Nowadays, landslide susceptibility maps are critically important for that still are the main drivers controlling the quality of landslide suscep-
stakeholders responsible for spatial planning and civil protection. In tibility models.
particular, the prediction of the spatial location of future landslides
has twofold regarding practical applications: on the one hand, it has
been included within development programmes aiming to avoid the Acknowledgments
most dangerous locations, namely for urbanization and critical infra-
structures; on the other hand, it has been crossed with the spatial distri- This work was financed by national funds through FCT – Portuguese
bution of people and assets to identify risk exposure. Along this work we Foundation for Science and Technology, I.P., under the framework of the
highlighted the existence of multiple sources of uncertainty within the project FORLAND – Hydro-geomorphologic risk in Portugal: driving
landslide susceptibility assessment process that need to be transferred forces and application for land use planning (PTDC/ATPGEO/1660/
to end-users. In particular, we focused our attention on the selection 2014). S.C. Oliveira is a post-doc fellow of the Portuguese Foundation
of the statistical method, the terrain mapping unit and the feature for Science and Technology (FCT) [grant number SFRH/BPD/85827/
type to represent landslides. 2012]. We are grateful to Tien Bui and an anonymous reviewer whose
Although bivariate statistical methods have been generating good pertinent comments and suggestions helped to improve the quality of
results when applied over grid-cells terrain units (e.g. Pereira et al., this work.
2012; Sterlacchini et al., 2011; Zêzere et al., 2004a, 2004b), they gener-
ated poor predictive results when applied over slope terrain units. Ap- References
parently, multivariate statistical methods perform better when
Alvioli, M., Marchesini, I., Reichenbach, P., Rossi, M., Fiorucci, F., Ardizzone, F., Guzzetti, F.,
computed over heterogeneous terrain units and should be selected to 2015. Automatic delineation of geomorphological slope-units and their optimization
assess landslide susceptibility based on slope terrain units, geo-hydro- for landslide susceptibility modelling. Geosci. Model Dev. Discuss.:1–33 http://dx.doi.
logical terrain units or census terrain units. org/10.5194/gmd-2016-118.
Ardizzone, F., Cardinali, M., Carrara, A., Guzzetti, F., Reichenbach, P., 2002. Impact of map-
The different types of terrain mapping units used in this work have ping errors on the reliability of landslide hazard maps. Nat. Hazards Earth Syst. Sci. 2:
advantages and drawbacks that should be considered in any landslide 3–14. http://dx.doi.org/10.5194/nhess-2-3-2002.
266 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267

Baeza, C., Lantada, N., Moya, J., 2010. Influence of sample and terrain unit on landslide sus- Glade, T., Crozier, M.J., 2005. A review of scale dependency in landslide hazard and risk
ceptibility assessment at La Pobla de Lillet, eastern Pyrenees, Spain. Environ. Earth Sci. analysis. In: Glade, T., Anderson, M., Crozier, M.J. (Eds.), Landslide Hazard and Risk.
60:155–167. http://dx.doi.org/10.1007/s12665-009-0176-4. John Wiley & Sons, Ltd., Chichester, West Sussex, England:pp. 75–138 http://dx.doi.
Blahut, J., van Westen, C.J., Sterlacchini, S., 2010. Analysis of landslide inventories for accu- org/10.1002/9780470012659.
rate prediction of debris-flow source areas. Geomorphology 119:36–51. http://dx.doi. Goetz, J.N., Brenning, A., Petschko, H., Leopold, P., 2015. Evaluating machine learning and
org/10.1016/j.geomorph.2010.02.017. statistical prediction techniques for landslide susceptibility modeling. Comput.
Brabb, E.E., Pampeyan, E.H., Bonilla, M.G., 1972. Landslide Susceptibility in San Mateo Geosci. 81:1–11. http://dx.doi.org/10.1016/j.cageo.2015.04.007.
County (California). Gorsevski, P.V., Gessler, P.E., Foltz, R.B., Elliot, W.J., 2006. Spatial prediction of landslide
Bradley, A.E., 1997. The use of the area under the ROC curve in the evaluation of machine hazard using logistic regression and ROC analysis. Trans. GIS 10:395–415. http://dx.
learning algorithms. Pattern Recogn. 30, 1145–1159. doi.org/10.1111/j.1467-9671.2006.01004.x.
Bruschi, V.M., Bonachea, J., Remondo, J., Gómez-Arozamena, J., Rivas, V., Méndez, G., Guillard, C., Zêzere, J.L., 2012. Landslide susceptibility assessment and validation in the
Naredo, J.M., Cendrero, A., 2013. Analysis of geomorphic systems' response to natural framework of municipal planning in Portugal: the case of Loures municipality. Envi-
and human drivers in northern Spain: implications for global geomorphic change. ron. Manag. 50:721–735. http://dx.doi.org/10.1007/s00267-012-9921-7.
Geomorphology 196:267–279. http://dx.doi.org/10.1016/j.geomorph.2012.03.017. Günther, A., Reichenbach, P., Malet, J.P., van Den Eeckhaut, M., Hervás, J., Dashwood, C.,
Budimir, M.E.A., Atkinson, P.M., Lewis, H.G., 2015. A systematic review of landslide prob- Guzzetti, F., 2013. Tier-based approaches for landslide susceptibility assessment in
ability mapping using logistic regression. Landslides 12:419–436. http://dx.doi.org/ Europe. Landslides 10:529–546. http://dx.doi.org/10.1007/s10346-012-0349-1.
10.1007/s10346-014-0550-5. Guzzetti, F., 2005. Landslide Hazard and Risk Assessment – Concepts, Methods and Tools
Carrara, A., Cardinali, M., Detti, R., Guzzetti, F., Pasqui, V., Reichenbach, P., 1991. GIS tech- for the Detection and Mapping of Landslides, for Landslides Susceptibility Zonation
niques and statistical models in evaluating landslide hazard. Earth Surf. Process. and Hazard Assessment, and for Landslide Risk Evaluation (PhD in Mathematich
Landf. 16:427–445. http://dx.doi.org/10.1002/esp.3290160505. naturwissenschaftli).
Carrara, A., Crosta, G., Frattini, P., 2003. Geomorphological and historical data in assessing Guzzetti, F., Carrara, A., Cardinali, M., Reichenbach, P., 1999. Landslide hazard evaluation:
landslide hazard. Earth Surf. Process. Landf. 28:1125–1142. http://dx.doi.org/10. a review of current techniques and their application in a multi-scale study, Central
1002/esp.545. Italy. Geomorphology 31:181–216. http://dx.doi.org/10.1016/S0169-
Carrara, A., Crosta, G., Frattini, P., 2008. Comparing models of debris-flow susceptibility in 555X(99)00078-1.
the alpine environment. Geomorphology 94:353–378. http://dx.doi.org/10.1016/j. Guzzetti, F., Cardinali, M., Reichenbach, P., Carrara, A., 2000. Comparing landslide maps: a
geomorph.2006.10.033. case study in the Upper Tiber River Basin, Central Italy. Environ. Manag. 25:247–263.
Catani, F., Lagomarsino, D., Segoni, S., Tofani, V., 2013. Landslide susceptibility estimation http://dx.doi.org/10.1007/s002679910020.
by random forests technique: sensitivity and scaling issues. Nat. Hazards Earth Syst. Guzzetti, F., Tonelli, G., 2004. Information system on hydrological and geomorphological
Sci. 13:2815–2831. http://dx.doi.org/10.5194/nhess-13-2815-2013. catastrophes in Italy (SICI): a tool for managing landslide and flood hazards. Nat. Haz-
Cervi, F., Berti, M., Borgatti, L., Ronchetti, F., Manenti, F., Corsini, A., 2010. Comparing pre- ards Earth Syst. Sci. 4:213–232. http://dx.doi.org/10.5194/nhess-4-213-2004.
dictive capability of statistical and deterministic methods for landslide susceptibility Guzzetti, F., Reichenbach, P., Cardinali, M., Galli, M., Ardizzone, F., 2005. Probabilistic land-
mapping: a case study in the northern Apennines (Reggio Emilia Province, Italy). slide hazard assessment at the basin scale. Geomorphology 72:272–299. http://dx.
Landslides 7:433–444. http://dx.doi.org/10.1007/s10346-010-0207-y. doi.org/10.1016/j.geomorph.2005.06.002.
Chazan, W., 1974. Le plan Zermos, Prévision des risques liés aux mouvements du sol et du Guzzetti, F., Galli, M., Reichenbach, P., Ardizzone, F., Cardinali, M., 2006. Landslide hazard
sous-sol et prévention de leurs effets. Ann. Min. 37–46. assessment in the Collazzone area, Umbria, Central Italy. Nat. Hazards Earth Syst. Sci.
Chen, Z., Wang, J., 2007. Landslide hazard mapping using logistic regression model in 6:115–131. http://dx.doi.org/10.5194/nhess-6-115-2006.
Mackenzie Valley, Canada. Nat. Hazards 42:75–89. http://dx.doi.org/10.1007/ Guzzetti, F., Mondini, A.C., Cardinali, M., Fiorucci, F., Santangelo, M., Chang, K.T., 2012.
s11069-006-9061-6. Landslide inventory maps: new tools for an old problem. Earth Sci. Rev. 112:42–66.
Chung, C.J.F., Fabbri, A.G., 1993. The representation of geoscience information for data in- http://dx.doi.org/10.1016/j.earscirev.2012.02.001.
tegration. Nonrenewable Resour. 2:122–139. http://dx.doi.org/10.1007/BF02272809. Heckmann, T., Gegg, K., Gegg, A., Becht, M., 2014. Sample size matters: investigating the
Chung, C.-J.F., Fabbri, A.G., 2003. Validation of spatial prediction models for landslide haz- effect of sample size on a logistic regression susceptibility model for debris flows.
ard mapping. Nat. Hazards 30:451–472. http://dx.doi.org/10.1023/B:NHAZ. Nat. Hazards Earth Syst. Sci. 14:259–278. http://dx.doi.org/10.5194/nhess-14-259-
0000007172.62651.2b. 2014.
Cohen, J., 1960. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. XX, Hosmer, D.W., Lemeshow, S., Sturdivant, R.X., 2013. Applied Logistic Regression. third ed.
37–46. Wiley, New York.
Conforti, M., Robustelli, G., Muto, F., Critelli, S., 2011. Application and validation of bivariate Hutchinson, J.N., 1995. Keynote paper: landslide hazard assessment. In: Bell, D.H. (Ed.),
GIS-based landslide susceptibility assessment for the Vitravo river catchment (Calabria, Landslides, Proceeding of 6th International Symposium on Landslides. Balkema, Rot-
South Italy). Nat. Hazards 61:127–141. http://dx.doi.org/10.1007/s11069-011-9781-0. terdam, pp. 1805–1841.
Corominas, J., van Westen, C., Frattini, P., Cascini, L., Malet, J.P., Fotopoulou, S., Catani, F., Jaiswal, P., van Westen, C.J., Jetten, V., 2010. Quantitative landslide hazard assessment
Van Den Eeckhaut, M., Mavrouli, O., Agliardi, F., Pitilakis, K., Winter, M.G., Pastor, along a transportation corridor in southern India. Eng. Geol. 116:236–250. http://
M., Ferlisi, S., Tofani, V., Hervás, J., Smith, J.T., 2014. Recommendations for the quanti- dx.doi.org/10.1016/j.enggeo.2010.09.005.
tative analysis of landslide risk. Bull. Eng. Geol. Environ. 73:209–263. http://dx.doi. Jiménez-Valverde, A., 2012. Insights into the area under the receiver operating character-
org/10.1007/s10064-013-0538-8. istic curve (AUC) as a discrimination measure in species distribution modelling. Glob.
Cox, D., 1958. The regression analysis of binary sequences. J. R. Stat. Soc. Ser. B Methodol. Ecol. Biogeogr. http://dx.doi.org/10.1111/j.1466-8238.2011.00683.x.
20, 215–242. Kanungo, D., Arora, M., Sarkar, S., Gupta, R., 2009. Landslide susceptibility zonation (LSZ)
Den Eeckhaut, M., Van Marre, A., Poesen, J., 2010. Comparison of two landslide suscepti- mapping–a review. J. South Asia Disaster Stud. 2, 81–105.
bility assessments in the Champagne–Ardenne region (France). Geomorphology 115: Landis, J.R., Koch, G.G., 1977. The measurement of observer agreement for categorical
141–155. http://dx.doi.org/10.1016/j.geomorph.2009.09.042. data. Biometrics 33, 159–174.
Erener, A., Düzgün, H.S.B., 2012. Landslide susceptibility assessment: what are the effects Lee, S., Choi, J., 2004. Landslide susceptibility mapping using GIS and the weight-of-evi-
of mapping unit and mapping method? Environ. Earth Sci. 66:859–877. http://dx.doi. dence model. Int. J. Geogr. Inf. Sci. 18:789–814. http://dx.doi.org/10.1080/
org/10.1007/s12665-011-1297-0. 13658810410001702003.
Fawcett, T., 2006. An introduction to ROC analysis. Pattern Recogn. Lett. 27, 861–874. Lee, C.-T., Huang, C.-C., Lee, J.-F., Pan, K.-L., Lin, M.-L., Dong, J.-J., 2008. Statistical approach
Felicísimo, Á.M., Cuartero, A., Remondo, J., Quirós, E., 2013. Mapping landslide susceptibil- to earthquake-induced landslide susceptibility. Eng. Geol. 100:43–58. http://dx.doi.
ity with logistic regression, multiple adaptive regression splines, classification and re- org/10.1016/j.enggeo.2008.03.004.
gression trees, and maximum entropy methods: a comparative study. Landslides 10: Lobo, J.M., Jiménez-valverde, A., Real, R., 2008. AUC: a misleading measure of the perfor-
175–189. http://dx.doi.org/10.1007/s10346-012-0320-1. mance of predictive distribution models. Glob. Ecol. Biogeogr. 17:145–151. http://dx.
Fell, R., Corominas, J., Bonnard, C., Cascini, L., Leroi, E., Savage, W.Z., 2008. Guidelines for doi.org/10.1111/j.1466-8238.2007.00358.x.
landslide susceptibility, hazard and risk zoning for land use planning. Eng. Geol. Magliulo, P., Di Lisio, A., Russo, F., 2009. Comparison of GIS-based methodologies for the
102:85–98. http://dx.doi.org/10.1016/j.enggeo.2008.03.022. landslide susceptibility assessment. GeoInformatica 13:253–265. http://dx.doi.org/
Fiorucci, F., Cardinali, M., Carlà, R., Rossi, M., Mondini, A.C., Santurri, L., Ardizzone, F., 10.1007/s10707-008-0063-2.
Guzzetti, F., 2011. Seasonal landslide mapping and estimation of landslide mobiliza- Marques, R., Amaral, P., Zêzere, J.L., Queiroz, G., Goulart, C., 2009. Estudo comparativo de
tion rates using aerial and satellite images. Geomorphology 129:59–70. http://dx. diferentes métodos probabilísticos para a avaliação da susceptibilidade à ocorrência
doi.org/10.1016/j.geomorph.2011.01.013. de movimentos de vertente: um caso de estudo no vale da Ribeira Quente (S. Miguel,
Fisher, R.A., 1936. The use of multiple measurements in taxonomic problems. Ann. Eugen- Açores). Publicações Da Associação Portuguesa de Geomorfólogos, pp. 183–190.
ics 7, 179–188. Matthews, B.W., 1975. Comparison of the predicted and observed secondary structure of
Galli, M., Ardizzone, F., Cardinali, M., Guzzetti, F., Reichenbach, P., 2008. Comparing land- T4 phage lysozyme. Biochim. Biophys. Acta 405, 442–451.
slide inventory maps. Geomorphology 94:268–289. http://dx.doi.org/10.1016/j. Mondini, A.C., Viero, A., Cavalli, M., Marchi, L., Herrera, G., Guzzetti, F., 2014. Comparison
geomorph.2006.09.023. of event landslide inventories: the Pogliaschina catchment test case, Italy. Nat. Haz-
Garcia, R.A.C., Zêzere, J.L., Oliveira, S.C., 2009. A influência da unidade de terreno na ards Earth Syst. Sci. 14:1749–1759. http://dx.doi.org/10.5194/nhess-14-1749-2014.
avaliação da susceptibilidade a movimentos de vertente. Publicações Da Associação Oliveira, S.C., Zêzere, J.L., Catalão, J., Nico, G., 2015a. The contribution of PSInSAR interfer-
Portuguesa de Geomorfólogos, pp. 169–174. ometry to landslide hazard in weak rock-dominated areas. Landslides 12:703–719.
Garcia, R.A.C., Oliveira, S.C., Zêzere, J.L., 2016. Assessing population exposure for landslide http://dx.doi.org/10.1007/s10346-014-0522-9.
risk analysis using dasymetric cartography. Nat. Hazards Earth Syst. Sci. 16: Oliveira, S.C., Zêzere, J.L., Garcia, R.A.C., 2015b. Structure and characteristics of landslide
2769–2782. http://dx.doi.org/10.5194/nhess-2016-202. input data and consequences on landslide susceptibility assessment and prediction
Gariano, S.L., Guzzetti, F., 2016. Landslides in a changing climate. Earth Sci. Rev. 162: capability. In: Lollino, G., Giordan, D., Crosta, G.B., Corominas, J., Azzam, R.,
227–252. http://dx.doi.org/10.1016/j.earscirev.2016.08.011. Wasowski, J., Sciarra, N. (Eds.), Engineering Geology for Society and Territory.
 J.L. Zêzere et al. / Science of the Total Environment 589 (2017) 250–267 267

Landslide Processes Vol. 2. Springer International Publishing, Switzerland: Van Den Eeckhaut, M., Vanwalleghem, T., Poesen, J., Govers, G., Verstraeten, G.,
pp. 1–2177. http://dx.doi.org/10.1007/978-3-319-09057-3. Vandekerckhove, L., 2006. Prediction of landslide susceptibility using rare events lo-
Pardeshi, S.D., Autade, S.E., Pardeshi, S.S., 2015. Landslide Hazard Assessment: Recent gistic regression: a case-study in the Flemish Ardennes (Belgium). Geomorphology
Trends and Techniques. 1988. Springerplus:pp. 1–11. http://dx.doi.org/10.1186/ 76:392–410. http://dx.doi.org/10.1016/j.geomorph.2005.12.003.
2193. Van Den Eeckhaut, M., Reichenbach, P., Guzzetti, F., Rossi, M., Poesen, J., 2009. Combined
Pereira, S., Zêzere, J.L., Bateira, C., 2012. Technical note: assessing predictive capacity and landslide inventory and susceptibility assessment based on different mapping units:
conditional independence of landslide predisposing factors for shallow landslide sus- an example from the Flemish Ardennes, Belgium. Nat. Hazards Earth Syst. Sci. 9:
ceptibility models. Nat. Hazards Earth Syst. Sci. 12:979–988. http://dx.doi.org/10. 507–521. http://dx.doi.org/10.5194/nhess-9-507-2009.
5194/nhess-12-979-2012. Varnes, D.J., 1984. Landslide hazard zonation: a review of principles and practice. United
Pereira, S., Zêzere, J.L., Quaresma, I.D., Bateira, C., 2014. Landslide incidence in the north of Nations Educational. Scientific and Cultural Organization, Paris http://dx.doi.org/10.
Portugal: analysis of a historical landslide database based on press releases and tech- 1016/B978-008043013-3/50018-3.
nical reports. Geomorphology 214:514–525. http://dx.doi.org/10.1016/j.geomorph. Visser, H., De Nijs, T., 2006. The map comparison kit. Environ. Model. Softw. 21:346–358.
2014.02.032. http://dx.doi.org/10.1016/j.envsoft.2004.11.013.
Pereira, S., Garcia, R.A.C., Zêzere, J., Oliveira, S., Silva, M., 2016. Landslide quantitative risk Vorpahl, P., Elsenbeer, H., Märker, M., Schröder, B., 2012. How can statistical models help
analysis of buildings at the municipal scale based on a rainfall triggering scenario. to determine driving factors of landslides? Ecol. Model. 239:27–39. http://dx.doi.org/
Geomat. Nat. Haz. Risk:1–25 http://dx.doi.org/10.1080/19475705.2016.1250116. 10.1016/j.ecolmodel.2011.12.007.
Perkins, N.J., Schisterman, E.F., 2005. The Youden index and the optimal cut-point Wang, X., Zhang, L., Wang, S., Lari, S., 2014. Regional landslide susceptibility zoning with
corrected for measurement error. Biom. J. http://dx.doi.org/10.1002/bimj.200410133. considering the aggregation of landslide points and the weights of factors. Landslides
Peterson, A.T., Papeş, M., Soberón, J., 2008. Rethinking receiver operating characteristic 11:399–409. http://dx.doi.org/10.1007/s10346-013-0392-6.
analysis applications in ecological niche modeling. Ecol. Model. http://dx.doi.org/10. van Westen, C.J., Castellanos, E., Kuriakose, S.L., 2008. Spatial data for landslide suscepti-
1016/j.ecolmodel.2007.11.008. bility, hazard, and vulnerability assessment: an overview. Eng. Geol. 102:112–131.
Petley, D., 2012. Global patterns of loss of life from landslides. Geology 40:927–930. http://dx.doi.org/10.1016/j.enggeo.2008.03.010.
http://dx.doi.org/10.1130/G33217.1. Yan, T., 1988. Recent advances of quantitative prognoses of landslides in China. Proceed-
Santangelo, M., Marchesini, I., Bucci, F., Cardinali, M., Fiorucci, F., Guzzetti, F., 2015. An ap- ings of the 5th International Symposim on Landslides, pp. 1263–1268.
proach to reduce mapping errors in the production of landslide inventory maps. Nat. Yilmaz, I., 2009. GIS based statistical and physical approaches to landslide susceptibility
Hazards Earth Syst. Sci. 15:2111–2126. http://dx.doi.org/10.5194/nhess-15-2111- mapping (Sebinkarahisar, Turkey). Bull. Eng. Geol. Environ. 68:459–471. http://dx.
2015. doi.org/10.1007/s10064-009-0188-z.
Sawatzky, D.L., Raines, G.L., Bonham-Carter, G.F., Looney, C.G., 2009. Spatial Data Modeller Yin, K.L., Yan, T.Z., 1988. Statistical prediction models for slope instability of metamor-
(SDM): ArcMAP 9.3 Geoprocessing Tools for Spatial Data Modelling Using Weights of phosed rocks. Landslides. Proc. 5th Symposium, Lausanne, 1988. Vol. 2,
Evidence, Logistic Regression, Fuzzy Logic and Neural Networks. pp. 1269–1272.
Soeters, R., van Westen, C.J., 1996. Slope instability recognition, analysis and zonation. Zêzere, J.L., 2002. Landslide susceptibility assessment considering landslide typology. A
Landslides - Investigation and Mitigation. National Academy Press, Washington D. case study in the area north of Lisbon (Portugal). Nat. Hazards Earth Syst. Sci. 2,
C., pp. 129–177 (Special Report 247). 73–82.
Steger, S., Brenning, A., Bell, R., Glade, T., 2016. The impact of systematically incomplete Zêzere, J.L., Reis, E., Garcia, R.A.C., Oliveira, S.C., Rodrigues, M.L., Vieira, G., Ferreira, A.B.,
and positionally inaccurate landslide inventories on statistical landslide susceptibility 2004a. Integration of spatial and temporal data for the definition of different land-
models. Nat. Hazards Earth Syst. Sci. Discuss.:1–25 http://dx.doi.org/10.5194/nhess- slide hazard scenarios in the area north of Lisbon (Portugal). Nat. Hazards Earth
2016-301. Syst. Sci. 4, 133–146.
Sterlacchini, S., Ballabio, C., Blahut, J., Masetti, M., Sorichetta, A., 2011. Spatial agreement of Zêzere, J.L., Rodrigues, M.L., Reis, E., Garcia, R.A.C., Oliveira, S.C., Vieira, G., Ferreira, A.B.,
predicted patterns in landslide susceptibility maps. Geomorphology 125:51–61. 2004b. Spatial and temporal data management for the probabilistic landslide hazard
http://dx.doi.org/10.1016/j.geomorph.2010.09.004. assessment considering landslide typology. In: Lacerda, Ehrlich, Fontoura, Sayão
Suzen, M.L., Doyuran, V., 2004. A comparison of the GIS based landslide susceptibility as- (Eds.), Landslides: Evaluation and Stabilization. Taylor & Francis Group, London,
sessment methods: multivariate versus bivariate. Environ. Geol. 45:665–679. http:// pp. 117–123.
dx.doi.org/10.1007/s00254-003-0917-8. Zêzere, J.L., Garcia, R.A.C., Oliveira, S.C., Reis, E., 2008. Probabilistic landslide risk analysis
Tien Bui, D., Tuan, T.A., Hoang, N.-D., Thanh, N., Nguyen, D.B., Liem, N.V., Pradhan, B., considering direct costs in the area north of Lisbon (Portugal). Geomorphology 94:
2016a. Spatial prediction of rainfall-induced landslides for the Lao Cai area (Vietnam) 467–495. http://dx.doi.org/10.1016/j.geomorph.2006.10.040.
using a hybrid intelligent approach of least squares support vector machines infer- Zêzere, J., Henriques, C., Garcia, R., Oliveira, S., Piedade, A., Neves, M., 2009. Effects of land-
ence model and artificial bee colony optimization. Landslides http://dx.doi.org/10. slide inventories uncertainty on landslide susceptibility modelling. In: Malet, J.-P.,
1007/s10346-016-0711-9. Remaître, A., Boggard, T. (Eds.), Proceedings of the Landslide Processes. From
Tien Bui, D., Ho, T.C., Pradhan, B., Pham, B.T., Nhu, V.H., Revhaug, I., 2016b. GIS-based Geomorphologic Mapping to Dynamic Modeling Conference. A Tribute to Prof. Theo
modeling of rainfall-induced landslides using data mining-based functional trees van Asch. CERG. Utrech University and University of Strasbourg, pp. 81–86.
classifier with AdaBoost, Bagging, and MultiBoost ensemble frameworks. Environ.
Earth Sci. http://dx.doi.org/10.1007/s12665-016-5919-4.