In this section, we first explain how we evaluate our findings and then share the results of the experiments.
Experimental Results
In this section, we unveil our findings and provide the evaluation results concerning different aspects, such as Local Moran’s I and the POI intensity per fishnet. Additionally, we present and compare POI-based LUM identification outcomes from these two distinct urban settings.
Using the point-based data from Google Maps API and ad listings data to predict LUM, we present the LUM results on a map to further understand their distribution across Ankara city.
Figure 8 shows the areas in Ankara according to their LUM values, in which gray areas show not applicable results. The colored squares indicate the LUM values of the associated places.
According to the map, as the color becomes dark, the LUM values increase. In general, areas that are far from the city center usually have lower LUM values. On the other hand, the areas with higher LUM values are places that have both residential and non-residential activities in place. Gray areas show the fishnets with not enough POI data, and the darker the color, the higher the LUM.
To evaluate our LUM findings, we first observed the overall results with respect to the ground truth. In general, we wanted to identify whether an area is mixed-use or not. On top of the entropy-based mix estimation, it was essential to label areas as mixed and non -mixed with a confidential threshold. We examined the classification accuracy, precision, recall, and F1-score. In order to classify areas whether they have mixed land use or not, we first defined a cutoff point. For this purpose, we first set 20% of the regions we managed to identify LUM values aside based on a stratified sampling where we obtain samples from places having [0–0.1), [0.1–0.2), [0.2–0.3), …, [0.9–1] LUM values. We then tested different thresholds on that portion via drawing ROCs for each of the cutoff points, which are depicted in
Figure 9.
As a result, we selected the one that has the largest AUC, which is cutoff = 0.5 with AUC = 0.57. After setting the cutoff point to 0.5, we assumed that places having equal or lower than 0.5 LUM values have no mixed land use and others have mixed land use.
Additionally, we also set two baselines to compare the results—one for all the fishnets with no mix representing homogeneous single land use and one for all the fishnets having mixed land use. The overall results are presented in
Table 2.
Overall, our only POI-based model’s accuracy is 51%, its precision is 48%, its recall is 69%, and its F1-score is 57%. To further understand the results, we scrutinized the areas with respect to neighborhoods and POI density.
In addition to directly comparing the predicted and ground truth LUM values, we also examined the local spatial autocorrelation. Local spatial autocorrelation measures landscape fragmentation in comparison to landscape metrics [
32]. It finds spatial dependency of features in space to imply landscape heterogeneity. In this study, we used one of the local spatial autocorrelation methods, Local Moran’s I, [
33] to identify the spatial pattern of different fishnet LUM values.
Local Moran’s I fundamentally has two interpretations. On the one hand, it lets you find hot spots. On the other hand, it identifies outliers. Thanks to Local Moran’s I (LMI), we detected local clusters and local spatial outliers to further analyze our LUM classification findings depending on the Local Moran’s I index of each fishnet’s LUM value.
As Tobler’s first law of geography states, spatial distance affects the relationship among things. Hence, ignoring spatial distance among fishnets will make the results biased. Therefore, we calculate the Local Moran’s I for each fishnet based on the assigned LUM values. In
Table 3, we presented the classification accuracy, precision, recall, and F1-score for different significance ranges of the Local Moran’s I analysis. The main motivation was to investigate whether the scores increase proportionally to the significance level, and it is confirmed that when the significance of LMI increases, our classification scores also become better. For instance, for
p-values less than and equal to 0.01, we saw 71% classification accuracy, 63% precision, 92% recall, and 75% F1-score, with the baseline of all 0 s being 58%.
Additionally, we look at the classification accuracy, precision, recall, and F1-score concerning the different numbers of POIs residing on fishnets. In order to specify the bins, we first started with equal-depth binning and then tweaked the ranges to make sure that distributions of LUM values in each bin are different from the others. We used a non-parametric Mann–Whitney U test with a p-value to make sure that they were not coming from the same distributions. As a result, we created five bins depending on the POI counts with LUM values coming from different distributions.
In
Table 4, the results are presented for different bins such as [4–5], [6–11], [12,17], [18–28], and [29–60]; as a result, we found that the accuracy of finding out whether a fishnet has mix land use or no-mix land use increases up to 65% followed by the 67% precision, 89% recall, and 77% F1-score when the POI count is between 29 and 60. In other words, considering that we are working with fishnets with an area of 0.2655 km
2, if the POI density ranges between 109 POI/km
2 and 226 POI/km
2, we can reach the highest mixed land use classification accuracy of 77% and F1-score of 80%.
Finally, we compared the results from Ankara to Kadıköy with the same assumption of having between 29 and 60 points within a fishnet eye, we find the following results presented in
Table 5.
Comparing the results from Ankara to those from Kadıköy, we observe that the latter demonstrates a slightly better accuracy and F1-score. This is likely due to Kadıköy’s higher density of mixed-use developments which aligned well with our POI-based methodology.