Spatially autocorrelated training and validation samples inflate performance assessment of convolutional neural networks

Kattenborn, Teja; Schiefer, Felix; Frey, Julian; Feilhauer, Hannes; Mahecha, Miguel D.; Dormann, Carsten F.

doi:10.1016/j.ophoto.2022.100018

Cited by 40 publications

(24 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note, that the transferability of the models was successfully tested with entirely independent image acquisitions, which may even include acquisition and site conditions that were not explicitly included in model training (cf. Kattenborn et al, 2022 ). Thus, the model presented here could be applied to quantify P. afra cover in restoration sites across the Albany Subtropical Thicket biome in South Africa.…”

Section: Discussionmentioning

confidence: 99%

“…To avoid optimistic model evaluation by spatially autocorrelated training and validation data ( Ploton et al, 2020 ; Kattenborn et al, 2022 ), we randomly split all available data on a plot basis, where a portion of plots was used for model training ( n = 24) and the remainder used for model testing ( n = 8). The training data was again split in training (7/8) and validation data (1/8), whereas the validation data was used to monitor the training process.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Automated mapping of Portulacaria afra canopies for restoration monitoring with convolutional neural networks and heterogeneous unmanned aerial vehicle imagery

Galuszynski

Duker

Potts

et al. 2022

PeerJ

Self Cite

View full text Add to dashboard Cite

Ecosystem restoration and reforestation often operate at large scales, whereas monitoring practices are usually limited to spatially restricted field measurements that are (i) time- and labour-intensive, and (ii) unable to accurately quantify restoration success over hundreds to thousands of hectares. Recent advances in remote sensing technologies paired with deep learning algorithms provide an unprecedented opportunity for monitoring changes in vegetation cover at spatial and temporal scales. Such data can feed directly into adaptive management practices and provide insights into restoration and regeneration dynamics. Here, we demonstrate that convolutional neural network (CNN) segmentation algorithms can accurately classify the canopy cover of Portulacaria afra Jacq. in imagery acquired using different models of unoccupied aerial vehicles (UAVs) and under variable light intensities. Portulacaria afra is the target species for the restoration of Albany Subtropical Thicket vegetation, endemic to South Africa, where canopy cover is challenging to measure due to the dense, tangled structure of this vegetation. The automated classification strategy presented here is widely transferable to restoration monitoring as its application does not require any knowledge of the CNN model or specialist training, and can be applied to imagery generated by a range of UAV models. This will reduce the sampling effort required to track restoration trajectories in space and time, contributing to more effective management of restoration sites, and promoting collaboration between scientists, practitioners and landowners.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Automated mapping of Portulacaria afra canopies for restoration monitoring with convolutional neural networks and heterogeneous unmanned aerial vehicle imagery

Galuszynski

Duker

Potts

et al. 2022

PeerJ

Self Cite

View full text Add to dashboard Cite

show abstract

“…1). Further work is necessary to explore how these dependences influence model prediction and interpretability (Kattenborn et al, 2022).…”

Section: Caveats and Future Researchmentioning

confidence: 99%

Insights into the vulnerability of vegetation to tephra fallouts from interpretable machine learning and big Earth observation data

Biass

Jenkins²,

Aeberhard³

et al. 2022

Nat. Hazards Earth Syst. Sci.

View full text Add to dashboard Cite

Abstract. Although the generally high fertility of volcanic soils is often seen as an opportunity, short-term consequences of eruptions on natural and cultivated vegetation are likely to be negative. The empirical knowledge obtained from post-event impact assessments provides crucial insights into the range of parameters controlling impact and recovery of vegetation, but their limited coverage in time and space offers a limited sample of all possible eruptive and environmental conditions. Consequently, vegetation vulnerability remains largely unconstrained, thus impeding quantitative risk analyses. Here, we explore how cloud-based big Earth observation data, remote sensing and interpretable machine learning (ML) can provide a large-scale alternative to identify the nature of, and infer relationships between, drivers controlling vegetation impact and recovery. We present a methodology developed using Google Earth Engine to systematically revisit the impact of past eruptions and constrain critical hazard and vulnerability parameters. Its application to the impact associated with the tephra fallout from the 2011 eruption of Cordón Caulle volcano (Chile) reveals its ability to capture different impact states as a function of hazard and environmental parameters and highlights feedbacks and thresholds controlling impact and recovery of both natural and cultivated vegetation. We therefore conclude that big Earth observation (EO) data and machine learning complement existing impact datasets and open the way to a new type of dynamic and large-scale vulnerability models.

show abstract

“…If training and test crowns are close to one another, spatial autocorrelative effects are likely to inflate the reported performance (52). To avoid this, individual tiles (rather than individual crowns) were assigned to training and test sets ensuring spatial separation.…”

Section: R a F Tmentioning

confidence: 99%

“…To be included in the training and test sets, a minimum crown polygon area coverage of a tile was set at 40%. Including overly sparse tiles was likely to lead to poor algorithm sensitivity while being too strict with coverage would have limited the amount of training and testing data available.If training and test crowns are close to one another, spatial autocorrelative effects are likely to inflate the reported performance(53). To avoid this, individual tiles (rather than individual crowns) were assigned to training and test sets ensuring spatial separation.…”

mentioning

confidence: 99%

Accurate delineation of individual tree crowns in tropical forests from aerial RGB imagery using Mask R-CNN

Hickman¹,

Ball

Jackson

et al. 2022

Preprint

View full text Add to dashboard Cite

Tropical forests are a major component of the global carbon cycle and home to two-thirds of terrestrial species. Upper-canopy trees store the majority of forest carbon and can be particularly vulnerable to drought events and storms. Monitoring their growth and mortality is essential to understanding forest resilience to climate change, but large trees are underrepresented in traditional field surveys, so estimates are poorly constrained. Aerial photographs provide spectral and textural information to discriminate between tree crowns in diverse, complex tropical canopies, potentially opening the door to landscape monitoring of large trees. Here we describe a new deep convolutional neural network machine learning method, Detectree2, which builds on the Mask R-CNN computer vision framework to recognise the irregular edges of individual tree crowns from airborne RGB imagery. We trained and evaluated this model with 3,600 manually delineated tree crowns at three sites in Malaysian Borneo and one site in French Guiana. As an example application of this tool, we combined the delineations with repeat lidar surveys of the four sites to estimate the growth and mortality of upper-canopy trees. Detectree2 delineated 65,000 upper-canopy trees across 14 km^2 of aerial images. The skill of the automatic method in delineating the upper-canopy tree crowns was good (F1 = 0.71). As predicted from previous field studies, we found that growth rate declined with tree height and tall trees had higher mortality rates than intermediate-size trees. In addition, tall trees in French Guiana had higher growth and mortality rates than those in Borneo. Our approach demonstrates that machine learning methods can automatically segment trees in abundant, cheap RGB imagery. This tool has many potential applications in forest ecology and conservation, from estimating carbon stocks to monitoring forest restoration. We demonstrate its use in tracking the growth and mortality rates of upper-canopy trees at scales much larger than those achievable with field data.

show abstract

Spatially autocorrelated training and validation samples inflate performance assessment of convolutional neural networks

Cited by 40 publications

References 37 publications

Automated mapping of Portulacaria afra canopies for restoration monitoring with convolutional neural networks and heterogeneous unmanned aerial vehicle imagery

Automated mapping of Portulacaria afra canopies for restoration monitoring with convolutional neural networks and heterogeneous unmanned aerial vehicle imagery

Insights into the vulnerability of vegetation to tephra fallouts from interpretable machine learning and big Earth observation data

Accurate delineation of individual tree crowns in tropical forests from aerial RGB imagery using Mask R-CNN

Contact Info

Product

Resources

About