Evaluating observed versus predicted forest biomass: R-squared, index of agreement or maximal information coefficient?

Abstract

The accurate prediction of forest above-ground biomass is nowadays key to implementing climate change mitigation policies, such as reducing emissions from deforestation and forest degradation. In this context, the coefficient of determination (R 2) is widely used as a means of evaluating the proportion of variance in the dependent variable explained by a model. However, the validity of R 2 for comparing observed versus predicted values has been challenged in the presence of bias, for instance in remote sensing predictions of forest biomass. We tested suitable alternatives, e.g. the index of agreement (d) and the maximal information coefficient (MIC). Our results show that d renders systematically higher values than R 2 , and may easily lead to regarding as reliable models which included an unrealistic amount of predictors. Results seemed better for MIC, although MIC favoured local clustering of predictions, whether or not they corresponded to the observations. Moreover, R 2 was more sensitive to the use of cross-validation than d or MIC, and more robust against overfitted models. Therefore, we discourage the use of statistical measures alternative to R 2 for evaluating model predictions versus observed values, at least in the context of assessing the reliability of modelled biomass predictions using remote sensing. For those who consider d to be conceptually superior to R 2 , we suggest using its square d 2 , in order to be more analogous to R 2 and hence facilitate comparison across studies.

Comparison of Machine Learning Methods for Mapping the Stand Characteristics of Temperate Forests Using Multi-Spectral Sentinel-2 Data

“…The low R 2 value were observed by several studies and reported by [27]. Valbuena et al [76] quoted from several researchers that lower R 2 value is not always an indicator for lower accuracy of predictions. Fatehi et al [77] also experienced very low R 2 value especially for stem density estimation using digital terrain model of 1-m grid (airborne laser scanning) and multi-spectral image of 30-m resolution (imaging spectroscopy) to predict tree density and forest productivity in a heterogeneous Alpine landscape.…”

Recommendations on benchmarks for photochemical grid model applications in China: Part I – PM<sub>2.5</sub> and chemical species

“…Compared with R 2 values, the original IOA results systematically higher values (Valbuena et al, 2019) thus is being adopted in an increasing number of studies partially because it makes results appear -better‖. However, the original and also being the most widely used IOA is problematic in that too much weight is given to the large errors when squared (Willmott et al, 30 2012) and relatively high IOA values could be obtained even when a model is performing poorly (Willmott et al, 1985;Pereira et al, 2017).…”

“…Newer versions as later proposed by Willmott overcome this problem by removing the squaring and are recommended over the original one (Willmott et al, 1985(Willmott et al, , 2012. Valbuena et al (2019) suggested using d 2 instead of d, at least for estimating forest biomass based on remote sensing to facilitate comparison with studies using correlation coefficient.…”

Comparison of Statistical Modelling Approaches for Estimating Tropical Forest Aboveground Biomass Stock and Reporting Their Changes in Low-Intensity Logging Areas Using Multi-Temporal LiDAR Data

“…Additionally, we demonstrate that methods such as RF and SVM that performed close to the OLS can be used to estimate and make inferences when necessary; that is, in situations where there exist non-linear or diverse relationships between dependent and independent variables [73]. The performances of RF and SVM, as the best among non-parametric approaches, may have been affected not only by the number of field plots but also by other factors such as bootstrapping of data to avoid overfitting R 2 values [74].…”