Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling

Elshall, Ahmed S.; Ye, Ming; Pei, Yongzhen; Zhang, Fan; Niu, Guo Yue; Barron‐Gafford, Greg A.

doi:10.1007/s00477-018-1592-3

Cited by 8 publications

(16 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This suggests again that one needs to be cautious when building autocorrelation into a data model. This is consistent with the finding of Evin et al (2013Evin et al ( , 2014 that accounting for autocorrelation before accounting for heteroscedasticity or jointly accounting for autocorrelation and heteroscedasticity can result in poor predictive performance. In summary, Fig.…”

Section: Predictive Performance With Total Uncertaintysupporting

confidence: 91%

“…While heteroscedasticity can be accounted for via residuals transformation (e.g., Thiemann et al, 200;T. Smith et al, 2010) or other similar approaches (Gragne et al, 2015), a linear heteroscedastic model σ t = σ 0 + σ 1 Y t is assumed here following the studies of Thyer et al (2009), Schoups and Vrugt (2010), and Evin et al (2013Evin et al ( , 2014. With the linear model, there is no need to estimate σ t for each piece of data.…”

Section: Data Modelsmentioning

confidence: 99%

“…It is worth pointing out that other methods exist to account for the three assumptions. Evin et al (2013) suggested accounting for residual heteroscedasticity before accounting for residual autocorrelation. Lu et al (2013) developed an iterative two-stage procedure to separately estimate physical model parameters and data model parameters.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Bayesian inference and predictive performance of soil respiration models in the presence of model discrepancy

Elshall

Niu

et al. 2019

Geosci. Model Dev.

Self Cite

View full text Add to dashboard Cite

Abstract. Bayesian inference of microbial soil respiration models is often based on the assumptions that the residuals are independent (i.e., no temporal or spatial correlation), identically distributed (i.e., Gaussian noise), and have constant variance (i.e., homoscedastic). In the presence of model discrepancy, as no model is perfect, this study shows that these assumptions are generally invalid in soil respiration modeling such that residuals have high temporal correlation, an increasing variance with increasing magnitude of CO2 efflux, and non-Gaussian distribution. Relaxing these three assumptions stepwise results in eight data models. Data models are the basis of formulating likelihood functions of Bayesian inference. This study presents a systematic and comprehensive investigation of the impacts of data model selection on Bayesian inference and predictive performance. We use three mechanistic soil respiration models with different levels of model fidelity (i.e., model discrepancy) with respect to the number of carbon pools and the explicit representations of soil moisture controls on carbon degradation; therefore, we have different levels of model complexity with respect to the number of model parameters. The study shows that data models have substantial impacts on Bayesian inference and predictive performance of the soil respiration models such that the following points are true: (i) the level of complexity of the best model is generally justified by the cross-validation results for different data models; (ii) not accounting for heteroscedasticity and autocorrelation might not necessarily result in biased parameter estimates or predictions, but will definitely underestimate uncertainty; (iii) using a non-Gaussian data model improves the parameter estimates and the predictive performance; and (iv) accounting for autocorrelation only or joint inversion of correlation and heteroscedasticity can be problematic and requires special treatment. Although the conclusions of this study are empirical, the analysis may provide insights for selecting appropriate data models for soil respiration modeling.

show abstract

Section: Predictive Performance With Total Uncertaintysupporting

confidence: 91%

Section: Data Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Bayesian inference and predictive performance of soil respiration models in the presence of model discrepancy

Elshall

Niu

et al. 2019

Geosci. Model Dev.

Self Cite

View full text Add to dashboard Cite

show abstract

“…To account for the trade-off between the three metrics, (Elshall et al, 2018) defined relative model score (RMS) that simultaneously measure all the three criteria. Scoring rules are commonly used in hydrology to assess predictive performance (e.g.…”

Section: Metrics For Evaluating Predictive Performancementioning

confidence: 99%

Bayesian Inference and Predictive Performance of Soil Respiration Models in the Presence of Model Discrepancy

Ye¹,

Elshall²

2018

Preprint

Self Cite

View full text Add to dashboard Cite

Abstract. Bayesian inference of microbial soil respiration models is often based on the assumptions that the residuals are independent (i.e. no temporal or spatial correlation), identically distributed (i.e. Gaussian noise) and with constant variance (i.e. homoscedastic). In the presence of model discrepancy, since no model is perfect, this study shows that these assumptions are generally invalid in soil respiration modeling such that residuals have high temporal correlation, an increasing variance with increasing magnitude of CO2 efflux, and non-Gaussian distribution. Relaxing these three assumptions stepwise results in eight data models. Data models are the basis of formulating likelihood functions of Bayesian inference. This study presents a systematic and comprehensive investigation of the impacts data model selection on Bayesian inference and predictive performance. We use three mechanistic soil respiration models with different levels of model fidelity (i.e. model discrepancy) with respect to number of carbon pools and explicit representations of soil moisture controls on carbon degradation, and accordingly have different levels of model complexity with respect to the number of model parameters. The study shows data models have substantial impacts on Bayesian inference and predictive performance of the soil respiration models such that: (i) the level of complexity of the best model is generally justified by the cross-validation results for different data models; (ii) not accounting for heteroscedasticity and autocorrelation might not necessarily result in biased parameter estimates or predictions, but will definitely underestimate uncertainty; (iii) using a non-Gaussian data model improves the parameter estimates and the predictive performance; and (iv) separate accounting for autocorrelation or joint inversion of correlation and heteroscedasticity can be problematic and requires special treatment. Although the conclusions of this study are empirical, the analysis may provide insights for selecting appropriate data models for soil respiration models.

show abstract

“…Thus, the scope of this work is the accurate estimation of BME given different Monte Carlo estimators. Studying the impact of prior distribution [72], likelihood function [73,74] , model fidelity [12,14,15,47] , prior model probability [13], input data [75], and observation data [2,12] on the magnitude of BME is beyond the scope of this work. The readers are refered to a recent review article that discusses multi-model analysis in hydrology using Bayesian techiques [76].…”

Section: Introductionmentioning

confidence: 99%

Making Steppingstones out of Stumbling Blocks: A Bayesian Model Evidence Estimator with Application to Groundwater Transport Model Selection

Elshall

2019

Water

View full text Add to dashboard Cite

Bayesian model evidence (BME) is a measure of the average fit of a model to observation data given all the parameter values that the model can assume. By accounting for the trade-off between goodness-of-fit and model complexity, BME is used for model selection and model averaging purposes. For strict Bayesian computation, the theoretically unbiased Monte Carlo based numerical estimators are preferred over semi-analytical solutions. This study examines five BME numerical estimators and asks how accurate estimation of the BME is important for penalizing model complexity. The limiting cases for numerical BME estimators are the prior sampling arithmetic mean estimator (AM) and the posterior sampling harmonic mean (HM) estimator, which are straightforward to implement, yet they result in underestimation and overestimation, respectively. We also consider the path sampling methods of thermodynamic integration (TI) and steppingstone sampling (SS) that sample multiple intermediate distributions that link the prior and the posterior. Although TI and SS are theoretically unbiased estimators, they could have a bias in practice arising from numerical implementation. For example, sampling errors of some intermediate distributions can introduce bias. We propose a variant of SS, namely the multiple one-steppingstone sampling (MOSS) that is less sensitive to sampling errors. We evaluate these five estimators using a groundwater transport model selection problem. SS and MOSS give the least biased BME estimation at an efficient computational cost. If the estimated BME has a bias that covariates with the true BME, this would not be a problem because we are interested in BME ratios and not their absolute values. On the contrary, the results show that BME estimation bias can be a function of model complexity. Thus, biased BME estimation results in inaccurate penalization of more complex models, which changes the model ranking. This was less observed with SS and MOSS as with the three other methods.

show abstract

Relative model score: a scoring rule for evaluating ensemble simulations with application to microbial soil respiration modeling

Cited by 8 publications

References 48 publications

Bayesian inference and predictive performance of soil respiration models in the presence of model discrepancy

Bayesian inference and predictive performance of soil respiration models in the presence of model discrepancy

Bayesian Inference and Predictive Performance of Soil Respiration Models in the Presence of Model Discrepancy

Making Steppingstones out of Stumbling Blocks: A Bayesian Model Evidence Estimator with Application to Groundwater Transport Model Selection

Contact Info

Product

Resources

About