Using Monte Carlo techniques to judge model prediction accuracy: Validation of the pesticide root zone model 3.12

Abstract

Individuals from the Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) Environmental Model Validation Task Force (FEMVTF) Statistics Committee periodically met to discuss the mechanism for conducting an uncertainty analysis of Version 3.12 of the pesticide root zone model (PRZM 3.12) and to identify those model input parameters that most contribute to model prediction error. This activity was part of a larger project evaluating PRZM 3.12. The goal of the uncertainty analysis was to compare site-specific model predictions and field measurements using the variability in each as a basis of comparison. Monte Carlo analysis was used as an integral tool for judging the model's ability to predict accurately. The model was judged on how well it predicts measured values, taking into account the uncertainty in the model predictions. Monte Carlo analysis provides the tool for inferring model prediction uncertainty. We argue that this is a fairer test of the model than a simple one-to-one comparison between predictions and measurements. Because models are known to be imperfect predictors prior to running the model, the inaccuracy in model predictions should be considered when models are judged for their predictive ability. Otherwise, complex models can easily fail a validation test. Few complex models, such as PRZM 3.12, would pass a typical model validation exercise. This paper describes the approaches to the validation of PRZM 3.12 used by the committee and discusses issues in sampling distribution selection and appropriate statistics for interpreting the model validation results.

Analysis of parameter uncertainty and sensitivity in PCPF-1 modeling for predicting concentrations of rice herbicides

“…[38][39][40] e choice of the shapes and types of distributions re ects the magnitude, range, and interpretation of each parameter. erefore, as recommended in previous reports, 10,30) the types of PDFs for the selected parameters remained unchanged among scenarios, while the means and the variances of PDFs were scenario (herbicides)-dependent ( Table 2). In general, the means, variances and ranges of an input have more in uence on the output uncertainty than the form of the distribution.…”

“…For modeling the pesticide fate in environment, incorporating a probabilistic approach such as the Monte Carlo (MC) technique 10,19,32) is useful for the evaluation of modeling uncertainties associated with input parameters as well as uncertainties due to the model itself. Since the MC technique provides insight into the model behavior, it was helpful to estimate the uncertain model input and increase the reliability of the model.…”

“…In general, modeling uncertainty was shown to lead to a lack of con dence of the model performance. 9,10) e uncertainty analysis is hence essential to provide assistance to users for model parameterization, calibration, and probabilistic modeling. [11][12][13][14] In the context of hydrologic models, general uncertainty was classi ed into three types [15][16][17] : the inherent variability in natural processes, model uncertainty, and parameter uncertainty.…”

“…18) However, the analyses in knowledge uncertainty for PCPF-1 have not been conducted yet, while other major pesticide fate and transport models, such as MACRO, RICEWQ, PEARL, and PRZM, 10,19,20) already provided the knowledge of the probabilistic ranges of the potential outputs of the given modeling system. 21) A sensitivity analysis on the other hand, is di erent from an uncertainty analysis as it attempts to determine the change in model output values that results from modest changes in model input values.…”

Integrated assessment of climate change impact on surface runoff contamination by pesticides

“…It does not simulate the transport of pesticide through the broader environment (e.g., streams, basin outlets). PRZM has been used extensively and several studies have shown that the model produces reasonable estimates (e.g., Acc e p ted P r e p r i nt Carbone et al 2002;Singh and Jones 2002;Warren-Hicks et al 2002;Farenhorst et al 2009) while requiring relatively low computational time (McQueen et al 2007). …”