A Review of Methodology for Evaluating the Performance of Atmospheric Transport and Dispersion Models and Suggested Protocol for Providing More Informative Results

Abstract

Many models exist for predicting the atmospheric transport and dispersion of material following its release into the atmosphere. The purpose of these models may be to support air quality assessments and/or to predict the hazard resulting from releases of harmful materials to inform emergency response actions. In either case it is essential that the user understands the level of predictive accuracy that might be expected. However, contrary to expectation, this is not easily determined from published comparisons of model predictions against data from dispersion experiments. The paper presents and reviews the methods adopted and issues involved in comparing the predictive performance of atmospheric transport and dispersion models to experimental data, by reference to a number of experimental data sets and comparison results. It then presents an approach which is designed to make the performance of atmospheric dispersion models more transparent, through clearly defining the basis on which the comparison is made, and comparing the performance of the chosen model to that of a reference model. Such an approach establishes a clear baseline against which the accuracy of models can be evaluated and the performance benefits of more sophisticated approaches quantified. The use of a simple analytic reference model applicable to continuous ground level releases in open terrain and urban areas is shown as a proof-of-principle.

Skill-Testing Chemical Transport Models across Contrasting Atmospheric Mixing States Using Radon-222

“…It is well established that model performance is not independent of the atmospheric mixing state (or "stability") [12,[72][73][74]. In Figure 2b, for example, across the diurnal cycle agreement between simulated and observed temperatures was usually the best when the ABL was the deepest and most uniformly mixed.…”

“…As much as it is true for CTMs themselves, the ways in which their performance is evaluated should also be subjected to periodic review. Mismatches in spatial and vertical scales between observations and model grid-cell dimensions, along with spatial and temporal heterogeneity of urban pollution sources, can lead to vastly-different assessments of model skill depending on the timescale or particular approach adopted for the evaluation [10][11][12].…”

“…The aim of this investigation is to introduce a new approach for skill-testing CTMs regarding: (i) reproducing meteorological conditions, (ii) transporting and mixing primary emissions in the atmospheric boundary layer (ABL), and (iii) forming secondary gaseous and aerosol products. Since understanding of the physical processes associated with transport and mixing under fair-weather turbulent daytime conditions is more advanced than processes dominating near quiescent nocturnal conditions near the surface [12][13][14][15][16], model skill will also be dependent on atmospheric mixing state. Furthermore, the formation of secondary pollutants is also dependent on factors related to the atmospheric mixing state (e.g., mixing volume, temperature, sunlight intensity, humidity, etc.…”

Street-in-Grid modeling of gas-phase pollutants in Paris city

“…The criteria applied to evaluate the comparisons are the statistics detailed in Hanna and Chang (2012) and Herring and Huq (2018): -0.3 < FB < 0.3; 0.7 < MG < 1.3; NMSE < 3; VG < 1.6; FAC2 ≥ 0.5; NAD < 0.3. Hanna and Chang (2012) and Herring and Huq (2018) also defined a less strict criteria to be applied to urban areas: -0.67 < FB < 0.67; NMSE < 6; FAC2 ≥ 0.3; NAD < 0.5. The definitions of these statistics are given in Annexe A1.…”

Skill-Testing Chemical Transport Models across Contrasting Atmospheric Mixing States Using Radon-222

“…The aim of this investigation is to introduce a new approach for skill-testing CTMs regarding: (i) reproducing meteorological conditions, (ii) transporting and mixing primary emissions in the atmospheric boundary layer (ABL), and (iii) forming secondary gaseous and aerosol products. Since understanding of the physical processes associated with transport and mixing under fair-weather turbulent daytime conditions is more advanced than processes dominating near quiescent nocturnal conditions near the surface [12][13][14][15][16], model skill will also be dependent on atmospheric mixing state.…”