Acceptance criteria for urban dispersion model evaluation

Abstract

The authors suggested acceptance criteria for rural dispersion models' performance measures in this journal in 2004. The current paper suggests modified values of acceptance criteria for urban applications and tests them with tracer data from four urban field experiments. For the arc-maximum concentrations, the fractional bias should have a magnitude\0.67 (i.e., the relative mean bias is less than a factor of 2); the normalized mean-square error should be \6 (i.e., the random scatter is less than about 2.4 times the mean); and the fraction of predictions that are within a factor of two of the observations (FAC2) should be [0.3. For all data paired in space, for which a threshold concentration must always be defined, the normalized absolute difference should be \0.50, when the threshold is three times the instrument's limit of quantification (LOQ). An overall criterion is then applied that the total set of acceptance criteria should be satisfied in at least half of the field experiments. These acceptance criteria are applied to evaluations of the US Department of Defense's Joint Effects Model (JEM) with tracer data from US urban field experiments in Salt Lake City (U2000), Oklahoma City (JU2003), and Manhattan (MSG05 and MID05). JEM includes the SCIPUFF dispersion model with the urban canopy option and the urban dispersion model (UDM) option. In each set of evaluations, three or four likely options are tested for meteorological inputs (e.g., a local building top wind speed, the closest National Weather Service airport observations, or outputs from numerical weather prediction models). It is found that, due to large natural variability in the urban data, there is not a large difference between the performance measures for the two model options and the three or four meteorological input options. The more detailed UDM and the state-of-the-art numerical weather models do provide a slight improvement over the other options. The proposed urban dispersion model acceptance criteria are satisfied at over half of the field experiments.

Source Apportionment and Data Assimilation in Urban Air Quality Modelling for NO2: The Lyon Case Study

“…The final estimates are compared to the measured concentrations. To evaluate the quality of the model results, we use six statistical indices: the bias, the fractional bias, the root mean square error (RMSE), the normalized mean square error (NMSE), the correlation coefficient (r) and the factor 2 (FAC2) [79]. The definition of these statistical indices are described in Table 1, where c m is the measured concentration and c p is the predicted concentration.…”

Prediction of Wind Environment and Indoor/Outdoor Relationships for PM2.5 in Different Building–Tree Grouping Patterns

“…It can be seen that there were strong correlations between the simulated and wind tunnel data (R 2 = 0.926 in wall A and R 2 = 0.991 in wall B). According to Hanna and Chang [63], a set of acceptance criteria: −0.3 < FB < 0.3, NMSE < 1.5, 0.5 < FAC2 < 2, NAD < 0.3, and R > 0.8 was recommended for further analysis. All of the metrics are within acceptance ranges, indicating the numerical models were suitable for predicting the airflow and pollutant dispersion within the street canyon (Table 1).…”

A Review of Methodology for Evaluating the Performance of Atmospheric Transport and Dispersion Models and Suggested Protocol for Providing More Informative Results

“…These typically include, the normalized mean square error (NMSE), the fraction of predictions within a factor of 2 of the observations (FAC2), the fractional bias (FB), the geometric mean bias (MG) and the geometric variance (VG) (see Appendix A for definitions). In the absence of any universally agreed performance criteria, when authors wish to compare their results to those of others, they generally cite the criteria for an 'acceptable model' proposed by Chang and Hanna (e.g., [16][17][18][19][20][21][22]), which are summarized in Table 2. The Chang and Hanna criteria were based on their experience in conducting a large number of model evaluation exercises [23].…”