The ACCENT-protocol: a framework for benchmarking and model evaluation

Abstract

We summarise results from a workshop on "Model Benchmarking and Quality Assurance" of the EU-Network of Excellence ACCENT, including results from other activities (e.g. COST Action 732) and publications. A formalised evaluation protocol is presented, i.e. a generic formalism describing the procedure how to perform a model evaluation. This includes eight steps and examples from global model applications are given for illustration. The first and important step is concerning the purpose of the model application, i.e. the addressed underlying scientific or political question. We give examples to demonstrate that there is no model evaluation per se, i.e. without a focused purpose. Model evaluation is testing, whether a model is fit for its purpose. The following steps are deduced from the purpose and include model requirements, input data, key processes and quantities, benchmark data, quality indicators, sensitivities, as well as benchmarking and grading. We define "benchmarking" as the process of comparing the model output against either observational data or high fidelity model data, i.e. benchmark data. Special focus is given to the uncertainties, e.g. in observational data, which have the potential to lead to wrong conclusions in the model evaluation if not considered carefully

Characterizing sampling biases in the trace gas climatologies of the SPARC Data Initiative

“…Stratospheric trace gas observations are often used to produce monthly zonal mean data sets, or climatologies [e.g., von Clarmann et al ., ; Grooß and Russell , ; Hassler et al ., ; Jones et al ., ; Randel and Wu , ]. Monthly zonal mean data products are typically used as prescribed forcing for models [e.g., Cionni et al ., ] and are also useful for comparison with similarly averaged chemistry climate model output [e.g., Grewe et al ., ; SPARC CCMVal , ].…”

Shyft v4.8: a framework for uncertainty assessment and distributed hydrologic modeling for operational hydrology

“…1.2 paragraph 2. Another challenge comes from the well-known problem of standardized hydrological benchmarking discussed in Abramowitz (2012) and Grewe et al (2012), as the definition of efficient (from a hydrological simulation point of view) software depends on the proper application of metrics and proper justification at each step.…”

Aircraft routing with minimal climate impact: the REACT4C climate cost function modelling approach (V1.0)

“…For example, modelling studies often simulate a wide range of optical thickness of contrails, which, for example, cannot be detected from satellite (Marquart et al, 2003;Kärcher et al, 2009). For all these reasons, the comparison of contrail properties such as ice water content and optical thickness for a limited number of simulations can just be seen as a sanity check rather than a hard benchmark test (Grewe et al, 2012b). Figure 10a shows observed and simulated ice water content in contrails as a cumulative probability density function.…”