2016 International Land Model Benchmarking (ILAMB) Workshop Report

Abstract

The ILAMB version 1 (v1) and ILAMB version 2 (v2) benchmarking systems compare model results with best-available observational data products, focusing on atmospheric CO 2 , surface fluxes, hydrology, soil carbon and nutrient biogeochemistry, ecosystem processes and states, and vegetation dynamics. ILAMBv2 is expected to become an integral part of the workflow for model frameworks, including the Accelerated Climate Modeling for Energy (ACME) model and the Community Earth System Model (CESM). Moreover, ILAMBv2 will contribute model analysis and evaluation capabilities to phase 6 of the Coupled Model Intercomparison Project (CMIP6) and future model and model-data intercomparison projects. Benchmarking Challenges and PrioritiesA variety of statistical approaches have been adopted to evaluate model accuracy through comparison with observations, including calculations of bias, root-mean-square error (RMSE), phase, amplitude, spatial distribution, Taylor diagrams and scores, functional relationship metrics, and perturbation and sensitivity tests. While many of these statistical measures are not independent, each provides slightly different information about contemporary model performance with respect to observational data and about implications for future projections from ESMs.However, developing metrics that make appropriate use of observational data remains a scientific challenge because of the spatial and temporal mismatch between models and measurements, poorly characterized uncertainties in observationally constrained data products, biases in reanalysis and forcing data, model simplifications, and structural and parametric uncertainties. A variety of benchmarking challenges and opportunities emerged from workshop breakout group meeting reports. Common themes included the following: › Need for collocated measurements, particularly around a core set of AmeriFlux and FLUXNET sites with a sustained record of observations for repeated model testing › Lack of quantified uncertainty information for observational data › Utility of functional response metrics and variable-to-variable comparisons › Value of metrics for future projections based on emergent constraints v › Unrealized opportunities for global observational data sets based on satellite remote sensing synthesized with ancillary databases, using new algorithms › Importance of applying statistical and machine learning methods to upscaling sparse measurements from sites to regions to the globe › Need for process-level benchmarks and metrics for extreme events › Opportunities for collaboration with earth system model developers (e.g., ACME, CESM, and others)Opportunities for collaboration with important field and laboratory experiments and monitoring activities, including AmeriFlux and FLUXNET, Integrated Carbon Observation System (ICOS), Next Generation Ecosystem Experiments (NGEE) Arctic, Arctic-Boreal Vulnerability Experiment (ABoVE), Spruce and Peatland Responses Under Climatic and Environmental Change (SPRUCE) project, Critical Zone Observatories (CZOs), Long-Te...

The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation

“…Model developers and software engineers require a systematic means for evaluating changes in model results to ensure that developments improve the scientific performance of target process representations while not adversely affecting results in other, possibly less familiar, parts of the model. To advance understanding and predictability of terrestrial biogeochemical processes and their interactions with hydrology and climate under conditions of increasing atmospheric carbon dioxide, rigorous analysis methods, employing best available observational data, are required to objectively assess and constrain model predictions, inform model development, and identify needed measurements and field experiments (Hoffman et al, 2017). Building upon past model evaluation work (Randerson et al, 2009), we developed an extensible model benchmarking package in support of the goals of the International Land Model Benchmarking (ILAMB; https://www.ilamb.org/) activity.…”

Evaluating Uncertainties in Marine Biogeochemical Models: Benchmarking Aerosol Precursors

“…However, we go further by noting that extensive validation must be supported if the new generation of models is to be successful. An automated framework is proposed and described based on the ILAMB package as a foundation and example [10]. The marine version is now known as IOMB, which stands for International Ocean Model Benchmarking.…”

“…However, these features are not fully considered in the present analysis due to limitations of the observational datasets. Other potential global analyses that could be performed with IOMB/ILAMB can be found in the documentation for the parent package [10].…”

“…This is with the intent to meet intensifying validation needs for biogeochemical modelers and measurement scientists within the ocean community. The tool is derived from the International Land Model Benchmarking (ILAMB), which is a closely related model validation project that is focused on terrestrial ecosystem dynamics [10]. Our fundamental strategy is to automate the confrontation of ocean physics, biology, and chemistry simulations with a wealth of observational datasets.…”

“…In the present work, we will use the well-known CLAW hypothesis (originally proposed by Robert Jay Charlson, James Lovelock, Meinrat Andreae and Stephen G. Warren, and the acronym comes from the first letter of their surnames) [9] exhibited with DMS as a primary example of how benchmarking can be accomplished. We will show that the methodology developed for the International Land Model Benchmarking (ILAMB) project [10], some of which are currently incorporated into the International Ocean Model Benchmarking (IOMB) package, can be adapted to validate not only traditional carbon cycling from the surface ocean to deep sea, but also aerosol precursor generation, along with their distributions and effects.…”