Failure analysis of parameter-induced simulation crashes in climate models

Abstract

Abstract. Simulations using IPCC (Intergovernmental Panel on Climate Change)-class climate models are subject to fail or crash for a variety of reasons. Quantitative analysis of the failures can yield useful insights to better understand and improve the models. During the course of uncertainty quantification (UQ) ensemble simulations to assess the effects of ocean model parameter uncertainties on climate simulations, we experienced a series of simulation crashes within the Parallel Ocean Program (POP2) component of the Community Climate System Model (CCSM4). About 8.5 % of our CCSM4 simulations failed for numerical reasons at combinations of POP2 parameter values. We applied support vector machine (SVM) classification from machine learning to quantify and predict the probability of failure as a function of the values of 18 POP2 parameters. A committee of SVM classifiers readily predicted model failures in an independent validation ensemble, as assessed by the area under the receiver operating characteristic (ROC) curve metric (AUC > 0.96). The causes of the simulation failures were determined through a global sensitivity analysis. Combinations of 8 parameters related to ocean mixing and viscosity from three different POP2 parameterizations were the major sources of the failures. This information can be used to improve POP2 and CCSM4 by incorporating correlations across the relevant parameters. Our method can also be used to quantify, predict, and understand simulation crashes in other complex geoscientific models.

The parametric sensitivity of CAM5's MJO

“…Although other methods including linear basis functions, polynomial chaos expansions [Lucas et al, 2013], and support vector machines [Bishop, 2007] were tried, random forests [Breiman, 2001] were selected as they yielded the best fits. Briefly, a random forest is an ensemble of randomized decision trees.…”

Application of all-relevant feature selection for the failure analysis of parameter-induced simulation crashes in climate models

“…Hence we decided to check whether a more elaborate method for analysis could decrease this number further. We have observed potential weak points of the analysis performed by Lucas et al (2013); namely, they had not fully take into account that the apparent importance of a variable for classification may be in fact the result of a spurious fluctuation. The problem is most acute when a sample used for a machine-learning algorithm is small.…”

“…It is by no means a trivial task since it involves the parameterisation of many processes that are not directly solved within the model. It has been shown by Lucas et al (2013) that certain combinations of these parameters lead to failure of a model, despite each individual parameter having a reasonable value. Authors of this study performed 540 simulations with randomly varied combinations of 18 parameters of the Parallel Ocean Program (POP2) (Smith et al, 2010) module in the Community Climate System Model Version 4 (CCSM4) (UCAR, 2010).…”

“…They reflect the best knowledge on the physical components of the climate; nevertheless, they contain several parameters, which are too weakly constrained by observations, and can potentially lead to a simulation crashing. Recently a study by Lucas et al (2013) has shown that machine learning methods can be used for predicting which combinations of parameters can lead to the simulation crashing and hence which processes described by these parameters need refined analyses. In the current study we reanalyse the data set used in this research using different methodology.…”

On the effect of model parameters on forecast objects

“…Some SA work is performed for assessing how model parameters impact the model itself, not as a means to some other goal. For example, Lucas et al (2013) use a global SA method to explore the effect of model pa-rameters on the probability of model crashes. By contrast, sometimes SA is performed as an intermediate step to another goal, such as the calibration of the model (Safta et al, 2015;Hacker et al, 2011;Laine et al, 2012;Ollinaho et al, 2014).…”