The Fast and the Robust: Trade‐Offs Between Optimization Robustness and Cost in the Calibration of Environmental Models

Abstract

Environmental modelers using optimization algorithms for model calibration face an ambivalent choice. Some algorithms, for example, Newton‐type methods, are fast but struggle to consistently find global parameter optima; other algorithms, for example, evolutionary methods, boast better global convergence but at much higher cost (e.g., requiring more objective function calls). Trade‐offs between accuracy/robustness versus cost are ubiquitous in numerical computation, yet environmental modeling studies have lacked a systematic framework for quantifying these trade‐offs. This study develops a framework for benchmarking stochastic optimization algorithms in the context of environmental model calibration, where multiple algorithm invocations are typically necessary. We define reliability as the probability of finding the desired (global or tolerable) optimum from random initial points and estimate the number of invocations to find the desired optimum with prescribed confidence (here 95%). A robust algorithm should achieve consistently high reliability across many problems. A characteristic efficiency metric for algorithm benchmarking is defined as the total cost (objective function calls over multiple invocations) to find the desired optimum with prescribed confidence. This approach avoids the pitfalls of existing approaches that compare costs without controlling the confidence in algorithm success. A case study illustrates the framework by benchmarking the Levenberg‐Marquardt and Shuffled Complex Evolution (SCE) algorithms over three catchments and four hydrological models. In 8 of 12 scenarios, Levenberg‐Marquardt is more efficient than SCE—by sacrificing some of its speed advantage to match SCE reliability through more invocations. The proposed framework is easy to apply and can help guide algorithm selection in environmental model calibration.

A Robust Gauss‐Newton Algorithm for the Optimization of Hydrological Models: Benchmarking Against Industry‐Standard Algorithms

“…Here, robustness is defined as the ability to reliably find a desired optimum (global or tolerable, see below) irrespective of the initial search point and consistently across multiple modeling scenarios (optimization problems). Efficiency is defined as the ability to find the desired optimum at low computational cost (Kavetski et al, ; Qin et al, ).…”

A Robust Gauss‐Newton Algorithm for the Optimization of Hydrological Models: From Standard Gauss‐Newton to Robust Gauss‐Newton

“…The reliability metrics in this study are derived from a general reliability metric, , defined as the fraction of invocations that find a solution (parameter set) with hydrological model performance within a tolerance τ ψ of the best‐known hydrological model performance metric (Kavetski et al, ), where ψ = { ψ m , m = 1, ⋯, M } are the performance metric values at the termination points of M algorithm invocations, and freq v is the frequency function, defined as the fraction of true values within its Boolean vector argument v , that is, where ℐ[ v ] is the indicator function (1 if v is true and 0 otherwise) and the subscript m indexes the elements of v . Equation assumes ψ is positive, and higher values indicate better performance.…”

“…Based on definitions of reliability in equations –, we define algorithm robustness as the ability of an algorithm to achieve high reliability consistently across a wide range of modeling scenarios (Kavetski et al, ).…”

“…An efficient optimization algorithm achieves high reliability at a low cost. The efficiency ratio for any two optimization algorithms A and B is given by Kavetski et al () as where the quantity M X , α = roundUp[log(1 − α )/ log (1 − ℛ X )] is the (estimated) number of invocations required by an algorithm to find the desired optimum with confidence α , is the corresponding typical total cost (expressed as the total number of objective function calls), and the subscript X is used to accommodate any definition of desired optima (here, X = G, T for global and tolerable optima, respectively). The function roundUp[ z ] returns the smallest integer greater than or equal to z .…”

“…Desirable attributes of optimization algorithms include (i) high robustness, which we define as the ability to consistently find good parameter sets, and (ii) low computational cost, which is typically defined as requiring as few objective function evaluations (model runs) as possible. Ultimately, an efficient algorithm is one that finds high‐quality optima at low cost (Kavetski et al, ). Attaining these attributes is a major research and operational challenge and is the main subject of this paper.…”