Sloppy-Model Universality Class and the Vandermonde Matrix

Abstract

In a variety of contexts, physicists study complex, nonlinear models with many unknown or tunable parameters to explain experimental data. We explain why such systems so often are sloppy; the system behavior depends only on a few 'stiff' combinations of the parameters and is unchanged as other 'sloppy' parameter combinations vary by orders of magnitude. We contrast examples of sloppy models (from systems biology, variational quantum Monte Carlo, and common data fitting) with systems which are not sloppy (multidimensional linear regression, random matrix ensembles). We observe that the eigenvalue spectra for the sensitivity of sloppy models have a striking, characteristic form, with a density of logarithms of eigenvalues which is roughly constant over a large range. We suggest that the common features of sloppy models indicate that they may belong to a common universality class. In particular, we motivate focusing on a Vandermonde ensemble of multiparameter nonlinear models and show in one limit that they exhibit the universal features of sloppy models.PACS numbers: 02.30. Zz, 87.15.Aa, 87.16.Xa, Systems with many parameters are often sloppy. For practical purposes their behavior depends only on a few stiffly constrained combinations of the parameters; other directions in parameter space can change by orders of magnitude without significantly changing the behavior. Given a suitable cost C(p) measuring the change in system behavior as the parameters p vary from their original values p (0) (e.g., a sum of squared residuals), the stiff and sloppy directions can be quantified as eigenvalues and eigenvectors of the Hessian of the cost:. Figure 1 shows the eigenvalues of the cost Hessian for many different systems; those in (a), (b), (c), (d) and (h) are all sloppy. The sensitivity of model behavior to changes along an eigenvector is given by the square root of the eigenvalue-therefore the range in eigenvalues of roughly one million for the sloppy models means that one must change parameters along the sloppiest eigendirection a thousand times more than along the stiffest eigendirection in order to change the behavior by the same amount. Although anharmonic effects rapidly become important along sloppy eigendirections, a principal component analysis of a Monte-Carlo sampling of lowcost states has a similar spectrum of eigenvalues [1]; the sloppy eigendirections become curved sloppy manifolds in parameter space. Similar sloppy behavior has been demonstrated in fourteen systems biology models taken from the literature [2,3], and in three multiparameter interatomic potentials fit to electronic structure data [4]. In these disparate models we see a common, peculiar behavior: the nth stiffest eigendirection is more important than the (n+1)th by a roughly constant factor, giving a total range of eigenvalues of typically over a million for any model with more than eight parameters. We call systems exhibiting these characteristic features sloppy models.This sloppiness has a number of important implications. In estimating predic...

Optimal execution with non-linear transient market impact

“…In general, the presence of many minima could reflect numerical error or a landscape that truly features multiple minima. The first case occurs frequently when the landscape is sloppy [18,51] global minimum is located inside a long narrow and parabolic shaped valley. Sloppiness can imply difficulties in finding the global minimum, because there is a manifold where the cost function is almost flat.…”

Cause and cure of sloppiness in ordinary differential equation models

“…in [6,14]. Thus, it could be shown that it is possible by optimal experimental design methods to diminish the sloppiness effect to a minimum.…”

“…The same effect has been independently observed in [12,13] for circadian clock systems described by ODEs and it has been * christian.toensing@fdm.uni-freiburg.de termed as rigidity of the mapping from the parameters to the solutions, where it occupies a space of lower dimension. Moreover, the effect has been shown to emerge naturally in special classes of systems [14] and a broad eigenvalue spectrum of the Hessian or its analogs is also observed in systems in a general physical context [15,16]. The impact of the existence of sloppy directions in the parameter space on optimization has been discussed in [17,18].…”

Analysis of Mechanistic Pathway Models in Drug Discovery: p38 Pathway

“…The pathway model here presented belongs to a class of so-called "sloppy" models (30) (see the Appendix), characterized by having a significant number of poorly known parameters with widely varying sensitivities. As an alternative to our approach, one can utilize methods to reduce the dimensionality of parameter space, transforming into a new set of dimensions (reflecting parameter combinations) that are organized by their overall sensitivities (30)(31)(32). In principle, such an approach can help optimize computations in parameter estimation and identify dependencies in parameter variation.…”

“…Note that the eigenvalues are more or less uniformly distributed over approximately 6 orders of magnitude (with occasionally a few exceptionally sloppy eigenvectors); this behavior is characteristic of other sloppy systems (30)(31)(32)(33) …”