Counting probability distributions: Differential geometry and model selection

Abstract

A central problem in science is deciding among competing explanations of data containing random errors. We argue that assessing the ''complexity'' of explanations is essential to a theoretically wellfounded model selection procedure. We formulate model complexity in terms of the geometry of the space of probability distributions. Geometric complexity provides a clear intuitive understanding of several extant notions of model complexity. This approach allows us to reconceptualize the model selection problem as one of counting explanations that lie close to the ''truth.'' We demonstrate the usefulness of the approach by applying it to the recovery of models in psychophysics.H ow does one decide among competing explanations of data, given limited observations? This problem of model selection is at the core of progress in science. It is particularly vexing in the statistical sciences, where sources of error are diverse and hard to control. For example, in psychology experiments, the participants are themselves a serious source of uncontrolled random variation. Choosing between candidate models that purport to describe underlying regularities about human behavior given noisy data is correspondingly problematic. Over the decades, scientists have used various statistical tools to select among alternative models of data but have lacked a clear theoretical framework for understanding model selection. The purpose of this article is to alert scientists to the importance of accounting for complexity when choosing among models and to provide a geometric formulation of complexity. Not only does a geometric approach recast model selection in a more intuitive and meaningful light, but it also provides insight into the relations among conventional statistical techniques and the inherent tradeoffs between model performance and complexity.

Toward a method of selecting among computational models of cognition.

“…A more technically rigorous presentation of the topic can be found in Myung, Balasubramanian, and Pitt (2000).…”

Assessing the distinguishability of models and the informativeness of data

“…One form of GMA that we have studied in prior work is model complexity (Myung, Balasubramanian & Pitt, 2000;Myung & Pitt, 1997;Pitt, Myung & Zhang, 2002). It is concerned with assessing the inherent flexibility of a model in fitting data.…”

Extending the ALCOVE model of category learning to featural stimulus domains

“…Indeed, many quantitative measures of model complexity, such as the Laplacian approximation (see Kass & Raftery, 1995, p. 777), explicitly measure the robustness of a model's fit to the data across the region of the parameter space surrounding the best-fitting parameter values. Accordingly, one way to address the model complexity issue would be to evaluate ALCOVE against the human data by using a measure that incorporates both data fit and model complexity components, such as those described by Kass and Raftery (1995) or Myung, Balasubramanian, and Pitt (2000). 1 An alternative approach that effectively sidesteps the detailed consideration of data fit and complexity is to evaluate a model in terms of its ability to capture fundamental qualitative features of the constraining data.…”