Toward a method of selecting among computational models of cognition.

Abstract

The question of how one should decide among competing explanations of data is at the heart of the scientific enterprise. Computational models of cognition are increasingly being advanced as explanations of behavior. The success of this line of inquiry depends on the development of robust methods to guide the evaluation and selection of these models. This article introduces a method of selecting among mathematical models of cognition known as minimum description length, which provides an intuitive and theoretically well-grounded understanding of why one model should be chosen. A central but elusive concept in model selection, complexity, can also be derived with the method. The adequacy of the method is demonstrated in 3 areas of cognitive modeling: psychophysics, information integration, and categorization.How should one choose among competing theoretical explanations of data? This question is at the heart of the scientific enterprise, regardless of whether verbal models are being tested in an experimental setting or computational models are being evaluated in simulations. A number of criteria have been proposed to assist in this endeavor, summarized nicely by Jacobs and Grainger (1994). They include (a) plausibility (are the assumptions of the model biologically and psychologically plausible?); (b) explanatory adequacy (is the theoretical explanation reasonable and consistent with what is known?); (c) interpretability (do the model and its parts-e.g., parameters-make sense? are they understandable?); (d) descriptive adequacy (does the model provide a good description of the observed data?); (e) generalizability (does the model predict well the characteristics of data that will be observed in the future?); and (f) complexity (does the model capture the phenomenon in the least complex-i.e., simplest-possible manner?).The relative importance of these criteria may vary with the types of models being compared. For example, verbal models are likely to be scrutinized on the first three criteria just as much as the last three to thoroughly evaluate the soundness of the models and their assumptions. Computational models, on the other hand, may have already satisfied the first three criteria to a certain level of acceptability earlier in their evolution, leaving the last three criteria to be the primary ones on which they are evaluated. This emphasis on the latter three can be seen in the development of quantitative methods designed to compare models on these criteria. These methods are the topic of this article.In the last two decades, interest in mathematical models of cognition and other psychological processes has increased tremendously. We view this as a positive sign for the discipline, for it suggests that this method of inquiry holds considerable promise. Among other things, a mathematical instantiation of a theory provides a test bed in which researchers can examine the detailed interactions of a model's parts with a level of precision that is not possible with verbal models. Furthermore, through systematic eval...

Comparing categorization models.

“…As noted previously, GCM is more flexible than GRT for this design, and flexibility has become a topical issue in model selection (e.g., Kass & Raftery, 1995;Myung & Pitt, 1997;Pitt, Myung, & Zhang, 2003). Although this difference in flexibility makes model selection complicated in cases in which both GRT and GCM qualitatively account for the data, it is of little concern for cases in which one model clearly fails.…”

Assessing the distinguishability of models and the informativeness of data

“…This problem can be remedied by calculating the complexities of the models and then shifting the decision criterion upward or downward to correct for the difference (note that complexity is a property of the model and the experimental design). After calculating the "geometric complexity" measure discussed by Pitt et al (2002), we found that the criterion should be shifted towards the FLMP distribution 7 by 1.88 (the adjusted criterion is indicated in the plots by the broken line). Although the correction is small, the improvement is dramatic because the LIM distribution is so peaked: The LIM error rate falls to 3.5% whereas the FLMP error rate rises to only 0.3%.…”

“…One form of GMA that we have studied in prior work is model complexity (Myung, Balasubramanian & Pitt, 2000;Myung & Pitt, 1997;Pitt, Myung & Zhang, 2002). It is concerned with assessing the inherent flexibility of a model in fitting data.…”

Testing adaptive toolbox models: A Bayesian hierarchical approach.

“…Another advantage of the BIC is that it is relatively straightforward to compute; the BIC is given by BIC ϭ Ϫ2 ϫ ln L ϩ k ϫ ln n, where ln L is the log maximum likelihood, k is the number of free parameters, and n is the sample size. This simplicity, however, comes at a cost: The BIC ignores interactions between parameters and is blind to differences in the parameters' functional form (Karabatsos, 2006;Myung & Pitt, 1997, 2009Pitt, Myung, & Zhang, 2002).…”