Adaptive Design Optimization: A Mutual Information-Based Approach to Model Discrimination in Cognitive Science

Abstract

Discriminating among competing statistical models is a pressing issue for many experimentalists in the field of cognitive science. Resolving this issue begins with designing maximally informative experiments. To this end, the problem to be solved in adaptive design optimization is identifying experimental designs under which one can infer the underlying model in the fewest possible steps. When the models under consideration are nonlinear, as is often the case in cognitive science, this problem can be impossible to solve analytically without simplifying assumptions. However, as we show in this paper, a full solution can be found numerically with the help of a Bayesian computational trick derived from the statistics literature, which recasts the problem as a probability density simulation in which the optimal design is the mode of the density. We use a utility function based on mutual information, and give three intuitive interpretations of the utility function in terms of Bayesian posterior estimates. As a proof of concept, we offer a simple example application to an experiment on memory retention.

Rich stimulus sampling for between-subjects designs improves model selection

“…However, there is very little coverage of the topic in the behavioral sciences, as evidenced by the small number of articles in psychological journals on choosing the best values along a continuum in the pursuit of either of these modeling goals. Although Myung, Pitt, and colleagues (Cavagnaro, Myung, Pitt, & Kujala, 2010;Cavagnaro, Pitt, & Myung, 2011;Myung & Pitt, 2009) recently have been actively developing methods of choosing a set of IV values in order to improve selection among known models, there are only a handful of other psychology publications involving general design approaches to optimal parameter estimation (Berger, 1994;Berger, King, & Wong, 2000;Passos & Berger, 2004;Vermeulen, Goos, & Vandebroek, 2008), with each receiving no more than a handful of citations (ranging from 0 to 8 in the Social Science Citation Index). One exception is the common use of adaptive methods in the area of psychophysics, where stimulus levels are dynamically chosen on the basis of unfolding performance in order to estimate the threshold or slope of an ogival psychometric function (Leek, 2001).…”

“…In the absence of extant models, a rich sampling technique may help map out the general shape of the relationship among the IVs and DVs and do so with sufficient accuracy of parameter estimates to provide utility. As models become more formalized later in the development of a scientific subdomain, there will need to be a greater emphasis on the types of formal value selection methods discussed elsewhere (Cavagnaro et al, 2010;Cavagnaro et al, 2011;Myung & Pitt, 2009). But the field may be slow to reach that stage if insufficient sampling of stimulus dimensions persists in the field of experimental psychology.…”

Entropy-Based Experimental Design for Optimal Model Discrimination in the Geosciences

“…Our proposed framework thus combines several advantageous properties of previous approaches: (1) It builds on the rigorous and consistent formulation of entropy-based OD for model choice as used in Cavagnaro et al [58] and Drovandi et al [59]. (2) For geoscientists, this establishes the link between optimal design and the mentality to view models as competing hypotheses.…”

“…Several authors have suggested the use of mutual information to measure the impact of potential future data on model discrimination (e.g., [57][58][59]). While Box and Hill [57] used a lower-order approximation of mutual information for the Box-Hill discrimination function, the recent approaches by Cavagnaro et al [58] and Drovandi et al [59] use a sample-based representation of the involved joint distributions. However, their approaches are limited to sequential design problems.…”

Optimal Design for Estimation in Diffusion Processes from First Hitting Times

“…Using MI for the selection of maximally informative experiments has been advocated by several recent lines of research, for example, in experimental psychology [5,19], computational neuroscience [22,21,15], and quantum physics [9]. An alternative approach is to maximize the expected Fisher information of the experiment as in [10].…”