A tutorial on adaptive design optimization

Abstract

Experimentation is ubiquitous in the field of psychology and fundamental to the advancement of its science, and one of the biggest challenges for researchers is designing experiments that can conclusively discriminate the theoretical hypotheses or models under investigation. The recognition of this challenge has led to the development of sophisticated statistical methods that aid in the design of experiments and that are within the reach of everyday experimental scientists. This tutorial paper introduces the reader to an implementable experimentation methodology, dubbed Adaptive Design Optimization, that can help scientists to conduct “smart” experiments that are maximally informative and highly efficient, which in turn should accelerate scientific discovery in psychology and beyond.

On the functional form of temporal discounting: An optimized adaptive test

“…Therefore, using informative priors is helpful for speeding up convergence, but not essential to implementing ADO (Myung et al, 2013). Most importantly, the priors should be defined in such a way that when data are generated from any one of the models under consideration, that model can be recovered correctly based on its posterior probability.…”

“…The present study demonstrates and implements a Bayesian inference method for discriminating among models of temporal discounting using Adaptive Design Optimization (Cavagnaro et al, 2010; Myung et al, 2013, ADO), which integrates likelihood-based data-modeling with adaptive experimental designs to maximize the efficiency and informativeness of an experiment. In an ADO experiment, stimuli are tailored to each participant by updating model and parameter estimates in real time as data are collected, and using the latest estimates to select stimuli that maximize the expected information gain about the models under consideration.…”

“…The main difference between the two methods is that DEEP was originally formulated for estimating the parameter(s) of an assumed model of preference, whereas ADO was originally formulated for selecting among a group of candidate models (Cavagnaro et al, 2010). However, in the Bayesian framework, model selection and parameter estimation are essentially the same problem, and the natural extension of ADO to the problem of estimating the parameter of an assumed model was made explicit by Myung et al (2013).…”

“…Thus, U ( d ) can be viewed as the “expected utility” of the design d , where the expectation is taken over possible models, parameters and experiment outcomes. Myung et al (see 2013, for details). …”

Optimal Design for Estimation in Diffusion Processes from First Hitting Times

“…Using MI for the selection of maximally informative experiments has been advocated by several recent lines of research, for example, in experimental psychology [5,19], computational neuroscience [22,21,15], and quantum physics [9]. An alternative approach is to maximize the expected Fisher information of the experiment as in [10].…”