A comparison of adaptive psychometric procedures based on the theory of optimal experiments and Bayesian techniques: Implications for cochlear implant testing

Abstract

Development of more efficient psychometric procedures has been the focus of numerous published studies. Many of these techniques are adaptive, utilizing the outcomes of previous trials to determine the next step in the experiment. Pelli (1987) proposed the ideal psychometric procedure, looking ahead over every possible trial sequence for the length of the experiment to maximize confidence in the outcome. Several psychometric procedures-for example, QUEST (Watson & Pelli, 1983), ZEST (King-Smith, Grigsby, Vingrys, Benes, & Supowit, 1994), PSI (Kontsevich & Tyler, 1999)-utilize a Bayesian formulation to include the outcomes of previous trials. Studies have suggested that these techniques are highly efficient for measuring psychophysical variables.One particular application for which efficient psychometric procedures are relevant is cochlear implant research. Psychophysical variables are the primary metric for gauging the operation and performance of a cochlear implant in vivo, since more invasive procedures are not feasible. However, a comprehensive assessment of cochlear implant performance requires collecting large amounts of psychophysical data in a time-consuming process. Hence, the motivation for developing more efficient psychometric procedures to rapidly collect cochlear implant psychophysical data is particularly strong, but any efficient procedures have potential applications in a variety of fields that utilize psychophysical data.A potential alternative framework to those cited above for developing quick and efficient psychometric methods is the theory of optimal experiments. Federov (1972) and Chernoff (1972) have written extensively about the theory of optimal experiments, and it has been developed fully by many others (MacKay, 1992;Whaite & Ferrie, 1997). The theory of optimal experiments has been extended to a number of fields with applications involving estimating parameters through sequential trials. On the basis of successes in other fields, the theory of optimal experiments could serve as a promising basis for an adaptive psychometric procedure, but it has not been investigated as such to date.The theory of optimal experiments requires a model of the process under investigation. The parameters of the model can be estimated iteratively on the basis of the outcomes of sequential trials through a two-stage process. First, the parameters for the model are estimated to best fit the observed data. Then a trial is conducted at the stimulus value that will provide the most information about the model, using the most recent estimate of the parameter values. The two processes controlled by the investigator are how to estimate the parameters from the observed data and what measure of information to use for selecting the stimulus for the next trial. This is a fairly general strategy for testing and parameter estimation; many techniques that are not directly derived from the theory of optimal ex- 311Copyright 2007 Numerous previous studies have focused on the development of quick and efficient adaptive psy...

Comparison of adaptive psychometric procedures motivated by the Theory of Optimal Experiments: Simulated and experimental results

“…This can result in an observed set of data that is poorly fit by the model of the psychometric function, producing inaccurate estimates of the parameters that will also lead to the selection of poor, uninformative stimulus values. This phenomenon has been observed in Remus and Collins ͑2007͒. Several examples of Bayesian adaptive psychometric procedures exist in the literature. The quick estimation by sequential testing ͑QUEST͒ method developed by Watson and Pelli ͑1983͒ uses Bayesian techniques to place the next stimulus at the current best estimate of the threshold.…”

“…This is true even for the Bayes Greedy procedure; while it is attempting to place the stimulus value for each trial at the threshold, a point on the psychometric function that does not provide information about the slope since psychometric functions with all slope values pass through that point, some stimulus values will be offset from the true threshold value ͑either due to quantization of the stimulus values not allowing the true threshold value to be sampled or imperfect estimates of the threshold value͒ and information about the slope parameter will be provided. However, collecting substantive information about the slope parameter typically requires a greater number of trials than were presented in the psychoacoustic study or in the simulations ͑Macmillan and Creelman, 1991; Remus and Collins, 2007͒. Thus, the main function of the variable slope parameter in the Bayes FIG and Bayes Greedy methods in this study was to allow flexibility in fitting the psychometric function.…”

“…Using the notation presented previously, the gain in Fisher information for each possible stimulus value x j for the next trial can be calculated using Eqs. ͑6͒-͑8͒ ͑Liao and Carin, 2004; Remus and Collins, 2007͒. Equation ͑6͒ calculates the two-element vector B consisting of the partial derivatives of the psychometric function with respect to each of the two parameters. The vector B is a function of the stimulus value x and is utilized in Eqs.…”

Bayesian Inference of Two-Dimensional Contrast Sensitivity Function from Data Obtained with Classical One-Dimensional Algorithms Is Efficient

“…A maximum Fisher information gain method was implemented (Liao and Carin, 2004; Remus and Collins, 2007) in which the next stimulus sample was selected, using a one-step-ahead search to maximize the determinant of the Fisher information matrix. This method optimizes the sampling step to get maximum information about the parameters.…”

“…The simple strategies included: (1) the up-down staircase method (Kaernbach, 1991) that changes the intensity of the stimulus “up” or “down” after every “negative” or “positive” response, respectively; and (2) the Ψ method (Kontsevich and Tyler, 1999) that uses parametric adaptive techniques to select the next stimulus, such that the associated response would minimize the expected entropy of the threshold and slope along the contrast dimension. The two novel 2-D adaptive methods included: (1) the quick contrast sensitivity function (qCSF; Lesmes et al, 2010) that optimizes sampling along the entire CSF curve and searches for the stimulus, the response to which would minimize the expected entropy in both contrast and SF space; and (2) the FIG (Fisher information gain) method (Remus and Collins, 2007), adapted to a 2-D model for purposes of the present study, that selects the next 2-D stimulus that maximizes the Fisher information gain of function parameters. For effective comparison, the CSF model, levels of spatial frequency measured, levels of contrast, and number of sampling trials were identical across all methods.…”