The psychometric function: II. Bootstrap-based confidence intervals and sampling

Abstract

The performance of an observer on a psychophysical task is typically summarized by reporting one or more response thresholds-stimulus intensities required to produce a given level of performance-and by a characterization of the rate at which performance improves with increasing stimulus intensity. These measures are derived from a psychometric function, which describes the dependence of an observer's performance on some physical aspect of the stimulus.Fitting psychometric functions is a variant of the more general problem of modeling data. Modeling data is a three-step process: First, a model is chosen, and the parameters are adjusted to minimize the appropriate error metric or loss function. Second, error estimates of the parameters are derived and third, the goodness of fit between model and the data is assessed. This paper is concerned with the second of these steps, the estimation of variability in fitted parameters and in quantities derived from them. Our companion paper (Wichmann & Hill, 2001) illustrates how to fit psychometric functions while avoiding bias resulting from stimulus-independentlapses, and how to evaluate goodness of fit between model and data.We advocate the use of Efron's bootstrap method, a particular kind of Monte Carlo technique, for the problem of estimating the variability of parameters, thresholds, and slopes of psychometric functions (Efron, 1979(Efron, , 1982 Efron & Gong, 1983; Efron & Tibshirani, 1991, 1993. Bootstrap techniques are not without their own assumptions and potential pitfalls. In the course of this paper, we shall discuss these and examine their effect on the estimates of variability we obtain. We describe and examine the use of parametric bootstrap techniques in finding confidence intervals for thresholds and slopes. We then explore the sensitivity of the estimated confidence interval widths to (1) sampling schemes, (2) mismatch of the objective function, and (3) accuracy of the originally fitted parameters. The last of these is particularly important, since it provides a test of the validity of the bridging as- The psychometric function relates an observer' s performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable,experimentersrequire an estimate of the variabilityof the parameters to assess whether differences across conditions are significant.Accurate estimates of variabilityare difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statisticaltechniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternativestatisticaltechniques based on Monte Carlo resampling methods. The present paper's principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in...

Speed limits: Orientation and semantic context interactions constrain natural scene discrimination dynamics.

“…Threshold and slope differences that exceed 95% of the simulated differences are considered significantly different on a 5% alpha level. Details and validity tests of the method are given by Wichmann and Hill (2001b). The results from these tests will be denoted Monte Carlo test (MCT).…”

Asymmetries in visuomotor recalibration of time perception: Does causal binding distort the window of integration?

“…PSS and JND were the only free parameters. This was done using the Matlab toolbox psignifit (Wichmann & Hill, 2001a, 2001b. Another psychometric function (two-criterion model; cf.…”

Modulation Enhancement in the Electrical Signal Improves Perception of Interaural Time Differences with Bimodal Stimulation

“…Psychometric functions were fitted to the results using the psignifit toolbox version 2.5.6 for MATLAB 1 which implements the maximum-likelihood method described by Wichmann and Hill (2001). The 68 % confidence intervals around the fitted values were obtained by the BCA bootstrap method implemented by psignifit, based on 1999 simulations.…”