Estimation of psychometric functions from adaptive tracking procedures

Abstract

Because adaptive tracking procedures are designed to avoid stimulus levels far from a target threshold value, the psychometric function constructed from the trial-by-trial data in the track may be accurate near the target level but a poor reflection of performance at levels far removed from the target. A series of computer simulations was undertaken to assess the reliability and accuracy of psychometric functions generated from data collected in up-down adaptive tracking procedures. Estimates of psychometric function slopes were obtained from trial-by-trial data in simulated adaptive tracks and compared with the true characteristics of the functions used to generate the tracks. Simulations were carried out for three psychophysical procedures and two target performance levels, with tracks generated by psychometric functions with three different slopes. The functions reconstructed from the tracking data were, for the most part, accurate reflections ofthe true generating functions when at least 200 trials were included in the tracks. However, for 50-and 100-trial tracks, slope estimates were biased high for all simulated experimental conditions. Correction factors for slope estimates from these tracks are presented. There was no difference in the accuracy and reliability of slope estimation due to target level for the adaptive track, and only minor differences due to psychophysical procedure. It is recommended that, ifboth threshold and slope of psychometric functions are to be estimated from the trial-by-trial tracking data, at least 100 trials should be included in the tracks, and a three-or four-alternative forced-choice procedure should be used. However, good estimates can also be obtained using the two-alternative forced-choice procedure or less than 100 trials if appropriate corrections for bias are applied.Adaptive testing procedures have become popular in psychophysical experiments over the past 20 years due to their efficiency and speed. In these procedures, the level of a stimulus on each experimental trial is determined by performance on previous trials. Such methods are characterized by their ability to converge rapidly on a given level of performance and to concentrate experimental trials in the vicinity of the final measurement of interest. Little experimental time and subject energy is expended on trials placed far from the point of interest on the psychometric function.The trade for this high efficiency, however, is the loss of information about the underlying function that defines

Adaptive psychophysical procedures

“…These results were obtained for either maximum-likelihood and probit analysis and they are in accordance with the study of McKee et al (1985). Similar results were found by Leek et al (1992) who compared the method of constant stimuli, APE, and UDTR.…”

“…by Emerson (1986a), Kershaw (1985), Leek, Hanna and Marshall (1992), Lieberman and Pentland (1982), Madigan and Williams (1987), Maloney (1990), Rose, Teller and Rendleman (1970), Simpson (1989) or Swanson and Birch (1992). In this case, a model for the psychophysical observer is chosen and responses are generated according to the probabilities associated with this observer model.…”

Slope bias of psychometric functions derived from adaptive data

“…Slope bias is larger for smaller step sizes (or, as Leek et al, 1992, put it, for a constant step size, slope bias increases with lower true slope of the PF). With constant step size, the amount of slope bias decreases with increasing run length.…”

“…The run lengths needed to get slope estimates with less than 10% error are rather high. Leek et al (1992) suggest runs of at least 200 trials. For shorter runs, they suggest correction factors.…”

Robust and efficient Bayesian adaptive psychometric function estimation

“…By fitting a parametric model to the sequence of responses using a maximum likelihood criterion, it is possible to estimate both the SRT and slope at a chosen performance level. It has, however, been found that the slope estimates obtained in this way may have considerable bias (Leek et al, 1992;Kaernbach, 2001;Treutwein and Hansstrasburger, 1999). Wetherill (1963) derived expressions for the asymptotic variance of the maximum likelihood estimators ofã andb in Eq.…”