Reaction time in differential and developmental research: A review and commentary on the problems and alternatives.

Abstract

Reaction time is believed to be a good indicator of the speed and efficiency of mental processes and is a ubiquitous variable in the behavioral sciences. Despite this popularity, there are numerous issues associated with using reaction time (RT), specifically in differential and developmental research. Here, we identify and focus on two main problems—unreliability and sensitivity to speed–accuracy interactions. The use of difference scores is a primary factor that leads to many RT measures having demonstrably low reliability, and RT measures in general often do not properly account for speed–accuracy interactions. Both factors jeopardize the validity and interpretability of results based on RT. Here, we evaluate conceptually and empirically how these issues affect individual differences research. Although the empirical evidence we provide are primarily within the domains of attention control and task switching, we highlight examples from various other areas of psychological inquiry. We also discuss many of the statistical and methodological alternatives available to researchers conducting correlational studies. Ultimately, we encourage researchers comparing individuals of differing cognitive and developmental levels to strongly consider using these alternatives in lieu of RT, specifically RT difference scores.

Why Most Studies of Individual Differences With Inhibition Tasks Are Bound To Fail

“…The question of why these correlations are so low has been the subject of recent work by Draheim, Mashburn, Martin, and Engle (2019), Hedge et al (2018) and Rey-Mermet et al (2018) among others. On one hand, they could reflect underlying true task performance that is uncorrelated or weakly correlated.…”

“…The results were somewhat disheartening with test-retest reliabilities for popular tasks in the range from .2 to .7. Draheim et al (2019) argue that commonly used response time difference scores are susceptible to low reliability and other artifacts such as speed-accuracy tradeoffs.…”

“…With a measure, say a working-memory capacity measure, there is neither a critical manipulation nor a critical contrast. In practice, because there is no subtraction, measures are often highly reliable and lead to higher correlations [Draheim et al (2019); Hedge:etal:2018]. We urge individual-difference researchers to draw a sharp distinction between scores from experimental tasks on one hand and psychological measures on the other.…”

“…One way of increasing individual variability, from Engle and colleagues, is to dispense with contrasts and difference scores altogether (Draheim et al, 2019;Kane & Engle, 2003;Unsworth, Schrock, & Engle, 2004). In this approach, task scores reflect the average rather than the difference among conditions.…”

Motivation and cognitive control in depression

“…For example, it is known that the reliability of the standard control tasks is relatively low when compared to the norms of the questionnaire-based research. This is especially true for difference scores (Draheim, Mashburn, Martin, & Engle, 2019), which are very often used as indexes of cognitive control in individual differences research. While there is no straightforward solution for this issue, recently there has been a growing interest in assessing the psychometric properties of cognitive control tasks and improving the measurement of controlrelated constructs.…”