How do headphone checks impact perception data?

Abstract

Headphone checks have rapidly become an assumed part of best practices in online perception studies. Do they actually improve our ability to find phonological patterns? This study attempts to replicate several perceptual effects that depend on different aspects of the acoustic signal, testing whether requiring participants to pass two commonly used headphone checks (Huggins pitch perception, Milne et al., 2021; dichotic loudness perception, Woods, Siegel, Traer, & McDermott, 2017) impacts the results or the sample population being used. Participants who pass the Huggins check exhibit a larger effect of spectral tilt on perceived vowel duration, but no other effects were strengthened by either headphone check. Headphone checks actually resulted in a weaker effect of coda voicing on perceived vowel duration. A look at who is excluded by the headphone checks shows that both of them produce exclusions that are imbalanced by participants' age, gender, education, and geographic region. These imbalanced exclusions have the potential to impact experimental results in a range of ways. Thus, while headphone checks are likely to be valuable for some types of experiments, for others they might have unintended effects, and for many studies they are unlikely to have any substantial effect on the results.