The Prevalence and Prevention of Crosstalk: A Multi-Institutional Study

Abstract

It is a common problem in psychology subject pools for past study participants to inform future participants of key experimental details (also known as crosstalk). Previous research (Edlund, Sagarin, Skowronski, Johnson, & Kutter, 2009) demonstrated that a combined classroom and laboratory treatment could significantly reduce crosstalk. The present investigation tested a laboratory-only treatment for the prevention of crosstalk at five universities, along with institutional-level moderators of crosstalk. Results indicated the presence of crosstalk at all universities and that the laboratory-based treatment was effective in reducing crosstalk. Importantly, crosstalk rates were higher (but successfully neutralized) in research pools with higher research credit requirements. Therefore, this research provides valuable guidance regarding crosstalk prevalence and its minimization by researchers.

Participant Crosstalk: Issues When Using the Mechanical Turk

“…In study three, the data from the mTurk workers suggest that some mTurk requestors try to prevent crosstalk (whereas others do not). Furthermore, in study three, it appears that requesting that mTurk workers not engage in crosstalk is a potentially successful way to reduce crosstalk (as is suggested by Edlund et al, 2009;Edlund et al, 2014). However, these can best be described as converging, but non-definitive pieces of evidence suggesting that crosstalk on mTurk can be reduced by requesting that mTurk workers not engage in crosstalk.…”

“…Edlund et al (2009) found that a combined classroom and laboratory treatment effectively reduced crosstalk to a barely detectable level (<1%). In addition, Edlund et al (2014) found evidence that a solely laboratory based treatment could be effective in reducing crosstalk, but only in institutions that demanded more out of their subject pools. However, to date, little research has investigated the prevention (or prevalence) of crosstalk outside of the typical introduction to psychology subject pool.…”

“…In this study, we wanted to test a simple survey that would be unlikely to be impacted directly by the crosstalk (rather than feature a study that could be meaningfully impacted by crosstalk). In many ways, this study represents a lower bound of crosstalk rates (as suggested by Edlund et al, 2014).…”

“…Some researchers have demonstrated that the rates can be as high as 78% (Lichtenstein, 1970), whereas other more contemporary multiinstitutional studies have suggested lower rates in the single digits (Edlund et al, 2014). Importantly, any crosstalk that reveals key experimental information is critically important to identify and prevent as it has the potential to alter participant responses and thereby, the validity of the research (Nichols & Edlund, 2015).…”

Practicing What We Preach (and Sometimes Study): Methodological Issues in Experimental Laboratory Research

“…Despite more than 40 years passing between investigations into crosstalk, recent research has again brought attention to this issue and examined possible solutions (Edlund, Sagarin, Skowronski, Johnson, & Kutter, 2009; Edlund et al, 2014). Differing from past examinations, Edlund and colleagues (2009) began to examine crosstalk by designing tasks aimed at detecting crosstalk without relying on self-reported rates.…”