The validity of the online thought-probing procedure of mind wandering is not threatened by variations of probe rate and probe framing

Abstract

Recently, there has been a surge of interest in the measurement of mind-wandering during ongoing tasks. The frequently used online thought-probing procedure (OTPP), in which individuals are probed on whether their thoughts are on-task or not while performing an ongoing task, has repeatedly been criticized, because variations in the frequency of thought probes and the order in which on-task and off-task thoughts are referred to have been shown to affect mind-wandering rates. Hitherto, it is unclear whether this susceptibility to measurement variation only affects mean response rates in probe-caught mind-wandering or poses an actual threat to the validity of the OTPP, endangering the replicability and generalizability of study results. Here we show in a sample of 177 students that variations of the frequency or framing of thought probes do not affect the validity of the OTPP. While we found that more frequent thought probing reduced the rate of probe-caught mind-wandering, we did not replicate the effect that mind-wandering is more likely to be reported when off-task thoughts are referred to first rather than second. Crucially, associations between probe-caught mind-wandering and task performance, as well as associations between probe-caught mind-wandering and covariates (trait mind-wandering, reaction-time variability in the metronome-response task, working-memory capacity) did not change with variations of the probing procedure. Therefore, it seems unlikely that the great heterogeneity in the way the OTPP is implemented across different studies endangers the replicability and generalizability of study results. Data and analysis code are available at https://osf.io/7w8bm/.

Testing the construct validity of competing measurement approaches to probed mind-wandering reports

“…The reactivity of TUT reports to probing was assessed by four studies. Three tested whether probe rate affected TUT rates: Robison et al (2019) probed subjects after 7% or 13% of task trials, Schubert et al (2020) probed after 3% or 6% of trials, and Seli, Carriere, et al (2013) presented 5-25 probes in a 15-min task . Results varied.…”

Individual Differences in Task-Unrelated Thought in University Classrooms

“…The study had several methodological strengths that we recommend for future studies. It used experience-sampling probes to assess in vivo TUTs, which demonstrate good construct validity (e.g., Kane et al, 2020;Robison et al, 2019;Schubert et al, 2020). We sampled TUT reports from hundreds of students across multiple meetings of multiple courses, reflecting multiple topics, at two universities serving different populations.…”

Mind wandering during reading: An interdisciplinary and integrative review of psychological, computing, and intervention research and theory

“…For example, Weinstein et al (2018) found that mind wandering was reported more frequently when probes had a mind wandering-positive frame ('my mind was on something other than the text') versus a task-positive frame ('my mind was on the text'). Whereas other methodological artefacts such as probe spacing (Seli et al, 2013) and proberesponse options (Seli, Beaty, et al, 2018) can influence mind wandering responses, Schubert et al (2019) suggest that these factors do not influence the reliability and generalizability of corresponding studies. Thus, if carefully administered, self-reports remain the most viable way to measure an internal phenomenon such as mind wandering.…”

“…Research has indicated that both (probe-and self-caught) methods produce reliable measures (Schubert et al, 2019;Varao-Sousa & Kingstone, 2019) within the limits of self-reporting. For example, mind wandering measures are internally consistent in that within-subject correlations are high (Faber et al, 2018a), they are reliably linked to predictable patterns in physiology (Smallwood et al, 2004), pupillometry (Franklin et al, 2013), eye gaze (Reichle et al, 2010), and they demonstrate predictive validity via negative associations with task performance (Randall et al, 2014).…”