Progressing, not regressing: a possible solution to the problem of regression to the mean in unconscious processing studies

Abstract

How convincing is current evidence for unconscious processing? Recently, a major criticism suggested that this evidence might be fully explained by a mere statistical phenomenon: regression to the mean (RttM). Since excluding participants based on an awareness assessment is a common practice in such studies, this post-hoc data selection might evoke RttM and lead to false effects that are driven by aware participants wrongfully classified as unaware. Here, we examined this criticism using both simulations and data from 15 studies probing unconscious processing (43 effects overall). In line with the original criticism, we confirmed that the reliability of awareness measures in the field is concerningly low. Yet using simulations we showed that reliability measures might be unsuitable for estimating error in awareness measures. Furthermore, we examined three proposed ways to assess whether an effect is genuine or reflects RttM; all suffered from substantial limitations, such as a lack of power or an unjustified linearity assumption. Accordingly, we suggest a new nonparametric solution, which enjoys high sensitivity and relatively high power. Together, this work emphasizes the need to account for the contribution of measurement error to effects of unconscious processing. It further suggests a way to meet the important challenge posed by RttM, in an attempt to establish a reliable and robust corpus of knowledge in studying unconscious processing.

Unconscious integration: Current evidence for integrative processing under subliminal conditions

“…Even with sufficient power, post‐hoc exclusion of participants/trials from the final analysis (i.e. excluding trials rated as ‘visible’ or excluding participants with above‐chance awareness) has been criticized on the ground that it leads to the regression to the mean (Rothkirch et al, 2022; Shanks, 2017; Yaron et al, 2022), which underestimates true awareness. Awareness measures are prone to measurement error (noise; Vadillo et al, 2022; Yaron et al, 2022), so the observed awareness score would be a combination of the true awareness score plus some measurement error (Rothkirch et al, 2022).…”

“…excluding trials rated as ‘visible’ or excluding participants with above‐chance awareness) has been criticized on the ground that it leads to the regression to the mean (Rothkirch et al, 2022; Shanks, 2017; Yaron et al, 2022), which underestimates true awareness. Awareness measures are prone to measurement error (noise; Vadillo et al, 2022; Yaron et al, 2022), so the observed awareness score would be a combination of the true awareness score plus some measurement error (Rothkirch et al, 2022). Positive error would more likely result in a larger observed score (since you are ‘adding’ to the true score), and likewise negative error (‘subtracting’ from the true score) often leads to a smaller observed score.…”

“…That is, the true awareness score of the ‘unaware’ participants/trials regresses towards the group mean (Rothkirch et al, 2022). Several solutions have been proposed to account for the regression to the mean (Leganes‐Fonteneau et al, 2021; Shanks, 2017; Skora, Livermore, Dienes, et al, 2022; Yaron et al, 2022). Regression to the mean is not an issue if the awareness test has high reliability (Rothkirch et al, 2022; Yaron et al, 2022).…”

“…Several solutions have been proposed to account for the regression to the mean (Leganes‐Fonteneau et al, 2021; Shanks, 2017; Skora, Livermore, Dienes, et al, 2022; Yaron et al, 2022). Regression to the mean is not an issue if the awareness test has high reliability (Rothkirch et al, 2022; Yaron et al, 2022). However, previous studies in the field did not assess/report the reliability of their awareness test, and this practice was only recently recommended by researchers (Shanks et al, 2021).…”

“…However, previous studies in the field did not assess/report the reliability of their awareness test, and this practice was only recently recommended by researchers (Shanks et al, 2021). Several analyses conducted on previous studies suggest that awareness test with high reliability is uncommon in the field (Rothkirch et al, 2022; Vadillo et al, 2022; Yaron et al, 2022).…”