The Bayesian One-sample t-Test Supersedes Correlation Analysis as a Test of Validity

Abstract

Validity of measurement is essential to the scientific endeavor. It refers to how accurately tools measure what they are intended to measure. Researchers rely on statistical approaches to test the validity of their measures. One such approach is correlation analysis. Even though correlation analysis can capture high nonsystematic error between measures, it can often lead to misleading conclusions when observations are measured with systematic error, as can be the case when examining validity. Here, we provide simulated data to demonstrate how correlation analysis is commonly used to test for validity and how this method can fail as a test of validity when the error is systematic. We further propose an alternative to correlation analysis, for cases where the use of a simple statistical test can be justified. Specifically, we suggest the use of the Bayesian one-sample t-test, which can identify both systematic and non-systematic error and moreover provide evidence for the null hypothesis of no differences between two measures. We further provide simulations showcasing the implementation of the Bayesian one-sample t-test and how it can be used to address the limitations of correlation analysis.

Phosphene and Motor Transcranial Magnetic Stimulation Thresholds Are Correlated: A Meta-Analytic Investigation

“…By inducing phosphenes, one can estimate a phosphene threshold (PT), which is the minimum required stimulation intensity to evoke self-reported phosphenes in at least half the trials (e.g., 5 out of 10 stimulations; see Mazzi et al, 2017). The PT has been extensively used to study vision and visual cognition (for recent reviews see Phylactou et al, 2022, 2023), as well as to measure occipital cortex excitability (for reviews see Aurora & Welch, 1998; Brigo et al, 2013).…”