Methods to split cognitive task data for estimating split-half reliability: A comprehensive review and systematic assessment

Abstract

Estimating the reliability of cognitive task datasets is commonly done via split-half methods. We review four methods that differ in how the trials are split into parts: a first-second half split, an odd-even trial split, a permutated split, and a Monte Carlo-based split. Additionally, each splitting method could be combined with stratification by task design. These methods are reviewed in terms of the degree to which they are confounded with four effects that may occur in cognitive tasks: effects of time, task design, trial sampling, and non-linear scoring. Based on the theoretical review, we recommend Monte Carlo splitting (possibly in combination with stratification by task design) as being the most robust method with respect to the four confounds considered. Next, we estimated the reliabilities of the main outcome variables from four cognitive task datasets, each (typically) scored with a different non-linear algorithm, by systematically applying each splitting method. Differences between methods were interpreted in terms of confounding effects inflating or attenuating reliability estimates. For three task datasets, our findings were consistent with our model of confounding effects. Evidence for confounding effects was strong for time and task design and weak for non-linear scoring. When confounding effects occurred, they attenuated reliability estimates. For one task dataset, findings were inconsistent with our model but they may offer indicators for assessing whether a split-half reliability estimate is appropriate. Additionally, we make suggestions on further research of reliability estimation, supported by a compendium R package that implements each of the splitting methods reviewed here.

TANGO: A reliable, open-source, browser-based task to assess individual differences in gaze understanding in 3 to 5-year-old children and adults

“…Figure 3 compares split-half reliability coefficients by splitting and stratification method (Pronk et al, 2021). In the hedge version, the split-half reliability coefficients ranged from 0.65 to 0.93.…”

“…First-second and odd-even splits belong to single sample methods since each participant has a single pair of performance scores, while permutated (without replacement) and Monte Carlo (with replacement) splits make use of resampling. Analyses were run using the function by_split from the splithalfr package (Pronk et al, 2021).…”

When Alcohol Adverts Catch the Eye: A Psychometrically Reliable Dual-Probe Measure of Attentional Bias

“…Analyses were conducted in RStudio (Version 1.2.5019). First, the internal reliability of the attentional bias measure was determined using a split-half analysis conducted with the multicon [37] and splithalfr packages [38]. Following this, bivariate Pearson and Spearman correlations between attentional bias (as provided by the dual-probe task) and demographics of interest (gender, age), habitual alcohol use, attentional control, and laboratory measures of drinking (change in subjective craving, beer consumption in the sham taste test) were examined.…”

Development and Validation of Nutrition Literacy Assessment Instrument for Chinese Pregnant Women

“…The spilt-half reliability was another indicator used for evaluating the reliability [ 23 ]. The questionnaire was randomly divided into two parts, and then Spearman-Brown and Guttman split-half coefficients were calculated between the scores of two parts.…”