Do We Overestimate the Within-Variability? The Impact of Measurement Error on Intraclass Coefficient Estimation

Abstract

Many psychological phenomena have a multilevel structure (e.g., individuals within teams or events within individuals). In these cases, the proportion of betweenvariance to total-variance (i.e., the sum between-variance and within-variance) is of special importance and usually estimated by the intraclass coefficient (1) [ICC(1)]. Our contribution firstly shows via mathematical proof that measurement error increases the within-variance, which in turn decreases the ICC(1). Further, we provide a numerical example, and examine the RMSEs, alpha error rates and the inclusion of zero in the confidence intervals for ICC(1) estimation with and without measurement error. Secondly, we propose two corrections [i.e., the reliability-adjusted ICC(1) and the measurement model-based ICC(1)] that yield correct estimates for the ICC(1), and prove that they are unaffected by measurement error mathematically. Finally, we discuss our findings, point out examples of the underestimation of the ICC(1) in the literature, and reinterpret the results of these examples in the light of our new estimator. We also illustrate the potential application of our work to other ICCs. Finally, we conclude that measurement error distorts the ICC(1) to a non-negligible extent.

Development, validation, and comparison of self-report measures for positive and negative affect in intensive longitudinal research.

“…For the NA and BA measures, the single items showed the highest proportion of within-person variance. It is known that single items contain more measurement error (i.e., residual variance) than composite scores, which induces a positive bias on the rICC estimates of single items (Wilms et al, 2020). To disentangle the residual from systematic within-person variance, we estimated three-level models, in which moments were nested in days, which were nested in persons.…”

Advancing motion denoising of multiband resting-state functional connectivity fMRI data

“…To allow investigators to approximate optimal volume censoring parameters for virtually any study protocol (i.e., a given number of runs per subject, and volumes per run) without having to perform the computationally intensive optimization procedures employed here, we sought to generalize our optimized LPD-FD and GEV-DV thresholds beyond the HCP 500 dataset. We first decomposed the observed between-subjects variance (across the full range of censoring thresholds) using a three-level hierarchical model ( 46–48 ), using combined LPF-FD and GEV-DV volume censoring (see Figure S24 ). We then estimated the between-subjects variance ( Figure S24B ) that would be observed in a dataset of a different size, but with an approximately equivalent distribution and character of motion (and pseudomotion) artifact throughout the sample (i.e., under the assumption that volumes and runs will be removed in identical proportion to the HCP 500 dataset, with proportional effects on bias and variance; see Estimation of Optimal Censoring Thresholds for Other Datasets in Methods ).…”

“…We first consider a variance decomposition of a given RSFC correlation for ROI pair k , based on a three-level hierarchical model with unweighted means ( 46–48 ). At the top level, the observed between-subjects variance can be described using the following relationship: where is the observed between-subjects variance for ROI pair k , is the observed between-runs variance for ROI pair k , is the estimated true between-subjects variance for ROI pair k, and is the harmonic mean of the number of runs across all subjects in the study.…”

Eudaemonic Well-Being in Midlife Women: Change in and Correspondence Between Concurrent and Retrospective Reports

“…For diary variables, the correlation between any two days within a person (ICC 1 ) was moderate at ~ r = 0.40. Assuming single item reliability of 0.50, reliability-adjusted ICC 1 s were 0.55 for autonomy (vs. 0.42 uncorrected), 0.52 for competence (vs. 0.39), and 0.58 for relatedness (vs. 0.44) (Wilms et al, 2020). It is therefore reasonable to conclude that approximately half of the variance was due to person, with the remainder due to changes among bursts and days.…”