Sample size requirements for the design of reliability study: review and new results

Abstract

The reliability of continuous or binary outcome measures is usually assessed by estimation of the intraclass correlation coefficient (ICC). A crucial step for this purpose is the determination of the required sample size. In this review, we discuss the contributions made in this regard and derive the optimal allocation for the number of subjects k and the number of repeated measurements n that minimize the variance of the estimated ICC. Cost constraints are discussed for both normally and non-normally distributed responses, with emphasis on the case of dichotomous assessments. Tables showing optimal choices of k and n are given along with the guidelines for the efficient design of reliability studies.

Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed

“…Rationale: These numbers are necessary to evaluate the precision of the study and to make further calculations (e.g., in meta-analysis) possible [20,31,58,59,97,98]. A flow diagram allows readers to follow the inclusion and exclusion process from the intended sample of raters and subjects to the actual sample.…”

“…Values of 0.60, 0.70, or 0.80 are often used as the minimum standards for reliability coefficients, but this may be only sufficient for group-level comparisons or research purposes [12,58,108]. For example, ICC values for a scale measuring pressure ulcer risk should be at least 0.90 or higher when applied in clinical practice [108].…”

“…Moreover, although the ICC is reported in many research reports, it is often not clear which ICC was used [14,43]. When continuous measurements are split into distinct categories (see item 2), it is recommended that results be calculated for the continuous measurement as well, because the transformation of continuous values into categories may cause difficulties in interpretation and lead to a reduction in statistical power [32,58,85]. The proposed measures of agreement include proportions of exact agreement [8,86,87], proportions of specific agreement [86,87], repeatability coefficients, and the graphical method proposed by Bland and Altman [88].…”

Assisted self-report of health-services use showed excellent reliability in a longitudinal study of older adults

“…This sample size also gave us 85% power to show that agreement is at least substantial (over 0.60 [12]) for continuous outcomes with a point estimate of agreement above 0.80 [13].…”

“…Anticipating the prevalence of binary outcomes to be above 30% and the estimated reliability coefficients to be over 0.80, 47 subjects per substudy (testeretest and interinterviewer reliability) ensured us 80% power to show that agreement is at least moderate (over 0.40 [12]) with a 5% significance level [13]. This sample size also gave us 85% power to show that agreement is at least substantial (over 0.60 [12]) for continuous outcomes with a point estimate of agreement above 0.80 [13].…”