Statistical Methodology for the Concurrent Assessment of Interrater and Intrarater Reliability: Using Goniometric Measurements as an Example

Abstract

Statistical methodology for the concurrent assessment of interrater and intrarater reliability is presented. Application of the methodology is illustrated with an example of one therapist using two goniometers repeatedly to measure knee joint angles. Methods for estimating the coefficients, testing hypotheses, constructing confidence intervals, and computing sample size requirements are provided. In addition, the calculation and clinical interpretation of the standard error of measurement (SEM) are discussed. It is recommended that (1) when both interrater and intrarater reliability are being assessed, a repeated-measures design be used to take advantage of the increased precision gained by using all observations in the statistical analysis, and (2) appropriate statistical tests, confidence intervals, and SEMs always be used in conjunction with the estimated reliability coefficients.

Consequences of increased neuromotor noise for reaching movements in persons with stroke

“…Across trials, the relative reliability [intra-class correlations, ICC(1,1); Shrout and Fleiss, 1979] and measurement agreement [standard error of measurement, SEM; Eliasziw et al, 1994] of movement time was high: control subjects (ICC=0.98, %SEM=10.5), less affected arm of stroke subjects (ICC=0.98, %SEM=11.6), and more affected arm of stroke subjects (ICC=0.98, %SEM=20.9). Linear regressions between movement time and task ID determined Fitts' slope and intercept coefficients.…”

Precision of shoulder anatomical landmark calibration by two approaches: A CAST-like protocol and a new anatomical palpator method

“…Mean A-Palp accuracy obtained after AL palpation on the specimen shoulder was 3.6 and 8.1 mm for the scapula and humerus, respectively. Mean (range) propagation of the A-Palp error on the continuous motion graph representation was 28 (18) and 58 (78) for scapula and humerus, respectively (Fig. 3).…”

“…ICC statistics was not performed due to the small sample size (two cases) and the range of the measurement that could lead to poor reliability estimation [18,19].…”

Use of Morphometry to Quantify Osteolysis after Total Hip Arthroplasty

“…Interobserver and intraobserver (test-retest) reliability of the morphometric grid were tested using various observers, as recommended in the development of a new tool [11,21,25,42]. Four health professionals were selected to represent the range and levels of experience involved in arthroplasty review.…”

“…It has been suggested a sample size of 20 to 50 observations is needed to obtain an intraclass correlation coefficient (ICC) with a standard error less than 0.05 [38]. Others recommend, at 80% power and a = 0.05, four raters would need to perform at least 30 observations each for a hypothesis test against a true reliability coefficient of 0.90 [11]. Consequently, a pragmatic decision was made to use 20 observations initially, as has been reported elsewhere [2,33], and 35 observations at the second stage.…”