Assessing reproducibility for interval data in health-related quality of life questionnaires: Which coefficient should be used?

Abstract

In contrast to other reliability estimates, test-retest reliability (or reproducibility) captures not only the measurement error of an assessment instrument, but also the stability of the construct measured. Consequently, one would expect any departure from identity (Y = X) of measurement pairs (X first, and Y second measurement) to be treated as 'error' by the respective reproducibility statistic, even if 'true' changes happened, e.g. worsening of a disease due to its natural course. The Pearson correlation, still often advocated for continuous measures in test-retest reliability studies, however captures the degree of linearity (Y = bX + a): perfect relationship can be computed, even if the measurement pairs differ not only by a additive constant 'a', but also because of a multiplication of the X-values with the slope 'b'. Therefore, intraclass correlation coefficients (ICCs) have been proposed as alternative statistics for reproducibility. However, only ICCs with absolute agreement definition of concordance capture the degree of identity. ICCs with a consistency definition of concordance measure the degree of additivity (Y = X + a). ICCs are calculated from repeated measures analyses of variance (ANOVAs), and a common population variance must be is assumed for the different measurements. Given this assumption, an ICC computed from a one-way ANOVA seems to be the best choice for this purpose. Otherwise, Lin's concordance correlation coefficient is recommended as identity measure.

Test–retest reliability of the Food Allergy Quality of Life Questionnaires (FAQLQ) for children, adolescents and adults

“…To investigate test-retest reliability of the FAQLQs, we used the intraclass correlation coefficient (ICC), using a one-way ANOVA [20,21,23]. Values should be above 0.70 for group comparison studies and above 0.90-0.95 for individual measurements over time [24].…”

“…However, different methods can be used to assess test-retest reliability, and there is much discussion in literature on the best way to do this [20]. A disadvantage of the ICC is that if patient groups are very homogeneous, the ICC tends to be low, because the ICC compares variance among patients to total variance.…”

“…The CCC takes into account not only mean differences between the first and second measurement, such as ICCs calculated by a oneway ANOVA, but also takes into account variance differences between the first and second measurement by reducing the magnitude of the resulting test-retest reliability estimate. In addition, the CCC is a better tool to distinguish between bias and imprecision [20,29]. There can be large differences in ICC and CCC scores, especially in studies with heterogeneous groups.…”

“…All patients who completed the first questionnaires (test) received the second questionnaires (re-test) 10-14 days after completion of the first. Patients who did not respond in time were excluded from the study [20,21] as well as patients who reported a clinically important change in disease between the measurements or within 2 months before the study. We defined a clinically important change in disease that could influence HRQL as a food allergic reaction of grade 3 or 4 according to the Mueller classification [22].…”

Évaluation métrologique de la Mesure de l’offre active de services sociaux et de santé en français en contexte minoritaire

“…Ce modèle est approprié lorsque des sujets s'autoévaluent à deux occasions (Schuck, 2004). Les énoncés qui ont obtenu un coefficient kappa inférieur à 0,20 (pauvre selon l'échelle de Landis et Koch, 1977) ont été examinés afin de déterminer s'ils devaient être modifiés ou éliminés du questionnaire.…”