A Review on Assessing Agreement

Abstract

Measurements serve as the basis for evaluation in almost all scientific disciplines, especially in physical sciences, medical studies, and health care. Issues related to reliable and accurate measurement have evolved over many decades. Requiring a measurement to be identical to the truth is sometimes impractical or impossible either because (i) the truth is simply not available or is measured with some error or (ii) some tolerable error is acceptable. Concepts of agreement, including reproducibility or reliability, are often used to determine whether the measurements can be used or not for evaluation. There has been substantial statistical literature in the last several decades on assessing agreement. This article provides a critical and comprehensive review on agreement concepts and their corresponding agreement indices developed for assessing agreement among measurements made on the same subject or experimental unit. The emphasis is on the intuitive understanding of concepts and on insights into both controversies and appropriate applications for continuous and categorical data. Four examples with either continuous or categorical measurements are used for illustration and discussion.

Using multiple agreement methods for continuous repeated measures data: a tutorial for practitioners

“…It is therefore not a "pure agreement index" [41]. The CCC has the disadvantage of being heavily dependent on the between-subject variability (and in our case also on the between-activity variability) and would therefore attain a high value for a population with substantial heterogeneity between subjects or activities even though the agreement within subjects might be low [2,11,12]. Similarly, if both the between subject and between-activity variances are very low, then the CCC is unlikely to attain a high value even if agreement within devices is reasonable.…”

“…Moreover, as for the intraclass correlation coefficient (ICC), it is not related to the actual scale of measurement or to the size of error which might be clinically allowable, which makes interpretation difficult [41]. As outlined in other papers [11,12,40], it is very easy to obtain an artificially high value of CCC and manipulation of the dataset can change the estimate of the CCC drastically. Nevertheless, the variance components are automatically generated in R which helps one to interpret the overall summary indices.…”

“…However, we would not recommend providing just the coverage probability index, if there is any disagreement, because it may hide important nuances in the data, particularly relating to the source of the disagreement in applied clinical studies. We therefore recommend that researchers also construct a Bland-Altman plot (which also depicts the raw mean bias and clinically acceptable difference) to provide a helpful visual examination of the data [12] (see Fig. 3).…”

“…With an emphasis on practical application and interpretation, the aim of this study is to show how these five approaches can be applied to the same agreement problem and showcase the strengths and weaknesses of each method so that researchers can decide which methods to use in their own studies. Reviews of agreement indices have already been presented in the literature by [2], Obuchowski et al (2015) [10], Barnhart et al (2016) [11], and Barnhart (2018) [12]; with the latter three papers including real life examples to compare between agreement indices. However, the examples provided were almost exclusively sourced from the fields of quantitative imaging and core laboratory research.…”

Comparing a Fitbit Wearable to an Electrocardiogram Gold Standard as a Measure of Heart Rate Under Psychological Stress: A Validation Study

“…Moreover, interparticipant variability in HR changes is echoed in a previous study that used Fitbit-measured HR as an indicator of stress [ 24 ]. As such, to account for this expected variability, the CIA was computed as it is less dependent on the between-subjects variability compared to the CCC [ 47 , 58 ]. However, the repeatability coefficient of Bland-Altman was found to be unacceptably high and warrants caution when interpreting the CIA.…”