Assessing method agreement for paired repeated binary measurements administered by multiple raters

Abstract

Method comparison studies are essential for development in medical and clinical fields. These studies often compare a cheaper, faster, or less invasive measuring method with a widely used one to see if they have sufficient agreement for interchangeable use. Moreover, unlike simply reading measurements from devices, e.g., reading body temperature from a thermometer, the response measurement in many clinical and medical assessments is impacted not only by the measuring device but also by the rater. For example, widespread inconsistencies are commonly observed among raters in psychological or cognitive assessment studies due to different characteristics such as rater training and experience, especially in large-scale assessment studies when many raters are employed. This paper proposes a model-based approach to assess agreement of two measuring methods for paired repeated binary measurements under the scenario where the agreement between two measuring methods and the agreement among raters are required to be studied simultaneously. Based upon the generalized linear mixed models (GLMM), the decision on the adequacy of interchangeable use is made by testing the equality of fixed effects of methods. Approaches for assessing method agreement, such as the Bland-Altman diagram and Cohen's kappa, are also developed for repeated binary measurements based upon the latent variables in GLMMs. We assess our novel model-based approach by simulation studies and a real clinical application, in which patients are evaluated repeatedly for delirium with two validated screening methods. Both the simulation studies and the real data analyses demonstrate that our proposed approach can effectively assess method agreement. (Nan Lin) arXiv:1809.09740v2 [stat.ME] 25 Jun 2019 αm   γ i + 1 J J j=1 α jm   , m = 1, 2.

Evaluation of the 3-Minute Diagnostic Confusion Assessment Method for Identification of Postoperative Delirium in Older Patients

“…An individual-level summary measure for each method was given based upon the latent variable formulation of the GLMM used for testing method agreement. 17 That is, a pair of model-estimated continuous delirium outcomes for the CAM and the 3D-CAM was determined for each of the 299 patients and used to plot the Bland-Altman diagram. A pair of model-estimated binary delirium outcomes was then generated based on the latent variable for the evaluation of Cohen κ.…”

“…An individual-level summary measure for each method was given based upon the latent variable formulation of the GLMM used for testing method agreement. 17 Additional Bland-Altman diagrams were generated for each of the 4 features (Figure 2). One feature, altered level of consciousness, was plotted on a log scale since the data must be normally distributed for Bland-Altman analysis.…”

“…We previously published a detailed description of the statistical methods that we used in this study. 17 Briefly, a generalized linear mixed model (GLMM) was used for interrater reliability as well as method agreement (CAM vs 3D-CAM). Even though only 1 CAM and 3D-CAM were conducted at any given interview (ie, 1 rater used the CAM and 1 the 3D-CAM), the GLMM method is able to provide an estimate of interrater reliability for each instrument.…”

Reliability of Markers for Breast Hypoplasia in the Early Postpartum Period

“…Despite a high percentage (97%) of overall agreement between raters for classifying breast type, weighted kappa revealed fair agreement between raters for classifying breast type; however, 0 was in the 95% CI, indicating no agreement. Kappa is affected by the prevalence of the attribute being observed and for rare findings low values of kappa may not necessarily reflect low rates of overall agreement (Wang et al, 2020). This was the situation with our sample because 90% of participants were observed to have Type 1 breasts and 10% Type 2 (no Types 3 or 4 breasts were identified).…”

“…For categorizing breast type, fair interrater reliability was considered if the weighted kappa was 0.21–0.40, moderate if it was 0.41– 0.6, substantial if it fell between 0.61–0.8 and almost perfect if it was greater than 0.8 (Denham, 2016). Kappa is affected by the prevalence of the observed attribute, meaning that for rare findings, a very low value of kappa may not necessarily reflect low rates of overall agreement (Wang et al, 2020).…”