Reducing interrater variability and improving health care: a meta‐analytical review

Tuijn, Saskia M.; Janssens, Frans; Robben, Paul; Bergh, Huub van den

doi:10.1111/j.1365-2753.2011.01705.x

Cited by 37 publications

(32 citation statements)

References 65 publications

(13 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, efforts were made to enhance the agreement on the HCT-CI scoring by both improving the instrument (the comorbidity coding tool) and training the evaluators (the methodology and the Web-based application). 32,41 The IRR was lowest (Kw, 0.433) when scores were compared between a single evaluator and multiple untrained evaluators in the clinic, suggesting the need for a training program to prepare experienced comorbidity evaluators at different institutions.…”

Section: Discussionmentioning

confidence: 99%

How I assess comorbidities before hematopoietic cell transplantation

Sorror

2013

Blood

197

135

View full text Add to dashboard Cite

The hematopoietic cell transplantationcomorbidity index (HCT-CI) is a comorbidity tool suited for recipients of HCT. The index has been shown to sensitively capture the prevalence and magnitude of severity of various organ impairments before HCT and to provide valuable prognostic information after HCT. Many investigators have validated the discriminative power of the HCT-CI, but others have not. One concern is the consistency in comorbidity coding across different evaluators, particularly in view of the relatively recent addition of the HCT-CI to the transplant evaluation process. In this article, comorbidity scoring was tested across different evaluators, and only a fair interobserver agreement rate could be detected. To address these issues, a brief training program is proposed here, consisting of systematic methodology for data acquisition and consistent guidelines for comorbidity coding that were summarized in a Webbased calculator. In a validation patient cohort, this training program was shown to improve the interevaluator agreement on HCT-CI scores to an excellent rate with weighted k values in the range of 0.89 to 0.97. This proposed training program will facilitate reliable assessment of comorbidities in the clinic and for research studies leading to standardization of the use of comorbidities in prediction of HCT outcomes. (Blood. 2013;121(15):2854-2863 IntroductionOrgan dysfunctions (comorbidities) were found to be associated with the outcome of treatment of a given primary disease 1,2 and, in particular, cancer. 3,4 In 2005, a hematopoietic cell transplantationcomorbidity index (HCT-CI) was introduced as a measure of organ dysfunctions that was suited for recipients of HCT. 5 The HCT-CI was developed from the historical Charlson comorbidity index 6 after introducing 3 conceptual changes: the use of laboratory and organ function tests to redefine pulmonary, hepatic, cardiac, and renal comorbidities; the inclusion of all comorbidities encountered in a cohort of HCT recipients at a single institution; and the estimation of new adjusted hazard ratios for the associations between comorbidities and nonrelapse mortality after HCT. These adjusted hazard ratios were then converted into weights that could be summated into a total score.In validation cohorts of recipients of allogeneic HCT from 2 different institutions, the HCT-CI was demonstrated to have higher discriminative power than the Charlson comorbidity index, both for non-relapse mortality and overall survival. 5,7 Many investigators reported on the valid association between HCT-CI scores and mortality in their respective single-institutions, [8][9][10][11][12][13][14][15][16][17] whereas a few others disagreed. [18][19][20][21][22] A discussion of the possible reasons for the lack of complete agreement by investigators on the validity of the HCT-CI is outside the scope of this article. Instead, the focus of this article is a single concern that is related to the degree of consistency in assigning comorbidity scores among evaluators. For example, a r...

show abstract

Section: Discussionmentioning

confidence: 99%

How I assess comorbidities before hematopoietic cell transplantation

Sorror

2013

Blood

197

135

View full text Add to dashboard Cite

show abstract

“…Interrater agreement and test‐retest agreement for all raters were at least substantial, supporting the stability of UMCT‐KE and UMCT‐KF ratings. Use of a well‐defined testing protocol and standardized instructions, and sufficient rater training are known to contribute to dependable results in reliability studies (Mong et al., 2010; Tuijin et al., 2012). However, wide confidence intervals with most of the lower limits extending down to moderate levels imply that further enhancements in test procedures and/or rater training may be needed.…”

Section: Discussionmentioning

confidence: 99%

Clinimetrics of the Upright Motor Control Test in chronic stroke

2017

View full text Add to dashboard Cite

IntroductionInsufficient literature exists regarding the clinimetric properties of the Upright Motor Control Test Knee Extension and Flexion subtests (UMCT‐KE and UMCT‐KF, respectively). This study examined the interrater and test‐retest reliability of these subtests, and determined the relationship between the UMCT‐KE and a clinical measure of muscle function in a sample of adults with chronic stroke.MethodsThree raters independently administered the UMCT‐KE and UMCT‐KF on adults with chronic stroke with spasticity/abnormal movement patterns. Testing of each participant occurred on two occasions (T1 and T2) separated by a two‐week interval. A fourth rater independently administered the Five Times Sit to Stand Test (FTSST), a measure of lower extremity muscle function (power), on T2.ResultsTwenty‐nine adults aged 55 ± 8 years, comprising 21 men (72%), and who were 9 ± 5 years poststroke, completed the study. Most of the participants (66%, 19/29) did not require an assistive device during walking. The UMCT‐KE and UMCT‐KF demonstrated substantial interrater reliability (W = 0.63–0.67 and 0.72–0.75, respectively) and substantial to almost perfect test‐retest reliability across the raters (W = 0.75–0.82 and 0.85–0.87, respectively). The UMCT‐KE showed positive inverse correlation with the FTSST (ρ = −0.52, p = .003).ConclusionsScores on both subtests are reproducible within raters and across different raters. The relationship of UMCT‐KE scores with FTSST scores implies that the UMCT‐KE can provide information that relates with the construct of muscle function in a weight‐bearing position.

show abstract

“…Many studies show that the training of professionals, improving diagnostic instruments or methodologies, and a combination of both training and methodology play significant roles in greater reliability 19 . Also, in TCM, two studies in patients with irritable bowel syndrome and rheumatoid arthritis demonstrated an improvement in the level of reliability with incorporation of a training phase in the study 20, 21.…”

Section: Discussionmentioning

confidence: 99%

Intrarater and interrater reliability of pulse examination in traditional Indian Ayurvedic medicine

Kurande

Waagepetersen

Toft

et al. 2013

Integrative Medicine Research

View full text Add to dashboard Cite

BackgroundIn Ayurveda, pulse examination (nadipariksha) is an important tool to assess the status of three doshas: vata, pitta, and kapha. Long historical use has been seen as a documentation of its efficacy; however, there is a lack of a quantitative measure of the reliability of the pulse examination method. The objective of this study was to test the intrarater and interrater reliability of pulse examination in Ayurveda.MethodsFifteen registered Ayurvedic doctors with 3–15 years of experience examined the pulse of 20 healthy volunteers twice, for a total of 600 examinations. The examinations were performed blind and in a random order. Only the current status of dosha-specific methods of pulse examination were considered. Cohen's weighted κ statistic was used as a measure of intrarater and interrater reliability, and a hypothesis of homogeneous diagnosis (random rating) was tested. Following this, we tested whether proportions of ratings were equal between doctors.ResultsAccording to the Landis and Koch scale, the level of reliability ranged from poor to moderate. It was observed that the doctors more frequently diagnosed a combination of two doshas than a single dosha. The κ values were generally larger for experienced doctors (p = 0.04).ConclusionExperience and proper training have important roles in pulse examination.

show abstract

Reducing interrater variability and improving health care: a meta‐analytical review

Cited by 37 publications

References 65 publications

How I assess comorbidities before hematopoietic cell transplantation

How I assess comorbidities before hematopoietic cell transplantation

Clinimetrics of the Upright Motor Control Test in chronic stroke

Intrarater and interrater reliability of pulse examination in traditional Indian Ayurvedic medicine

Contact Info

Product

Resources

About