When to use agreement versus reliability measures

Vet, Henrica C.W. de; Terwee, Caroline B.; Knol, Dirk L.; Bouter, Lex M.

doi:10.1016/j.jclinepi.2005.10.015

Cited by 1,390 publications

(1,333 citation statements)

References 9 publications

Supporting

Mentioning

1,305

Contrasting

Unclassified

Order By: Relevance

“…The function “lmer” was used for dichotomous, ordinal, and continuous variables. Odds ratios (OR) were calculated and significance set using Fisher's exact test and the package “epicalc.”27 The intraclass correlation coefficient (ICC) was calculated as defined in formulas 1 to 4 (Data S1), all redefined 4, 28, 29…”

Section: Methodsmentioning

confidence: 99%

“…Examinations should therefore have good reliability (how well can patients be distinguished from each other), high agreement (low measurement error for repeated assessments)4 and results of the examination should be valid (accurate) 5. Inaccurate or unreliable assessment of an underlying problem might lead to misdiagnosis or inappropriate further testing and treatments.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Rater Agreement on Gait Assessment during Neurologic Examination of Horses

Olsen

Dunkel

Barker

et al. 2014

Veterinary Internal Medicne

View full text Add to dashboard Cite

BackgroundReproducible and accurate recognition of presence and severity of ataxia in horses with neurologic disease is important when establishing a diagnosis, assessing response to treatment, and making recommendations that might influence rider safety or a decision for euthanasia.ObjectivesTo determine the reproducibility and validity of the gait assessment component in the neurologic examination of horses.AnimalsTwenty‐five horses referred to the Royal Veterinary College Equine Referral Hospital for neurological assessment (n = 15), purchased (without a history of gait abnormalities) for an unrelated study (n = 5), or donated because of perceived ataxia (n = 5).MethodsUtilizing a prospective study design; a group of board‐certified medicine (n = 2) and surgery (n = 2) clinicians and residents (n = 2) assessed components of the equine neurologic examination (live and video recorded) and assigned individual and overall neurologic gait deficit grades (0–4). Inter‐rater agreement and assessment‐reassessment reliability were quantified using intraclass correlation coefficients (ICC).ResultsThe ICCs of the selected components of the neurologic examination ranged from 0 to 0.69. “Backing up” and “recognition of mistakes over obstacle” were the only components with an ICC > 0.6. Assessment‐reassessment agreement was poor to fair. The agreement on gait grading was good overall (ICC = 0.74), but poor for grades ≤ 1 (ICC = 0.08) and fair for ataxia grades ≥ 2 (ICC = 0.43). Clinicians with prior knowledge of a possible gait abnormality were more likely to assign a grade higher than the median grade.Conclusion and Clinical ImportanceClinicians should be aware of poor agreement even between skilled observers of equine gait abnormalities, especially when the clinical signs are subtle.

show abstract

Section: Methodsmentioning

confidence: 99%

mentioning

confidence: 99%

Rater Agreement on Gait Assessment during Neurologic Examination of Horses

Olsen

Dunkel

Barker

et al. 2014

Veterinary Internal Medicne

View full text Add to dashboard Cite

show abstract

“…The ideal scenario would be that the SDC is smaller than MIC [8], but unfortunately these values were not available in the literature.…”

Section: Reproducibility Thicknessmentioning

confidence: 99%

“…Agreement assesses how close the results of repeated measurements are, by estimating the measurement error in repeated measurements. Reliability assesses whether study subjects could be distinguished from each other, despite measurement errors [8,31].…”

Section: Introductionmentioning

confidence: 99%

An investigation of the reproducibility of ultrasound measures of abdominal muscle activation in patients with chronic non-specific low back pain

et al. 2009

View full text Add to dashboard Cite

Ultrasound (US) measures are used by clinicians and researchers to evaluate improvements in activity of the abdominal muscles in patients with low back pain. Studies evaluating the reproducibility of these US measures provide some information; however, little is known about the reproducibility of these US measures over time in patients with low back pain. The objectives of this study were to estimate the reproducibility of ultrasound measurements of automatic activation of the lateral abdominal wall muscles using a leg force task in patients with chronic low back pain. Thirty-five participants from an existing randomised, blinded, placebo-controlled trial participated in the study. A reproducibility analysis was undertaken from all patients using data collected at baseline and after treatment. The reproducibility of measurements of thickness, muscle activation (thickness changes) and muscle improvement/deterioration after intervention (differences in thickness changes from single images made before and after treatment) was analysed. The reproducibility of static images (thickness) was excellent (ICC 2,1 = 0.97, 95% CI = 0.96-0.97, standard error of the measurement (SEM) = 0.04 cm, smallest detectable change (SDC) = 0.11 cm), the reproducibility of thickness changes was moderate (ICC 2,1 = 0.72, 95% CI 0.65-0.76, SEM = 15%, SDC 41%), while the reproducibility of differences in thickness changes from single images with statistical adjustment for duplicate measures was poor (ICC 2,1 = 0.44, 95% CI 0.33-0.58, SEM = 21%, SDC = 66.5%). Improvements in the testing protocol must be performed in order to enhance reproducibility of US as an outcome measure for abdominal muscle activation.

show abstract

“…Through this relationship, kappa coefficients depend on the prevalence of the trait under study, which limits the possibility to compare them among studies with different prevalence. Several authors (Thompson and Walter, 1988; Feinstein and Cicchetti, 1990; Cicchetti and Feinstein, 1990; Byrt et al ., 1993; de Vet et al ., 2006) proposed the use of absolute agreement measures (e.g. the proportion of items classified in the same category by the two observers) to avoid that dependency.…”

Section: Introductionmentioning

confidence: 99%

Comparing dependent kappa coefficients obtained on multilevel data

Vanbelle

2017

Biometrical J

View full text Add to dashboard Cite

Reliability and agreement are two notions of paramount importance in medical and behavioral sciences. They provide information about the quality of the measurements. When the scale is categorical, reliability and agreement can be quantified through different kappa coefficients. The present paper provides two simple alternatives to more advanced modeling techniques, which are not always adequate in case of a very limited number of subjects, when comparing several dependent kappa coefficients obtained on multilevel data. This situation frequently arises in medical sciences, where multilevel data are common. Dependent kappa coefficients can result from the assessment of the same individuals at various occasions or when each member of a group is compared to an expert, for example. The method is based on simple matrix calculations and is available in the R package “multiagree”. Moreover, the statistical properties of the proposed method are studied using simulations. Although this paper focuses on kappa coefficients, the method easily extends to other statistical measures.

show abstract

When to use agreement versus reliability measures

Abstract: If the research question concerns the distinction of persons, reliability parameters are the most appropriate. But if the aim is to measure change in health status, which is often the case in clinical practice, parameters of agreement are preferred.

Cited by 1,390 publications

References 9 publications

Rater Agreement on Gait Assessment during Neurologic Examination of Horses

Rater Agreement on Gait Assessment during Neurologic Examination of Horses

An investigation of the reproducibility of ultrasound measures of abdominal muscle activation in patients with chronic non-specific low back pain

Comparing dependent kappa coefficients obtained on multilevel data

Contact Info

Product

Resources

About