Does the Clinical Context Improve the Reliability of Rheumatologists Grading Digital Ulcers in Systemic Sclerosis?

Abstract

ObjectiveDigital ulcers (DUs) are often a primary end point in systemic sclerosis (SSc; scleroderma) clinical trials, although the reliability of rheumatologists grading DUs is poor to moderate at best. DU assessment in recent trials has been based upon visual inspection alone, which potentially misses “real‐world” clinical contextual information. Our aim was to investigate whether this clinical information improves the reliability of rheumatologists grading DUs. A secondary aim was to assess agreement between patients and rheumatologists.MethodsEighty images of a range of digital lesions were collected from patients with SSc with the clinical context: pain (severity and temporal relationship), lesion duration, and discharge (patient reported and clinician observed). Raters received all images either with or without the clinical context, and graded these images (using a custom‐built interface) on an ordinal scale of severity: no ulcer, inactive ulcer, or active ulcer. Patients also graded their lesion(s) on the same scale.ResultsFifty‐one rheumatologists from 15 countries completed the study (26 without and 25 with context): 4,590 (including 510 repeated) image gradings were obtained. Context did not significantly increase (without and with context) either intra‐ (0.64, 0.71) or interrater (0.32, 0.36) reliability. Pain (visual analog scale and temporal relationship) and discharge (patient reported and clinician observed) were associated with increased lesion severity, and duration with reduced severity. Agreement between individual patients and rheumatologists was poor without and with context (0.19, 0.28).ConclusionThe overall intra‐ and interrater reliability of DU grading did not significantly improve with the clinical context. Agreement between patients and rheumatologists was poor.

Reliability of digital ulcer definitions as proposed by the UK Scleroderma Study Group: A challenge for clinical trial design

“…The overall inter-rater reliability was lower than previously reported. 1–3 In our previous web-based DU reliability study, 3 the addition of ‘real-world’ (e.g. pain and discharge) clinical contextual information (which did not include information on previous debridements) did not significantly increase the inter-rater reliability (κ = 0.32 without or 0.36 with the contextual information).…”

“…In a recent reliability study, the agreement between individual patients and rheumatologists was poor with and without the clinical context (0.28 and 0.19, respectively). 3 The appearance of digital lesions does not always correlate with patient symptoms and there is a major unmet need to develop patient-reported outcome (PRO) instruments for capturing the multifaceted patient experience of SSc-DU to facilitate future clinical trials.…”

Defining Skin Ulcers in Systemic Sclerosis: Systematic Literature Review and Proposed World Scleroderma Foundation (WSF) Definition

“…A recent study by Hughes et al (32), evaluated whether the reliability among rheumatologists grading DUs improves by providing the assessor with clinical information. They used 80 images and 51 rheumatologists (web based).…”

“…The addition of clinical information did not statistically improve the inter- or intra-rater reliability. They concluded that the inter- and intra-rater reliability of DU grading did not improve by providing patients’ clinical context (32). This study emphasizes the need for a more uniform definition that is widely acceptable and reliable.…”

Quantifying Digital Ulcers in Systemic Sclerosis: Reliability of Computer‐Assisted Planimetry in Measuring Lesion Size

“…Photographs of 107 finger lesions were collected from 2 ongoing studies of SSc‐related digital ulcers (one of which is described in ref. ). All finger lesions were accepted into the study.…”