FRI0597 Validation of web-based calibration modules for imaging scoring systems based on principles of artificial intelligence: the sparcc mri sacroiliac joint inflammation score

Abstract

BackgroundThe application and appropriate use of imaging-based scoring instruments is usually based on passive learning from published manuscripts while real-time interaction with instrument developers is often non-feasible. Moreover, most instruments lack knowledge transfer tools that would facilitate attainment of pre-specified performance targets for reader reliability.Objectives1. To develop a web-based calibration module for the SPARCC MRI SIJ Inflammation Score based on consensus scores from these instrument developers, experiential game psychology, and real-time iterative feedback. 2. To test the feasibility and attainment of pre-specified performance targets for reader reliability.MethodsThe scoring of inflammatory lesions of the SIJ on MRI using the SPARCC method is based on SIJ quadrants and the calibration module is comprised of 50 DICOM cases, each with scans from baseline and 12 weeks after the start of TNF inhibitor therapy. Scans are scored blinded-to-time-point. Continuous visual real-time feedback regarding concordance/discordance of scoring per SIJ quadrant with expert readers is provided by a color-coding scheme. Reliability is additionally assessed by real-time intra-class correlation coefficient with the first ICC data being provided after 20 cases. Accreditation for SPARCC BME score is achieved with status and change score ICC of >0.8 and>0.7 and is based on the final 20 cases. 26 readers scored the SPARCC BME module (7 rheumatology fellows, 2 chiropracters, 1 undergraduate, 8 rheumatologists, 8 radiologists) with 21 having no prior experience. Feasibility was assessed by 8-item survey.ResultsThe majority of readers achieved accreditation for SPARCC BME score on the basis of sufficient reliability with instrument developers for both status and change scores, irrespective of prior experience (table 1). All readers who completed the module a second time, 6 months after the first exposure, achieved accreditation for SPARCC BME score. All readers rated the modules as easy and intuitive with average time for reading each case for SPARCC BME being 8 min.Abstract FRI0597 – Table 1*Proficiency targets for reader reliability**7 rheumatology fellows, 2 chiropractors, 1 undergraduateConclusionsExperiential web-based learning is an effective and feasible calibration tool to achieve proficiency targets in the scoring of MRI scans for SIJ inflammatory lesions.Disclosure of InterestW. Maksymowych Shareholder of: CaRE Arthritis, S. Krabbe: None declared, D. Biko: None declared, P. Weiss: None declared, M. Maksymowych: None declared, J. Cheah: None declared, G. Kröber: None declared, U. Weber: None declared, K. Danebod: None declared, P. Bird: None declared, P. Chiowchanwisawakit: None declared, J. Moeller: None declared, M. Francavilla: None declared, J. Stimec: None declared, T. Kogay: None declared, V. Zubler: None declared, M. Battish: None declared, N. Winn: None declared, D. Rumsey: None declared, R. Guglielmi: None declared, S. Pedersen: None declared, H. Boutrup: None declared, S. Shafer: None declare...

Spondyloarthritis Research Consortium of Canada sacroiliac joint inflammation and structural scores: change score reliability and recalibration utility in children

“…Although most confidence intervals overlapped between the two scoring exercises, there was a notable and consistent trend of [1,2]. These pre-specified cutoffs reflect that change score reliability was expected to be lower than the status scores based on prior studies [11]. Further, the reliability for the SSS lesions was anticipated to be lower than for SIS lesions due to the smaller degree of change in structural lesions and hence greater difficulty in their reliable detection.…”

“…Pre-specified acceptability thresholds for change scores were based upon standards listed on the CaREarthritis platform and acceptability thresholds used in prior published validation studies of the SPARCC SIS and SSS [1,2]. Reliability for change scores is typically lower than for status scores when rating adult studies [11]. Hence, the lower pre-specified cut-offs than for status scores.…”

“…First, the scoring of sclerosis is not included in the RETIC module since sclerosis is not part of the SSS scoring method in adults. Second, the reliability for change scores is typically lower than for status scores when rating adult studies [11], hence the pre-specified cut-offs of > 0.7 and > 0.5 for the SIS and SSS components, respectively. Third, scoring of change in structural lesions is less reliable as estimated using the ICC due to the much slower evolution of structural lesions.…”

“…Even raters experienced in scoring adult SIJ may benefit from enhanced calibration to optimize interreader reliability. Real-time iterative calibration (RETIC) with real-time feedback is a new concept in imaging research based on principles of artificial intelligence that aims to enhance reader-expert calibration using a Web-based interactive scoring interface [11]. This provides continuous visual real-time feedback to readers using a color-coding scheme regarding concordance/discordance of their scores for MRI lesions with expert reader scores that are embedded in the scoring platform.…”

“…Intraclass correlation coefficient (ICC) with expert reader scores, including change scores, was provided after the first 10 cases, then again after the next 10 cases. A priori ICC acceptability thresholds for this calibration exercise were > 0.7 for SIS and > 0.5 for all SSS change scores [11]. After the interactive training module, we conducted a second scoring exercise.…”