Central Reading of Ulcerative Colitis Clinical Trial Videos Using Neural Networks

Abstract

BACKGROUND AND AIMS: Endoscopic disease activity scoring in ulcerative colitis (UC) is useful in clinical practice but done infrequently. It is required in clinical trials, where it is expensive and slow because human central readers are needed. A machine learning algorithm automating the process could elevate clinical care and facilitate clinical research. Prior work using single-institution databases and endoscopic still images has been promising. METHODS: Seven hundred and ninety-five full-length endoscopy videos were prospectively collected from a phase 2 trial of mirikizumab with 249 patients from 14 countries, totaling 19.5 million image frames. Expert central readers assigned each full-length endoscopy videos 1 endoscopic Mayo score (eMS) and 1 Ulcerative Colitis Endoscopic Index of Severity (UCEIS) score. Initially, video data were cleaned and abnormality features extracted using convolutional neural networks. Subsequently, a recurrent neural network was trained on the features to predict eMS and UCEIS from individual full-length endoscopy videos. RESULTS: The primary metric to assess the performance of the recurrent neural network model was quadratic weighted kappa (QWK) comparing the agreement of the machine-read endoscopy score with the human central reader score. QWK progressively penalizes disagreements that exceed 1 level. The model's agreement metric was excellent, with a QWK of 0.844 (95% confidence interval, 0.787-0.901) for eMS and 0.855 (95% confidence interval, 0.80-0.91) for UCEIS. CONCLUSIONS: We found that a deep learning algorithm can be trained to predict levels of UC severity from full-length endoscopy videos. Our data set was prospectively collected in a multinational clinical trial, videos rather than still images were used, UCEIS and eMS were reported, and machine learning algorithm performance metrics met or exceeded those previously published for UC severity scores.

Training and deploying a deep learning model for endoscopic severity grading in ulcerative colitis using multicenter clinical trial data

“… Graphical summary of the data set characteristics and the experimental results reported by Gottlieb and colleagues, 23 Ozawa and colleagues, 21 Stidham and colleagues, 20 Yao and colleagues, 22 and by this study. …”

“…The difference on performance between the models trained on highresolution data obtained from single-center data sets (Stidham and colleagues, Ozawa and colleagues) versus those evaluated on clinical trials (Gottlieb and colleagues, Yao and colleagues, Gutierrez and colleagues), suggest that singlecenter studies might be over optimistic in their Figure 8. Graphical summary of the data set characteristics and the experimental results reported by Gottlieb and colleagues, 23 Ozawa and colleagues, 21 Stidham and colleagues, 20 Yao and colleagues, 22 and by this study.…”

“…[27][28][29] In Figure 8, we present a comparison between our approach and the previously published studies on automatic MCES prediction by Ozawa and colleagues, 21 Stidham and colleagues, 20 Yao and colleagues, 22 and the concurrent work of Gottlieb and colleagues. 23 Our motivation is to enable a fair assessment of our approach in comparison to previous efforts, but it is important to note that the experimental conditions between these studies are remarkably different.…”

“…Independent studies have explored the feasibility of developing AI systems to assess the severity of UC from still images. [19][20][21] The recent publication by Yao and colleagues 22 and the concurrent work to ours by Gottlieb and colleagues 23 evaluated the use of an MCES scoring system for unaltered endoscopic videos obtained from clinical trials. These studies demonstrated not only that evaluating full videos poses a greater challenge when compared to the grading of still frames but also shown that a significant drop in performance is to be expected when a model is evaluated on an external set of colonoscopies obtained from a multicenter study as part of a clinical trial.…”

Artificial intelligence in gastrointestinal endoscopy for inflammatory bowel disease: a systematic review and new horizons

“…Endoscopic AI diagnosis has also been used in two later randomized controlled trials to overcome the inter-observer variability affecting the off-site reading of pre and post-treatment endoscopic videos. In a branch study of a phase II multicenter, randomized, double-blind, parallel placebo controlled study of mirikizumab, Gottlieb et al 17 developed a CNN system to assess mucosal activity according to MES and UCEIS. A total of 795 prospectively recorded full-length endoscopy procedure videos were adopted for the training set (80% of total video frames) and a hold-out test (20% of total video frames).…”