A deep learning framework for autonomous detection and classification of Crohnʼs disease lesions in the small bowel and colon with capsule endoscopy

Abstract

Background and study aims Small bowel ulcerations are efficiently detected with deep learning techniques, whereas the ability to diagnose Crohnʼs disease (CD) in the colon with it is unknown. This study examined the ability of a deep learning framework to detect CD lesions with pan-enteric capsule endoscopy (CE) and classify lesions of different severity. Patients and methods CEs from patients with suspected or known CD were included in the analysis. Two experienced gastroenterologists classified anonymized images into normal mucosa, non-ulcerated inflammation, aphthous ulceration, ulcer, or fissure/extensive ulceration. An automated framework incorporating multiple ResNet-50 architectures was trained. To improve its robustness and ability to characterize lesions, image processing methods focused on texture enhancement were employed. Results A total of 7744 images from 38 patients with CD were collected (small bowel 4972, colon 2772) of which 2748 contained at least one ulceration (small bowel 1857, colon 891). With a patient-dependent split of images for training, validation, and testing, ulcerations were diagnosed with a sensitivity, specificity, and diagnostic accuracy of 95.7 % (CI 93.4–97.4), 99.8 % (CI 99.2–100), and 98.4 % (CI 97.6–99.0), respectively. The diagnostic accuracy was 98.5 % (CI 97.5–99.2) for the small bowel and 98.1 % (CI 96.3–99.2) for the colon. Ulcerations of different severities were classified with substantial agreement (κ = 0.72). Conclusions Our proposed framework is in excellent agreement with the clinical standard, and diagnostic accuracies are equally high for the small bowel and colon. Deep learning approaches have a great potential to help clinicians detect, localize, and determine the severity of CD with pan-enteric CE.

Edge artificial intelligence wireless video capsule endoscopy

“…The main obstacle in adopting these networks for devices with limited computational resources such as our camera pill with a memory size of only 8MB, is their large size, as for instance, YOLO V3 being one of the smallest and most popular networks requires an approximate memory allocation of 237MB. In two independent studies 30 , 31 , we showed that ZF-Net based DNN as backbone for a Faster R-CNN to detect and localize colorectal polyps, and a ResNet-50 based convolutional neural network (CNN) to detect and classify lesions in Crohn’s disease in both small and large bowel required an approximate memory allocation of 375MB and 167MB, respectively. Keeping the constraints dictated by the memory allowance and network size in mind, we designed an optimized YOLO-based DNN of approximately 3.2MB to detect and localize colorectal polyps, and implemented the solution in our camera pill.…”

“…On the algorithmic side, we will improve our DNN to include various types of lesions, such as bleeding, or inflammatory bowel diseases. After pruning and optimization, our multi-lesion classifier based on a ResNet-50 CNN 31 will be implemented onboard, enabling us to detect and classify lesions in Crohn’s disease in both small and large bowel. Our DNN will also identify important anatomical landmarks (e.g., flexures), and we will replace the mock-up DNN with pruned networks trained for detecting a variety of lesions, e.g., bleeding of the GI tract.…”

Artificial Endoscopy and Inflammatory Bowel Disease: Welcome to the Future

“…DL methods for autonomous detection and classification of CD lesions have also been applied to panenteric capsule endoscopy system that is now available allowing simultaneous investigation of the small bowel and colon. AI technology has increased the diagnostic yield and reduced interobserver variability in this integrated procedure [56,57].…”

Key research questions for implementation of artificial intelligence in capsule endoscopy

“…AI is being implemented in several other areas of GI endoscopies, such as early detection of gastric neoplasia, 9 , 10 Barrett’s esophagus, 11 endoscopic ultrasound, 12 , 13 and grading of mucosal inflammation in ulcerative colitis. 14 – 18 An additional field with the fast development of AI research is capsule endoscopy (CE), with several publications evaluating deep learning for automated detection of inflammatory lesions, 19 – 26 vascular lesions, 27 , 28 protruding and neoplastic lesions/masses, 29 and scoring of bowel cleanliness. 30…”

“…[1][2][3][4][5][6][7][8] Some of the systems are already being used routinely across the world, even though real-world implementation results are still lacking. field with the fast development of AI research is capsule endoscopy (CE), with several publications evaluating deep learning for automated detection of inflammatory lesions, [19][20][21][22][23][24][25][26] vascular lesions, 27,28 protruding and neoplastic lesions/ masses, 29 and scoring of bowel cleanliness. 30 However, there are still multiple challenges in the way of implementation of the impressive experimental performance of AI in CE in clinical practice.…”