DeepMed: A unified, modular pipeline for end-to-end deep learning in computational pathology

Abstract

The interpretation of digitized histopathology images has been transformed thanks to artificial intelligence (AI). End-to-end AI algorithms can infer high-level features directly from raw image data, extending the capabilities of human experts. In particular, AI can predict tumor subtypes, genetic mutations and gene expression directly from hematoxylin and eosin (H&E) stained pathology slides. However, existing end-to-end AI workflows are poorly standardized and not easily adaptable to new tasks. Here, we introduce DeepMed, a Python library for predicting any high-level attribute directly from histopathological whole slide images alone, or from images coupled with additional meta-data (https://github.com/KatherLab/deepmed). Unlike earlier computational pipelines, DeepMed is highly developer-friendly: its structure is modular and separates preprocessing, training, deployment, statistics, and visualization in such a way that any one of these processes can be altered without affecting the others. Also, DeepMed scales easily from local use on laptop computers to multi-GPU clusters in cloud computing services and therefore can be used for teaching, prototyping and for large-scale applications. Finally, DeepMed is user-friendly and allows researchers to easily test multiple hypotheses in a single dataset (via cross-validation) or in multiple datasets (via external validation). Here, we demonstrate and document DeepMed's abilities to predict molecular alterations, histopathological subtypes and molecular features from routine histopathology images, using a large benchmark dataset which we release publicly. In summary, DeepMed is a fully integrated and broadly applicable end-to-end AI pipeline for the biomedical research community.

Weakly supervised end-to-end artificial intelligence in gastrointestinal endoscopy

“…To improve explainability and identify potential biases in the system, we performed a systematic reader study, i.e. a manual review of the 60 highest scoring images (2 highest scoring images for each of the 30 highest scoring examinations) 28 for each of the 23 disease categories in the gastroscopy data set. A trained observer with experience in endoscopy (> 600 gastroscopy, > 300 colonoscopy examinations) categorized each of these 60 × 23 = 1380 images into one of five categories: (1) device (e.g.…”

“…All source codes for the deep learning training and deployment are publicly available under an open-source license at https://github.com/KatherLab/deepmed with an extensive user manual 28 .…”

Predicting the HER2 status in oesophageal cancer from tissue microarrays using convolutional neural networks

“…Several computational toolboxes currently allow for training predictive models on whole slide images (WSIs) stained using hematoxylin and eosin (H&E) [26][27][28][29]. We compared the results of our approach against CLAM [26], a publicly available pipeline for WSI classification.…”

Artificial intelligence to identify genetic alterations in conventional histopathology

“…Histological images are large and have to be tessellated before processing because the size of the WSI is too large [133]. The initial study by Coudray et al [29] established a simple yet powerful workflow for the prediction of molecular alterations from such WSIs: the ‘patch‐based’ weakly supervised workflows (‘vanilla workflow’).…”

“…The results of these approaches are typically similar [135]. In all approaches, tile‐level predictions are ultimately pooled for each slide by some type of averaging [133]. In general, these weakly supervised workflows are efficient because they only require a single ground truth for the whole slide.…”