Interactive phenotyping of large-scale histology imaging data with HistomicsML

Abstract

Whole-slide imaging of histologic sections captures tissue microenvironments and cytologic details in expansive high-resolution images. These images can be mined to extract quantitative features that describe histologic elements, yielding measurements for hundreds of millions of objects. A central challenge in utilizing this data is enabling investigators to train and evaluate classification rules for identifying objects related to processes like angiogenesis or immune response. Here we present HistomicsML, an interactive machinelearning framework for large whole-slide imaging data. HistomicsML uses active learning direct user feedback, making classifier training efficient and scalable in datasets containing 10 8 + histologic objects. We demonstrate how HistomicsML can be used to phenotype microvascular structures in gliomas to predict survival, and to explore the molecular pathways associated with these phenotypes. Our approach enables researchers to unlock phenotypic information from digital pathology datasets to investigate prognostic image biomarkers and genotype-phenotype associations. INTRODUCTIONSlide scanning microscopes can digitize entire histologic sections at 200X-400X magnification, generating expansive high-resolution images containing 10 9 + pixels. For cancer tissues, these images contain important biologic and prognostic information, capturing the diverse cytologic elements involved in angiogenesis, immune response, and tumor/stroma interactions. Image analysis algorithms can mine wholeslide images to delineate objects like cell nuclei, and to extract 10s-100s of quantitative features that describe the shape, color, and texture of each object. These histology-omic or "histomic" features can be used to train machine-learning algorithms to classify important elements like tumor-infiltrating lymphocytes, vascular endothelial cells, or fibroblasts. Identifying these elements in tissues requires considerable expertise, and imparting this knowledge to algorithms enables precise characterization of large imaging datasets in ways not possible by subjective visual assessment. Quantitative measures of the abundance, morphologies and spatial patterns of these elements can help investigators understand relationships between histologic phenotypes and survival, treatment response, and underlying molecular mechanisms. Studies that generate whole slide images can yield histomic annotations of 10 8 + objects, and a central challenge in utilizing this data is in enabling domain experts to train classification rules and to evaluate their accuracy. With each image containing up to 10 6 + discrete objects, facilitating interaction with domain experts requires fluid navigation of gigapixel images, visualization of derived image segmentation boundaries, mechanisms to intelligently acquire training data from experts, and to visualize classifications for millions of objects.Histopathology image analysis has received significant attention with algorithms having been developed to predict metastasis (1), survival (...

Report on computational assessment of Tumor Infiltrating Lymphocytes from the International Immuno-Oncology Biomarker Working Group

“…In a research environment, it is not unusual for computational algorithms to be frequently tweaked in a closedloop fashion. This tweaking can be as simple as altering hyperparameters, but can include more drastic changes like modifications to the algorithm or (inter)active machine learning 83,84 . From a standard regulatory perspective, this is problematic as validation requires a defined "lockdown" and version control; any change generally requires at least partial re-validation 64,85 .…”

Glioma Through the Looking GLASS: the Glioma Longitudinal Analysis consortium, molecular evolution of diffuse gliomas

“…On the basis of this information, the GLASS expert pathology committee will formulate a (tentative) review diagnosis and thereby select patient samples that can be used for further study by the GLASS consortium. Corresponding whole-slide images of all patient samples that are included in GLASS will then be made available as a digital resource for further image analysis 102,103 .…”

ARA: accurate, reliable and active histopathological image classification framework with Bayesian deep learning

“…Variants of active learning depend on the way the new samples are selected. One method of choice is selection which maximises the diversity of the training set 37 . Gal et al 38 proposed an active learning procedure for Bayesian deep learning models, where new samples are added in each iteration based on their uncertainty estimated using variational dropout.…”

“…This technique can be conceptualised by an analogy to a diligent student, who while taking a course actively asks the teacher for more examples on topics which are hard for them to understand. There are several attempts at active learning for histopathological image classification in the literature, using both traditional machine learning 37,[39][40][41][42] and deep learning [43][44][45][46] . However, none of these approaches utilised uncertainty for selection of new samples in active training.…”