Deep Multimodal Case–Based Retrieval for Large Histopathology Datasets

Jiménez–del–Toro, Oscar; Otálora, Sebastian; Atzori, Manfredo; Müller, Henning

doi:10.1007/978-3-319-67434-6_17

Cited by 15 publications

(12 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Whole slide image (WSI) scanning now enables the on-screen visualization of high-resolution images from patient tissue slides. This opens the door to a wide range of image analysis and image navigation solutions [2], similar to what happened in the field of radiology, but with images of a much larger size in resolution up to 100, 000 2 pixels. With fully digital workflows, pathologists obtain faster access to relevant patient data stored in such hospital repositories [2].…”

Section: Introductionmentioning

confidence: 92%

Deep Learning Based Retrieval System for Gigapixel Histopathology Cases and Open Access Literature

Otálora

Schaer²,

Jiménez–del–Toro³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Clinical practice is getting increasingly stressful for pathologists due to increasing complexity and time constraints. Histopathology is slowly shifting to digital pathology, thus creating opportunities to allow pathologists to improve reading quality or save time using Artificial Intelligence (AI)-based applications. We aim to enhance the practice of pathologists through a retrieval system that allows them to simplify their workflow, limit the need for second opinions, while also learning in the process. In this work, an innovative retrieval system for digital pathology is integrated within a Whole Slide Image (WSI) viewer, allowing to define regions of interest in images as queries for finding visually similar areas using deep representations. The back-end similarity computation algorithms are based on a multimodal approach, allowing to exploit both text information and content-based image features. Shallow and deep representations of the images were evaluated, the later showed a better overall retrieval performance in a set of 112 whole slide images from biopsies. The system was also tested by pathologists, highlighting its capabilities and suggesting possible ways to improve it and make it more usable in clinical practice. The retrieval system developed can enhance the practice of pathologists by enabling them to use their experience and knowledge to properly control artificial intelligence tools for navigating repositories of images for decision support purposes.

show abstract

Section: Introductionmentioning

confidence: 92%

Deep Learning Based Retrieval System for Gigapixel Histopathology Cases and Open Access Literature

Otálora

Schaer²,

Jiménez–del–Toro³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this paper, we propose a novel CBHIR framework for diagnostically relevant region retrieval from largescale WSI-database based on graph convolutional network (GCN) [45] and hashing method. Different from the present sub-regions retrieval frameworks [23,30,57,58], we propose constructing graphs for the sub-regions in the WSI to describe the structural information within the regions and employing GCNs with differentiable pooling modules to encode the graph features into uniform retrieval indexes. Both the local features and structural information in the regions are effectively preserved in the encoding.…”

Section: Introductionmentioning

confidence: 99%

Histopathology WSI Encoding based on GCNs for Scalable and Efficient Retrieval of Diagnostically Relevant Regions

Zheng,

Jiang,

Zhang

et al. 2021

Preprint

View full text Add to dashboard Cite

Content-based histopathological image retrieval (CB-HIR) has become popular in recent years in the domain of histopathological image analysis. CBHIR systems provide auxiliary diagnosis information for pathologists by searching for and returning regions that are contently similar to the region of interest (ROI) from a pre-established database. While, it is challenging and yet significant in clinical applications to retrieve diagnostically relevant regions from a database that consists of histopathological whole slide images (WSIs) for a query ROI. In this paper, we propose a novel framework for regions retrieval from WSI-database based on hierarchical graph convolutional networks (GCNs) and Hash technique. Compared to the present CBHIR framework, the structural information of WSI is preserved through graph embedding of GCNs, which makes the retrieval framework more sensitive to regions that are similar in tissue distribution. Moreover, benefited from the hierarchical GCN structures, the proposed framework has good scalability for both the size and shape variation of ROIs. It allows the pathologist defining query regions using free curves according to the appearance of tissue. Thirdly, the retrieval is achieved based on Hash technique, which ensures the framework is efficient and thereby adequate for practical large-scale WSI-database. The proposed method was validated on two public datasets for histopathological WSI analysis and compared to the state-of-the-art methods. The proposed method achieved mean average precision above 0.857 on the ACDC-LungHP dataset and above 0.864 on the Camelyon16 dataset in the irregular region retrieval tasks, which are superior to the state-of-the-art methods. The average retrieval time from a database within 120 WSIs is 0.802 ms.

show abstract

“…However, their retrieval targets are diagnostically relevant regions, which are not predefined, making the task unnecessarily challenging. Jimenez-del-Toro et al propose a retrieval system for whole-slide histopathology images utilizing both the image and its diagnostic report information [48]. Fusing multi-modality information can boost the discrimination ability of the fused representation.…”

Section: Introductionmentioning

confidence: 99%

Interactive thyroid whole slide image diagnostic system using deep representation

Chen

Shi

Liang

et al. 2020

Computer Methods and Programs in Biomedicine

View full text Add to dashboard Cite

Background and objectives-The vast size of the histopathology whole slide image poses formidable challenges to its automatic diagnosis. With the goal of computer-aided diagnosis and the insights that suspicious regions are generally easy to identify in thyroid whole slide images (WSIs), we develop an interactive whole slide diagnostic system for thyroid frozen sections based on the suspicious regions preselected by pathologists.Methods-We propose to generate feature representations for the suspicious regions via extracting and fusing patch features using deep neural networks. We then evaluate region classification and retrieval on four classifiers and three supervised hashing methods based on the feature representations. The code is released at https://github.com/PingjunChen/ ThyroidInteractive.Results-We evaluate the proposed system on 345 thyroid frozen sections and achieve 96.1% cross-validated classification accuracy, and retrieval mean average precision (MAP) of 0.972.Conclusions-With the participation of pathologists, the system possesses the following four notable advantages compared to directly handling whole slide images: 1) Reduced interference of irrelevant regions; 2) Alleviated computation and memory cost. 3) Fine-grained and precise suspicious region retrieval. 4) Cooperative relationship between pathologists and the diagnostic *

show abstract

Deep Multimodal Case–Based Retrieval for Large Histopathology Datasets

Cited by 15 publications

References 16 publications

Deep Learning Based Retrieval System for Gigapixel Histopathology Cases and Open Access Literature

Deep Learning Based Retrieval System for Gigapixel Histopathology Cases and Open Access Literature

Histopathology WSI Encoding based on GCNs for Scalable and Efficient Retrieval of Diagnostically Relevant Regions

Interactive thyroid whole slide image diagnostic system using deep representation

Contact Info

Product

Resources

About