TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification

Shao, Zhuchen; Bian, Hao; Chen, Yang; Wang, Yifeng; Zhang, Jian; Ji, Xiangyang; zhang, yongbing

doi:10.48550/arxiv.2106.00908

Cited by 9 publications

(18 citation statements)

References 28 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the current MIL methods assume that the instances in each bag are independent and identically distributed, thereby neglecting the correlation among different instances. Shao et al [239] present TransMIL to explore both morphological and spatial information in weakly supervised WSI classification. Specifically, TransMIL aggregates morphological information with two transformer-based modules and a position encoding layer as shown in Fig.…”

Section: Tumor Classificationmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

show abstract

Section: Tumor Classificationmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Another example is mi-Net [45], which pools predictions from single instances to derive a bag-level prediction. Other architectures adopted to MIL scenarios includes capsule networks [46], transformer [39] and graph neural networks [44]. Moreover, many works focus on the attentionbased pooling operators, like AbMILP introduced in [20] that weights the instances embeddings to obtain a bag embedding.…”

Section: Multiple Instance Learningmentioning

confidence: 99%

“…Table 3: Our ProtoMIL achieves state-of-the-art results on the Camelyon16 dataset in terms of AUC metric, surpassing even the transformer-based architecture. Notice that values for comparison marked with "*" and "**" are taken from [25] and [39], respectively.…”

Section: Positive Bagmentioning

confidence: 99%

“…Results. We compare ProtoMIL to other state-of-the-art MIL techniques, including both traditional mean and max MIL pooling, RNN, attention-based MIL pooling, and transformer-based MIL pooling [39]. ProtoMIL performs on par in terms of accuracy and slightly outperforms other models on AUC metric (Table 3) while providing a better understanding of its decision process, as presented in Figure 7 and Supplementary Materials.…”

Section: Camelyon16 Datasetmentioning

confidence: 99%

“…In recent years, with the rapid development of deep learning, MIL is combined with many neural network-based models [15,20,25,28,35,39,40,44,45,46]. Many of them embed all instances of the bag using a convolutional block of a deep network and then aggregate those embed-dings.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ProtoMIL: Multiple Instance Learning with Prototypical Parts for Whole-Slide Image Classification

Rymarczyk¹,

Pardyl²,

Jarosław³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multiple Instance Learning (MIL) gains popularity in many real-life machine learning applications due to its weakly supervised nature. However, the corresponding effort on explaining MIL lags behind, and it is usually limited to presenting instances of a bag that are crucial for a particular prediction. In this paper, we fill this gap by introducing ProtoMIL, a novel self-explainable MIL method inspired by the case-based reasoning process that operates on visual prototypes. Thanks to incorporating prototypical features into objects description, ProtoMIL unprecedentedly joins the model accuracy and fine-grained interpretability, which we present with the experiments on five recognized MIL datasets.

show abstract

Artificial intelligence to identify genetic alterations in conventional histopathology

Cifci

Foersch

Kather

2022

The Journal of Pathology

View full text Add to dashboard Cite

Precision oncology relies on the identification of targetable molecular alterations in tumor tissues. In many tumor types, a limited set of molecular tests is currently part of standard diagnostic workflows. However, universal testing for all targetable alterations, especially rare ones, is limited by the cost and availability of molecular assays. From 2017 to 2021, multiple studies have shown that artificial intelligence (AI) methods can predict the probability of specific genetic alterations directly from conventional hematoxylin and eosin (H&E) tissue slides. Although these methods are currently less accurate than gold standard testing (e.g. immunohistochemistry, polymerase chain reaction or next-generation sequencing), they could be used as pre-screening tools to reduce the workload of genetic analyses. In this systematic literature review, we summarize the state of the art in predicting molecular alterations from H&E using AI. We found that AI methods perform reasonably well across multiple tumor types, although few algorithms have been broadly validated. In addition, we found that genetic alterations in FGFR, IDH, PIK3CA, BRAF, TP53, and DNA repair pathways are predictable from H&E in multiple tumor types, while many other genetic alterations have rarely been investigated or were only poorly predictable. Finally, we discuss the next steps for the implementation of AI-based surrogate tests in diagnostic workflows.

show abstract

TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification

Cited by 9 publications

References 28 publications

Transformers in Medical Imaging: A Survey

Transformers in Medical Imaging: A Survey

ProtoMIL: Multiple Instance Learning with Prototypical Parts for Whole-Slide Image Classification

Artificial intelligence to identify genetic alterations in conventional histopathology

Contact Info

Product

Resources

About