Leveraging medical Twitter to build a visual–language foundation model for pathology AI

Huang, Zhi; Bianchi, Federico; Yüksekgönül, Mert; Montine, Thomas J.; Zou, James

doi:10.1101/2023.03.29.534834

Cited by 2 publications

(1 citation statement)

References 49 publications

(62 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Though no foundational model has been trained that focuses solely on neuropathology, several general-purpose digital pathology models have been introduced. Huang et al developed a visual–language model based on the OpenPath dataset (208,414 image pairs), which originated from medical twitter [ 49 ]. The model was able to retrieve relevant images from text input.…”

Section: Semi-supervised and Self-supervised Machine Learningmentioning

confidence: 99%

Practical Application of Deep Learning in Diagnostic Neuropathology—Reimagining a Histological Asset in the Era of Precision Medicine

Rich,

Tosefsky,

Martin

et al. 2024

Cancers

View full text Add to dashboard Cite

In the past few decades, neuropathology has experienced several paradigm shifts with the introduction of new technologies. Deep learning, a rapidly progressing subfield of machine learning, seems to be the next innovation to alter the diagnostic workflow. In this review, we will explore the recent changes in the field of neuropathology and how this has led to an increased focus on molecular features in diagnosis and prognosis. Then, we will examine the work carried out to train deep learning models for various diagnostic tasks in neuropathology, as well as the machine learning frameworks they used. Focus will be given to both the challenges and successes highlighted therein, as well as what these trends may tell us about future roadblocks in the widespread adoption of this new technology. Finally, we will touch on recent trends in deep learning, as applied to digital pathology more generally, and what this may tell us about the future of deep learning applications in neuropathology.

show abstract

Section: Semi-supervised and Self-supervised Machine Learningmentioning

confidence: 99%

Practical Application of Deep Learning in Diagnostic Neuropathology—Reimagining a Histological Asset in the Era of Precision Medicine

Rich,

Tosefsky,

Martin

et al. 2024

Cancers

View full text Add to dashboard Cite

show abstract

Fostering transparent medical image AI via an image-text foundation model grounded in medical literature

Kim

Gadgil

DeGrave

et al. 2023

Preprint

View full text Add to dashboard Cite

Building trustworthy and transparent image-based medical AI systems requires the ability to interrogate data and models at all stages of the development pipeline: from training models to post-deployment monitoring. Ideally, the data and associated AI systems could be described using terms already familiar to physicians, but this requires medical datasets densely annotated with semantically meaningful concepts. Here, we present a foundation model approach, named MONET (Medical cONcept rETriever), which learns how to connect medical images with text and generates dense concept annotations to enable tasks in AI transparency from model auditing to model interpretation. Dermatology provides a demanding use case for the versatility of MONET, due to the heterogeneity in diseases, skin tones, and imaging modalities. We trained MONET on the basis of 105,550 dermatological images paired with natural language descriptions from a large collection of medical literature. MONET can accurately annotate concepts across dermatology images as verified by board-certified dermatologists, outperforming supervised models built on previously concept-annotated dermatology datasets. We demonstrate how MONET enables AI transparency across the entire AI development pipeline from dataset auditing to model auditing to building inherently interpretable models.

show abstract

Leveraging medical Twitter to build a visual–language foundation model for pathology AI

Cited by 2 publications

References 49 publications

Practical Application of Deep Learning in Diagnostic Neuropathology—Reimagining a Histological Asset in the Era of Precision Medicine

Practical Application of Deep Learning in Diagnostic Neuropathology—Reimagining a Histological Asset in the Era of Precision Medicine

Fostering transparent medical image AI via an image-text foundation model grounded in medical literature

Contact Info

Product

Resources

About