A visual-language foundation model for computational pathology

Lu, Ming Y.; Chen, Bowen; Williamson, Drew F. K.; Chen, Richard J.; Liang, Ivy; Ding, Tong; Jaume, Guillaume; Odintsov, Igor; Le, Long Phi; Gerber, Georg; Parwani, Anil V.; Zhang, Andrew; Mahmood, Faisal

doi:10.1038/s41591-024-02856-4

Cited by 18 publications

(1 citation statement)

References 68 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The EchoCLIP model uses a ConvNeXt-Base 26 image encoder and a Byte-Pair Encoding text tokenizer 27 . The text encoder architecture is a decoder-only transformer identical to the architecture used by the original CLIP paper 23 and has an input context length of 77 tokens. Despite not being directly trained on specific interpretation tasks, EchoCLIP can accurately identify implanted devices as well as assess cardiac form and function (Table 2 ).…”

Section: Resultsmentioning

confidence: 99%

Vision–language foundation model for echocardiogram interpretation

Christensen,

Vukadinovic,

Yuan

et al. 2024

Nat Med

View full text Add to dashboard Cite

The development of robust artificial intelligence models for echocardiography has been limited by the availability of annotated clinical data. Here, to address this challenge and improve the performance of cardiac imaging models, we developed EchoCLIP, a vision–language foundation model for echocardiography, that learns the relationship between cardiac ultrasound images and the interpretations of expert cardiologists across a wide range of patients and indications for imaging. After training on 1,032,975 cardiac ultrasound videos and corresponding expert text, EchoCLIP performs well on a diverse range of benchmarks for cardiac image interpretation, despite not having been explicitly trained for individual interpretation tasks. EchoCLIP can assess cardiac function (mean absolute error of 7.1% when predicting left ventricular ejection fraction in an external validation dataset) and identify implanted intracardiac devices (area under the curve (AUC) of 0.84, 0.92 and 0.97 for pacemakers, percutaneous mitral valve repair and artificial aortic valves, respectively). We also developed a long-context variant (EchoCLIP-R) using a custom tokenizer based on common echocardiography concepts. EchoCLIP-R accurately identified unique patients across multiple videos (AUC of 0.86), identified clinical transitions such as heart transplants (AUC of 0.79) and cardiac surgery (AUC 0.77) and enabled robust image-to-text search (mean cross-modal retrieval rank in the top 1% of candidate text reports). These capabilities represent a substantial step toward understanding and applying foundation models in cardiovascular imaging for preliminary interpretation of echocardiographic findings.

show abstract

Section: Resultsmentioning

confidence: 99%