Multiplexed tumor profiling with generative AI accelerates histopathology workflows and improves clinical predictions

Abstract

Understanding the spatial heterogeneity of tumors and its links to disease initiation and progression is a cornerstone of cancer biology. Presently, histopathology workflows heavily rely on hematoxylin & eosin (H&E) and serial immunohistochemistry (IHC) staining, a cumbersome, tissue-exhaustive process that results in non-aligned tissue images. We propose the VirtualMultiplexer, a generative AI toolkit that effectively synthesizes multiplexed IHC images for several antibody markers only from an input H&E image. The VirtualMultiplexer captures biologically relevant staining patterns across tissue scales without requiring consecutive tissue sections, image registration, or extensive expert annotations. Thorough qualitative and quantitative assessment indicates that the VirtualMultiplexer achieves rapid, robust and precise generation of virtually multiplexed imaging datasets of high staining quality that are indistinguishable from the real ones. The VirtualMultiplexer is successfully transferred across tissue-scales, patient cohorts, and cancer types with no need for model fine-tuning. Crucially, the virtually multiplexed images enabled training a Graph-Transformer that simultaneously learns from the joint spatial distribution of several proteins to predict clinically relevant endpoints. We observe that this multiplexed learning scheme was able to greatly improve the clinical prediction, as corroborated across several downstream tasks, independent patient cohorts and cancer types. Our results showcase the clinical relevance of AI-assisted multiplexed tumor imaging, accelerating histopathology workflows and cancer biology.