An Unsupervised Graph Embeddings Approach to Multiplex Immunofluorescence Image Exploration

Abstract

Understanding the complex biology of the tumor microenvironment (TME) is necessary to understand the mechanisms of action of immuno-oncology therapies and to match the right therapies to the right patients. Multiplex immunofluorescence (mIF) is a useful technology that has tremendous potential to further our understanding of cancer patho-biology; however, tools that fully leverage the high dimensionality of this data are still in their infancy. We describe here a novel deep learning pipeline aimed to allow Graph-based Inspection of Tissues via Embeddings, GraphITE. GraphITE transforms mIF data into a graph representation, where unsupervised learning algorithms can be utilised to generate embeddings representing cellular `neighbourhoods'. The embeddings can be downprojected and explored for clustering analysis, and patterns can be mapped back to the image as well as interrogated for phenotypical, morphological, or structural distinctiveness. GraphITE supports the extraction of information not only on the phenotypes of individual cells or the relationships between specific cell types, but is able to characterize cell neighborhoods to look for more complex interactions, thereby allowing pathologists and data scientists to explore mIF data sets, uncovering patterns that are otherwise obscured by the high-dimensionality of the data. In this work, we showcase the current setup of the system, going from raw input data all the way to a user friendly exploration tool. Using this tool, we show how the data can be navigated in a way previously not possible.

Unsupervised discovery of tissue architecture in multiplexed imaging

“…Our method is generally applicable to images of cells in their native tissue context collected via highly multiplexed single-cell imaging data such as codetection by indexing (CODEX), cyclic immunofluorescence (CyCIF), imaging mass cytometry (IMC), multiplexed ion beam imaging (MIBI) and likewise multiplexed spatial platforms. The central aspect of UTAG is the combination of two employed deep learning of graphs of cellular proximity with cellular phenotypes for cell type prediction 19 , inference of cellular communication 20 and data exploration 21 . These models are computationally expensive to train and their results heavily depend on training data, which may preclude joint analysis of expression and morphological features across studies and data types.…”

Interpretable spatial cell learning enhances the characterization of patient tissue microenvironments with highly multiplexed imaging data

“…To overcome this challenge, there has been increased interest in applying graph neural networks (GNN) [12,13] to spatial cell analysis, in which both cell marker expressions and spatial information are taken into consideration [14][15][16][17][18][19][20]. Some methods focus only on tissue-scale classification [14] or cell phenotype annotation [19]; other methods have been developed for microenvironment analysis [15,16]; and finally, some are directed at general tissue structure classification, through integrating GNN and unsupervised learning algorithms [17,18]. However, these methods solely focus on either patient-level outcome [14] or cell-scale analysis [15][16][17][18][19].…”

“…Some methods focus only on tissue-scale classification [14] or cell phenotype annotation [19]; other methods have been developed for microenvironment analysis [15,16]; and finally, some are directed at general tissue structure classification, through integrating GNN and unsupervised learning algorithms [17,18]. However, these methods solely focus on either patient-level outcome [14] or cell-scale analysis [15][16][17][18][19].…”

SPACE-GM: geometric deep learning of disease-associated microenvironments from multiplex spatial protein profiles

“…There has been increased interest in applying graph-based deep learning methods to spatial cellular structures in recent literature [14][15][16] . Graph neural networks 17,18 (GNNs), a class of deep learning methods designed for graph structures, have been applied to a variety of analysis tasks, including cell type prediction 19 , representation learning 20 , cellular communication modeling 21 and tissue structure detection 22 . As most of these methods are designed for cellular property modeling, there still exists a gap between cellular-level graph analysis and patient-level phenotypes.…”