Multimodal single cell data integration challenge: results and lessons learned

Abstract

Biology has become a data-intensive science. Recent technological advances in single-cell genomics have enabled the measurement of multiple facets of cellular state, producing datasets with millions of single-cell observations. While these data hold great promise for understanding molecular mechanisms in health and disease, analysis challenges arising from sparsity, technical and biological variability, and high dimensionality of the data hinder the derivation of such mechanistic insights. To promote the innovation of algorithms for analysis of multimodal single-cell data, we organized a competition at NeurIPS 2021 applying the Common Task Framework to multimodal single-cell data integration. For this competition we generated the first multimodal benchmarking dataset for single-cell biology and defined three tasks in this domain: prediction of missing modalities, aligning modalities, and learning a joint representation across modalities. We further specified evaluation metrics and developed a cloud-based algorithm evaluation pipeline. Using this setup, 280 competitors submitted over 2600 proposed solutions within a 3 month period, showcasing substantial innovation especially in the modality alignment task. Here, we present the results, describe trends of well performing approaches, and discuss challenges associated with running the competition.

The performance of deep generative models for learning joint embeddings of single-cell multi-omics data

“…For the Multiome dataset, MultiVI's graph connectivity score is considerably lower for small sample sizes, while all models improve performance with an increasing number of cells. With the best performing models reaching a score of almost 1 for higher cell numbers in the case of the Multiome dataset, this is in line with the scores achieved by the models of the NeurIPS competition [30]. For the CITE-seq dataset, the performance of TotalVI increased with increasing cell numbers, achieving the highest graph connectivity score for 5,000 and 10,000 cells.…”

“…We compare our results with the metric values achieved by the models of the NeurIPS 2021 competition for the integration of the Multiome dataset (data points were extracted via WebPlotDigitizer-4.5 [44] from Supplementary Figure 6 of [30]). However, as we merely used a subset of at most 10,000 cells of the original benchmark dataset, we expect our investigated algorithms to score higher for most metrics if they were to be subjected to the complete benchmark dataset.…”

“…While this impact is minor for Cobolt, scMM, and scMVP, with Cobolt performing best throughout, MultiVI shows a surge with increasing cell numbers. The maximum score that models reach in our analysis slightly exceeds the median of the trajectory conservation scores of around 0.9 achieved by models of the NeurIPS 2021 competition [30].…”

“…Specifically, the NeurIPS benchmark dataset is a multi-donor (10 donors), multi-site (4 sites), multi-omics bone marrow dataset comprising two data types [30]:…”

“…Deep learning approaches, known for their ability to learn complex non-linear patterns from data, have become a popular building block for integrating different data types [15, 21]. For example, in 2021’s Conference on Neural Information Processing Systems (NeurIPS) competition (https://openproblems.bio/neurips_2021), which addressed the topic of multimodal single-cell data integration, neural networks proved to be the most popular, with shallow deep learning models being among the best-performing methods [30]. Specifically, deep generative models (DGMs) such as variational autoencoders (VAEs) are increasingly employed to infer joint embeddings, i.e., low-dimensional representations, from multi-omics datasets, allowing all further downstream analyses to be performed simultaneously within this joint latent space ( Figure 1 B ).…”