How does the structure of data impact cell–cell similarity? Evaluating how structural properties influence the performance of proximity metrics in single cell RNA-seq data

Abstract

Accurately identifying cell-populations is paramount to the quality of downstream analyses and overall interpretations of single-cell RNA-seq (scRNA-seq) datasets but remains a challenge. The quality of single-cell clustering depends on the proximity metric used to generate cell-to-cell distances. Accordingly, proximity metrics have been benchmarked for scRNA-seq clustering, typically with results averaged across datasets to identify a highest performing metric. However, the ‘best-performing’ metric varies between studies, with the performance differing significantly between datasets. This suggests that the unique structural properties of an scRNA-seq dataset, specific to the biological system under study, have a substantial impact on proximity metric performance. Previous benchmarking studies have omitted to factor the structural properties into their evaluations. To address this gap, we developed a framework for the in-depth evaluation of the performance of 17 proximity metrics with respect to core structural properties of scRNA-seq data, including sparsity, dimensionality, cell-population distribution and rarity. We find that clustering performance can be improved substantially by the selection of an appropriate proximity metric and neighbourhood size for the structural properties of a dataset, in addition to performing suitable pre-processing and dimensionality reduction. Furthermore, popular metrics such as Euclidean and Manhattan distance performed poorly in comparison to several lessor applied metrics, suggesting that the default metric for many scRNA-seq methods should be re-evaluated. Our findings highlight the critical nature of tailoring scRNA-seq analyses pipelines to the dataset under study and provide practical guidance for researchers looking to optimize cell-similarity search for the structural properties of their own data.

The Specious Art of Single-Cell Genomics

“…To assess heterogeneity within clusters or relationships between clusters, similarity metrics or distances can be calculated between the cells [33] and displayed with qualitative or quantitative visuals which preserve these metrics, including hierarchical relationship diagrams such as dendrograms and trees [78, 79], or graph-based network diagrams [80, 81]. Higher-level diagrams that do not seek to display all point-wise information can also be used to represent the results of other inter-cluster analyses [68, 82].…”

“…However, this procedure is taken as a baseline [6,28], and there is little discussion of the logic behind this coupling. We do present analyses alternatively measuring distortions with the L 1 metric, given its more desirable properties in higher dimensions than Euclidean (L 2 ) distance (see above), but other choices of distance metrics are possible and, whether in ambient or reduced space, can provide different implications and interpretations of the dataset's properties [33]. In light of this, one might surmise that the non-linear methods instead learn other manifold-specific 'metrics' from cell neighborhoods by identifying 'biological geometries' (though this is not justified by the original authors [4,5]).…”

“…We utilize the L 2 norm as it is the default metric of UMAP/t-SNE. We also present results with the L 1 norm (shown in Supplementary Materials), as L 1 is more suitable for measuring distance in high dimensions, particularly in comparison to other L norms [29,30], and is commonly applied to transcriptomic data [31][32][33], with comparable performance to the probabilistic Jensen-Shannon divergence in single-cell distance calculations [34].…”

The impact of similarity metrics on cell type clustering in highly multiplexed in situ imaging cytometry data

“…A recent study [18] demonstrated that for single-cell RNA-seq datasets, correlation-based metrics outperformed distance-based metrics. This point was further illustrated by [57], where many distance metrics were benchmarked for clustering scRNA-seq data. The authors showed the significance of the impact of distance metrics on clustering and that correlation-based distances tend to perform better on average.…”

Effect of distance measures on confidences of t-SNE embeddings and its implications on clustering for scRNA-seq data

“…Since the calculation of distance measures constituted the most computationally expensive part of the algorithm, we aimed to limit the number of used distance measures. Many studies have demonstrated the functionality of Cosine similarity and Kullback-Leibler divergence in clustering, as they can effectively measure the (dis)similarity of clusters, especially when dealing with high-dimensional data, such as natural language processing applications and sc-RNA seq analysis 21 – 23 . Therefore, despite the availability of other distance measures that demonstrate comparable performance, such as Jaccard and Motyka, these two measures have been specifically included in the study.…”