Supervised application of internal validation measures to benchmark dimensionality reduction methods in scRNA-seq data

Abstract

A typical single-cell RNA sequencing (scRNA-seq) experiment will measure on the order of 20 000 transcripts and thousands, if not millions, of cells. The high dimensionality of such data presents serious complications for traditional data analysis methods and, as such, methods to reduce dimensionality play an integral role in many analysis pipelines. However, few studies have benchmarked the performance of these methods on scRNA-seq data, with existing comparisons assessing performance via downstream analysis accuracy measures, which may confound the interpretation of their results. Here, we present the most comprehensive benchmark of dimensionality reduction methods in scRNA-seq data to date, utilizing over 300 000 compute hours to assess the performance of over 25 000 low-dimension embeddings across 33 dimensionality reduction methods and 55 scRNA-seq datasets. We employ a simple, yet novel, approach, which does not rely on the results of downstream analyses. Internal validation measures (IVMs), traditionally used as an unsupervised method to assess clustering performance, are repurposed to measure how well-formed biological clusters are after dimensionality reduction. Performance was further evaluated over nearly 200 000 000 iterations of DBSCAN, a density-based clustering algorithm, showing that hyperparameter optimization using IVMs as the objective function leads to near-optimal clustering. Methods were also assessed on the extent to which they preserve the global structure of the data, and on their computational memory and time requirements across a large range of sample sizes. Our comprehensive benchmarking analysis provides a valuable resource for researchers and aims to guide best practice for dimensionality reduction in scRNA-seq analyses, and we highlight Latent Dirichlet Allocation and Potential of Heat-diffusion for Affinity-based Transition Embedding as high-performing algorithms.

Disentangling single-cell omics representation with a power spectral density-based feature extraction

“…We assessed the performance of scPSD transformation, independent of any downstream analyses, in improving the cell-type clustering tendency and reducing the complexity of single-cell omics data. We previously proposed ( 14 ) a supervised application of internal validation measures (IVMs) such as silhouette score (SS) ( 39 ) and variance ratio criterion (VRC) ( 40 ), to quantify the compactness and separation of annotated cell-type clusters. We also defined a measure of the complexity of a multi-class dataset inspired by the Fisher's discriminant ratio (FDR) ( 28 ) (detailed in online Methods) to quantify the pairwise difference and dispersion of individual features among different cell types.…”

“…SCALE (single-cell ATAC-seq analysis via latent feature extraction) ( 13 ). DR methods have varied performance in separating biological clusters as per our recent comprehensive benchmarking ( 14 ) and often perform poorly in facilitating the detection of rare cell populations ( 15 ). Furthermore, the capacity of different DR methods in extracting features from other single-cell omics, beyond scRNA-sequencing data, is undetermined and yet to be assessed systematically.…”

An introduction to representation learning for single-cell data analysis

“…While not all representation learning methods require the same pre-processing steps, it is necessary to carefully consider these in order to suppress spurious or irrelevant variation and make data more amenable to effective representation learning and subsequent analyses. 45 Because pre-processing impacts the performance of any subsequent representation learning tools, iterative refinement of pre-processing on the basis of downstream outcomes may be important as an investigation matures.…”

Recent developments in application of single-cell RNA sequencing in the tumour immune microenvironment and cancer therapy

“…Different approaches have been proposed, such as principal component analysis (PCA) [ 54 ], nonnegative matrix factorization (NMF) [ 55 ], and deep neural networks [ 56 ]. The detailed features of these methods have been described in other reviews [ 57 , 58 ]. Zero inflation is another challenge of scRNA-seq data analysis.…”