A quantitative framework for evaluating single-cell data structure preservation by dimensionality reduction techniques

Heiser, Cody N.; Lau, Ken S.

doi:10.1101/684340

Cited by 3 publications

(4 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of good comparison and benchmark studies have already been performed on various steps related to scRNAseq processing and analysis and can guide the choice of methodology [3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21]. However, these recommendations need constant updating and often leave open many details of an analysis.…”

Section: Introductionmentioning

confidence: 99%

pipeComp, a general framework for the evaluation of computational pipelines, reveals performant single cell RNA-seq preprocessing tools

2020

View full text Add to dashboard Cite

We present pipeComp (https://github.com/plger/pipeComp), a flexible R framework for pipeline comparison handling interactions between analysis steps and relying on multi-level evaluation metrics. We apply it to the benchmark of single-cell RNA-sequencing analysis pipelines using simulated and real datasets with known cell identities, covering common methods of filtering, doublet detection, normalization, feature selection, denoising, dimensionality reduction, and clustering. pipeComp can easily integrate any other step, tool, or evaluation metric, allowing extensible benchmarks and easy applications to other fields, as we demonstrate through a study of the impact of removal of unwanted variation on differential expression analysis.

show abstract

Section: Introductionmentioning

confidence: 99%

pipeComp, a general framework for the evaluation of computational pipelines, reveals performant single cell RNA-seq preprocessing tools

2020

View full text Add to dashboard Cite

show abstract

“…The performances of the DR methods on the real and simulated CyTOF data are overall comparable, indicating the validity of our evaluation results as well as the simulation approach. Previous comparative works in the field of scRNAseq have supported the notions that tSNE and UMAP are the top performers in general and even linear methods are well-suited for certain workflows 20,37,38 . Few attempts at tackling this issue for CyTOF data have been made and the field seems to think in general that the best methodologies for scRNA-seq data can be directly applied on CyTOF data.…”

Section: Discussionmentioning

confidence: 91%

“…Here, we treat PCDs (of the original space data and DR embedding space data, respectively) as empirical one-dimensional distributions: in practice, we flatten the 𝑁 × 𝐶 matrices into vectors to represent observations from distributions that encapsulate the underlying global structure. Following the procedure of a previous usage of EMD as a global metric 20 , we perform min-max normalization of each vector to account for the difference in scale (original space PCDs can be on a different scale than embedding space PCDs). Subsequently, EMD is calculated after normalization.…”

Section: Accuracy -Global Structure Preservationmentioning

confidence: 99%

See 1 more Smart Citation

Comparative Analysis of Dimension Reduction Methods for Cytometry by Time-of-Flight Data

Wang

Yang

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently developed single-cell profiling technologies, such as single cell sequencing (scRNA-seq) and mass cytometry (CyTOF), provide critical biological insights from a large number of cells and a large number of features. While experimental and informatic techniques around scRNA-seq are much more advanced, research around CyTOF data analysis has severely lagged behind. However, CyTOF profiles the proteomics makeup of the single cells and are more closely related to disease phenotypes than scRNA-seq. CyTOF data are also dramatically different from scRNA-seq data in many aspects. This calls for the evaluation and development of statistical methods for CyTOF data. Dimension reduction (DR) is one of the most critical first steps of analyses of scRNA-seq and CyTOF data. Here, we benchmark 20 of these methods on 110 real and 425 synthetic CyTOF datasets, including 10 Imaging CyTOF datasets, for accuracy, scalability and usability. In particular, we checked the concordance of DR for CyTOF data against scRNA-seq data that were generated from the same samples. We found that SAUCIE and scvis surprisingly perform well across all categories, whereas UMAP particularly excels in downstream analyses and MDS preserves local and global structure well. Our results highlight the complementarity of existing tools, and that the choice of method should depend on the underlying data structure of the single cells. Based on these results, we develop a set of guidelines to help users select the best DR method for their dataset. Our freely available data and evaluation pipeline will aid in the development of improved DR algorithms specifically tailored to analyze data generated from the increasingly popular CyTOF technique.

show abstract

pipeComp, a general framework for the evaluation of computational pipelines, reveals performant single-cell RNA-seq preprocessing tools

Germain

Sonrel

Robinson

2020

Preprint

View full text Add to dashboard Cite

The massive growth of single-cell RNA-sequencing (scRNAseq) and methods for its analysis still 1 lacks sufficient and up-to-date benchmarks that would guide analytical choices. Moreover, current 2 studies are often focused on isolated steps of the process. Here, we present a flexible R framework 3 for pipeline comparison with multi-level evaluation metrics and apply it to the benchmark of 4 scRNAseq analysis pipelines using datasets with known cell identities. We evaluate common steps 5 of such analyses, including filtering, doublet detection, normalization, feature selection, denoising, 6 dimensionality reduction and clustering. On the basis of these analyses, we make a number of 7 concrete recommendations about analysis choices. The evaluation framework, pipeComp, has 8 been implemented so as to easily integrate any other step or tool, allowing extensible benchmarks 9 and easy application to other fields (https://github.com/plger/pipeComp). 10 Background 11 Single-cell RNA-sequencing (scRNAseq) and the set of attached analysis methods are evolving 12 fast, with more than 560 software tools available to the community [1] , roughly half of which are 13 dedicated to tasks related to data processing such as clustering, ordering, dimension reduction 14 or normalization. This increase in the number of available tools follows the development of new 15 sequencing technologies and the growing number of reported cells, genes and cell populations [2] . 16 As data processing is a critical step in any scRNAseq analysis, affecting downstream analysis and 17 interpretation, it is critical to evaluate the available tools. 18A number of good comparison and benchmark studies have already been performed on vari-19 ous steps related to scRNAseq processing and analysis and can guide the choice of methodology 20 ] ). However these recommendations need constant up-21 dating and often leave open many details of an analysis. Another missing aspect of current 22 benchmarking studies is their limitation to capture all aspects of scRNAseq processing workflow. 23 Although previous benchmarks already brought valuable recommendations for data processing, 24 some only focused on one aspect of data processing (e.g., [14] ), did not evaluate how the tool se-25 lection affects downstream analysis (e.g., [17] ) or did not tackle all aspects of data processing, such 26 as doublet identification or cell filtering (e.g., [18] ). A thorough evaluation of the tools covering all 27 major processing steps is however urgently needed as previous benchmarking studies highlighted 28 that a combination of tools can have a drastic impact on downstream analysis, such as differential 29 expression analysis and cell-type deconvolution [18,3] . It is then critical to evaluate not only the 30 single effect of a preprocessing method but also its positive or negative interaction with all parts 31 of a workflow. 32Here, we develop a flexible R framework for pipeline comparison and evaluate the various 33 steps of analysis leading from an initial count matrix to a clu...

show abstract

A quantitative framework for evaluating single-cell data structure preservation by dimensionality reduction techniques

Cited by 3 publications

References 40 publications

pipeComp, a general framework for the evaluation of computational pipelines, reveals performant single cell RNA-seq preprocessing tools

pipeComp, a general framework for the evaluation of computational pipelines, reveals performant single cell RNA-seq preprocessing tools

Comparative Analysis of Dimension Reduction Methods for Cytometry by Time-of-Flight Data

pipeComp, a general framework for the evaluation of computational pipelines, reveals performant single-cell RNA-seq preprocessing tools

Contact Info

Product

Resources

About