Sensitivity, specificity, and reproducibility of RNA-Seq differential expression calls

Abstract

BackgroundThe MAQC/SEQC consortium has recently compiled a key benchmark that can serve for testing the latest developments in analysis tools for microarray and RNA-seq expression profiling. Such objective benchmarks are required for basic and applied research, and can be critical for clinical and regulatory outcomes. Going beyond the first comparisons presented in the original SEQC study, we here present extended benchmarks including effect strengths typical of common experiments.ResultsWith artefacts removed by factor analysis and additional filters, for genome scale surveys, the reproducibility of differential expression calls typically exceed 80% for all tool combinations examined. This directly reflects the robustness of results and reproducibility across different studies. Similar improvements are observed for the top ranked candidates with the strongest relative expression change, although here some tools clearly perform better than others, with typical reproducibility ranging from 60 to 93%.ConclusionsIn our benchmark of alternative tools for RNA-seq data analysis we demonstrated the benefits that can be gained by analysing results in the context of other experiments employing a reference standard sample. This allowed the computational identification and removal of hidden confounders, for instance, by factor analysis. In itself, this already substantially improved the empirical False Discovery Rate (eFDR) without changing the overall landscape of sensitivity. Further filtering of false positives, however, is required to obtain acceptable eFDR levels. Appropriate filters noticeably improved agreement of differentially expressed genes both across sites and between alternative differential expression analysis pipelines.ReviewersAn extended abstract of this research paper was selected for the Camda Satellite Meeting to Ismb 2015 by the Camda Programme Committee. The full research paper then underwent one round of Open Peer Review under a responsible Camda Programme Committee member, Lan Hu, PhD (Bio-Rad Laboratories, Digital Biology Center-Cambridge). Open Peer Review was provided by Charlotte Soneson, PhD (University of Zürich) and Michał Okoniewski, PhD (ETH Zürich). The Reviewer Comments section shows the full reviews and author responses.Electronic supplementary materialThe online version of this article (doi:10.1186/s13062-016-0169-7) contains supplementary material, which is available to authorized users.

Transcriptional Profiling of Non-injured Nociceptors After Spinal Cord Injury Reveals Diverse Molecular Changes

“…Library preparation was performed using the Illumina sequencing kit for high output 75-cycles for 25-30 M total single-end reads per sample. DESeq2 analyses 1 of differential expression were performed, and outliers beyond 30%-50% of the mean for each group of animals were eliminated (Love et al, 2014;Conesa et al, 2016;Labaj and Kreil, 2016;Wu and Wu, 2016).…”

Talking the talk, but not walking the walk: RT-qPCR as a paradigm for the lack of reproducibility in molecular research

“…A multiplatform examination of RNA‐seq data by the Sequencing Quality Control (SEQC) project found that relative, but not absolute, gene expression measurements can be measured accurately and reliably across laboratories and RNA‐seq platforms and that RNA‐seq and microarray‐based models were comparable in clinical endpoint prediction . Current focus is on data analysis in general, where under controlled conditions reproducibility ranges from 60% to 93% . Normalisation is a particular aspect where the application of different methodologies can generate discordant results .…”

BERMUDA: A novel deep transfer learning method for single-cell RNA sequencing batch correction reveals hidden high-resolution cellular subtypes

“…Single-cell RNA sequencing (scRNA-seq) was developed to characterize high-throughput gene expression profiles for populations of individual cells, which has enabled an unprecedented resolution of cellular heterogeneity in complex tissues. Widespread adoption of scRNA-seq techniques have produced large complex datasets, which present new computational challenges for evaluating experimental reproducibility and combining data from different batches and platforms [5,[7][8][9][10][11].…”

“…Single-cell RNA sequencing (scRNA-seq) was developed to characterize high-throughput gene expression profiles for populations of individual cells, which has enabled an unprecedented resolution of cellular heterogeneity in complex tissues. Widespread adoption of scRNA-seq techniques have produced large complex datasets, which present new computational challenges for evaluating experimental reproducibility and combining data from different batches and platforms [5,[7][8][9][10][11].There have been many attempts to combine gene expression data from different experiments to achieve a more comprehensive understanding of the underlying cellular heterogeneity. The first generation of tools were adapted from linear model analysis of microarrays [12][13][14] and were subsequently modified for RNA-seq data via generalized linear [15] or negative binomial models [16].…”