Differential gene expression analysis tools exhibit substandard performance for long non-coding RNA-sequencing data

Abstract

BackgroundLong non-coding RNAs (lncRNAs) are typically expressed at low levels and are inherently highly variable. This is a fundamental challenge for differential expression (DE) analysis. In this study, the performance of 25 pipelines for testing DE in RNA-seq data is comprehensively evaluated, with a particular focus on lncRNAs and low-abundance mRNAs. Fifteen performance metrics are used to evaluate DE tools and normalization methods using simulations and analyses of six diverse RNA-seq datasets.ResultsGene expression data are simulated using non-parametric procedures in such a way that realistic levels of expression and variability are preserved in the simulated data. Throughout the assessment, results for mRNA and lncRNA were tracked separately. All the pipelines exhibit inferior performance for lncRNAs compared to mRNAs across all simulated scenarios and benchmark RNA-seq datasets. The substandard performance of DE tools for lncRNAs applies also to low-abundance mRNAs. No single tool uniformly outperformed the others. Variability, number of samples, and fraction of DE genes markedly influenced DE tool performance.ConclusionsOverall, linear modeling with empirical Bayes moderation (limma) and a non-parametric approach (SAMSeq) showed good control of the false discovery rate and reasonable sensitivity. Of note, for achieving a sensitivity of at least 50%, more than 80 samples are required when studying expression levels in realistic settings such as in clinical cancer research. About half of the methods showed a substantial excess of false discoveries, making these methods unreliable for DE analysis and jeopardizing reproducible science. The detailed results of our study can be consulted through a user-friendly web application, giving guidance on selection of the optimal DE tool (http://statapps.ugent.be/tools/AppDGE/).Electronic supplementary materialThe online version of this article (10.1186/s13059-018-1466-5) contains supplementary material, which is available to authorized users.

Long noncoding RNA expression profiling in cancer: Challenges and opportunities

“…Therefore, accurate differential expression analysis of lncRNAs requires larger sample sizes compared to mRNA-focused analyses. 23 One attractive technology that can help increase lncRNA detection sensitivity and reproducibility is (lnc)RNA capture sequencing. 24 This technology applies biotinylated probes designed to capture (lnc)RNAs of interest.…”

dearseq: a variance component score test for RNA-Seq differential analysis that effectively controls the false discovery rate

“…As highlighted by both Conesa et al [17] and Assefa et al [15], engaging in realistic yet clear simulations is difficult. One has to find the right balance between the controlled settings necessary to know the ground truth, and the realism necessary to be convincing that the results would translate in real-world analyses.…”

“…When comparing DEA methods, the evaluation of their empirical FDR with respect to the targeted (nominal) level is often overlooked [5,6,7,8,9,10]. Nonetheless, some issues with inflated FDR in DEA have been previously reported in the literature [11,12,13,14,15], but those warnings have made little apparent impact on DEA practices.…”

SPsimSeq: semi-parametric simulation of bulk and single cell RNA sequencing data

“…There are also fully non-parametric approaches that employ subsampling from real data [3]. Although non-parametric simulators generate realistic synthetic data, they have limited flexibility and require a large source dataset to subsample from [1].…”