cdev: a ground-truth based measure to evaluate RNA-seq normalization performance

Abstract

Normalization of RNA-seq data has been an active area of research since the problem was first recognized a decade ago. Despite the active development of new normalizers, their performance measures have been given little attention. To evaluate normalizers, researchers have been relying on ad hoc measures, most of which are either qualitative, potentially biased, or easily confounded by parametric choices of downstream analysis. We propose a metric called condition-number based deviation, or cdev, to quantify normalization success. cdev measures how much an expression matrix differs from another. If a ground truth normalization is given, cdev can then be used to evaluate the performance of normalizers. To establish experimental ground truth, we compiled an extensive set of public RNA-seq assays with external spike-ins. This data collection, together with cdev, provides a valuable toolset for benchmarking new and existing normalization methods.

Genomic variant benchmark: if you cannot measure it, you cannot improve it

“…This dataset was employed to compare the performance of various tools in identifying differentially expressed genes [ 36 ]. Moreover, to evaluate the effectiveness of between-sample normalization methods, an integral step in RNA-seq data analysis, an experimental ground truth was established by compiling publicly available RNA-seq assays with external spike-ins [ 37 ]. These spike-ins, typically added to biological samples at known concentrations, provide a reliable reference for evaluation.…”

Temporal progress of gene expression analysis with RNA-Seq data: A review on the relationship between computational methods

“…The method: deviation based on the number of conditions, or cdev, to quantify the success of normalization, cdev measures how much one expression array differs from another. More information about this method can be found at [80] . Other options that can be considered are [81] , [82] .…”

Evaluation of clustering-based differential expression analysis methods for RNA-seq data

“…Many studies have been reported on e cient and accurate means of identifying DEGs [4][5][6]. This includes statistical methods [7][8][9], data normalization methods [10][11][12][13][14], R packages [15][16][17][18], and graphical user interfaces [19][20][21][22]. Conventional differential expression (DE) analysis such as edgeR [15] and DESeq2 [16] typically consists of two steps (data normalization X and DEG identi cation Y), and each R package has its own methods for the X-Y pipeline [23].…”

“…Recently, an increasing number of papers have been evaluated using extensive simulation scenarios in which the percentage of DEGs (P DEG ) in the data exceeds 50% (i.e., P DEG > 50%) [14,[29][30][33][34][35], and such real data do exist [36]. Although MBCdeg1 and 2 generally perform signi cantly worse at P DEG > 50% [29], the basic CPM normalization method may perform relatively well under these conditions [35].…”