Benchmarking Algorithms for Gene Set Scoring of Single-cell ATAC-seq Data

Abstract

Gene set scoring (GSS) has been routinely conducted for gene expression analysis of bulk or single-cell RNA-seq data, which helps to decipher single-cell heterogeneity and cell-type-specific variability by incorporating prior knowledge from functional gene sets. Single-cell assay for transposase accessible chromatin using sequencing (scATAC-seq) is a powerful technique for interrogating single-cell chromatin-based gene regulation, and genes or gene sets with dynamic regulatory potentials can be regarded as cell-type specific markers as if in scRNA-seq. However, there are few GSS tools specifically designed for scATAC-seq, and the applicability and performance of RNA-seq GSS tools on scATAC-seq data remain to be investigated. We systematically benchmarked ten GSS tools, including four bulk RNA-seq tools, five single-cell RNA-seq (scRNA-seq) tools, and one scATAC-seq method. First, using matched scATAC-seq and scRNA-seq datasets, we find that the performance of GSS tools on scATAC-seq data is comparable to that on scRNA-seq, suggesting their applicability to scATAC-seq. Then the performance of different GSS tools were extensively evaluated using up to ten scATAC-seq datasets. Moreover, we evaluated the impact of gene activity conversion, dropout imputation, and gene set collections on the results of GSS. Results show that dropout imputation can significantly promote the performance of almost all GSS tools, while the impact of gene activity conversion methods or gene set collections on GSS performance is more GSS tool or dataset dependent. Finally, we provided practical guidelines for choosing appropriate pre-processing methods and GSS tools in different scenarios.

scATAC-seq preprocessing and imputation evaluation system for visualization, clustering and digital footprinting

“…Several comparison studies conducted direct [ 14 ] or indirect [ 8 , 13 ] benchmarking, or cross-referencing scATAC-seq results with relevant scRNA-seq [ 15–17 ], driving conclusions on imputation step quality for scATAC-seq.…”

“…Still, the most recent papers [ 15 , 16 ] employ imputation frameworks as a preprocessing step for general scATAC-seq pipeline evaluation (gene scoring [ 15 ], single-cell integration [ 16 ]) with no focus on the impact of the different imputation strategies. Independent benchmark studies [ 14 , 17 ] and method presenting papers [ 8 , 13 ] have provided effective benchmarking protocols for comparing scATAC-seq imputation methods; however, these works do not explicitly investigate the contribution of preprocessing and postprocessing steps, which can have a significant impact on the imputation framework results.…”