Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data

Abstract

We present a systematic evaluation of state-of-the-art algorithms for inferring gene regulatory networks (GRNs) from single-cell transcriptional data. As the ground truth for assessing accuracy, we use synthetic networks with predictable trajectories, literature-curated Boolean models, and diverse transcriptional regulatory networks. We develop a strategy to simulate single-cell transcriptional data from synthetic and Boolean networks that avoids pitfalls of previously-used methods. Furthermore, we collect networks from multiple experimental single-cell RNA-seq datasets. We develop an evaluation framework called BEELINE. We find that the AUPRC and early precision of the algorithms are moderate. The methods are better in recovering interactions in synthetic networks than Boolean models. The algorithms with the best early precision values for Boolean models also perform well on experimental datasets. Techniques that do not require pseudotime-ordered cells are generally more accurate. Based on these results, we present recommendations to end users. BEELINE will aid the development of GRN inference algorithms.Single-cell RNA-sequencing technology has made it possible to trace cellular lineages during differentiation and to identify new cell types 1,2 . A central question that arises now is whether we can discover the gene regulatory networks (GRNs) that control cellular differentiation and drive transitions from one cell type to another. In such a GRN, each edge connects a transcription factor (TF) to a gene it regulates. Ideally, the edge is directed from the TF to the target gene, represents direct rather than indirect regulation, and corresponds to activation or inhibition.Single-cell expression data are especially promising for computing GRNs because, unlike bulk transcriptomic data, they do not obscure biological signals by averaging over all the cells in a sample. However, these data have features that pose significant difficulties, e.g., Users may view, print, copy, and download text and data-mine the content in such documents, for the purposes of academic research, subject always to the full Conditions of use:

GRNUlar: Gene Regulatory Network reconstruction using Unrolled algorithm from Single Cell RNA-Sequencing data

“…In this section, we evaluate the methods on seven datasets from five experiments which include human mature hepatocytes (hHEP) (Camp et al, 2017) , human embryonic stem cells (hESC) (Chu et al, 2016), mouse embryonic stem cells (mESC) (Hayashi et al, 2018), mouse dendritic cells (mDC) (Shalek et al, 2014), and three lineages of mouse hematopoietic stem cells (Nestorowa et al, 2016): erythroid lineage (mHSC-E), granulocyte-macrophage lineage (mHSC-GM) and lymphoid lineage (mHSC-L). These are the same datasets used in Pratapa et al (2020) and we use their corresponding ground-truth networks for our experiments as well. For each dataset there are three versions of groundtruth networks: cell-type-specific ChIP-seq, nonspecific ChIP-seq and functional interaction networks collected from STRING.…”

“…It has been a long-standing challenge to reconstruct these networks computationally from gene-expression data (Chen et al, 1998;Kim et al, 2003). Recently, single cell RNA-Sequencing (scRNA-Seq) technologies provide unprecedented scale of genome-wide gene-expression data from thousands of single cells, which can lead to the inference of more reliable and detailed regulatory networks (Chen and Mar, 2018;Pratapa et al, 2020). GRNBoost2 (Moerman et al, 2019) and GENIE3 (Vân Anh Huynh-Thu et al, 2010) are among the top performing methods for GRN inference (Chen and Mar, 2018;Pratapa et al, 2020).…”

“…Recently, single cell RNA-Sequencing (scRNA-Seq) technologies provide unprecedented scale of genome-wide gene-expression data from thousands of single cells, which can lead to the inference of more reliable and detailed regulatory networks (Chen and Mar, 2018;Pratapa et al, 2020). GRNBoost2 (Moerman et al, 2019) and GENIE3 (Vân Anh Huynh-Thu et al, 2010) are among the top performing methods for GRN inference (Chen and Mar, 2018;Pratapa et al, 2020). They are based on fitting a regression function between the expression values of the TFs and other genes.…”

“…While methods for GRN inference from scRNA-Seq data are improving, simulators which generate synthetic scRNA-Seq data guided by GRNs are also progressing (Dibaeinia and Sinha, 2019;Pratapa et al, 2020). These scRNA-Seq simulators can generate realistic looking data, and have modeled sources of variation in single cell RNA-Seq data, such as noise intrinsic to the process of transcription, extrinsic variation indicative of different cell states, technical variation and measurement noise and Shrivastava et al Fig.…”

“…bias. The primary application of these realistic simulators is to benchmark the performance of GRN inference methods (Dibaeinia and Sinha, 2019;Chen and Mar, 2018;Pratapa et al, 2020). These evaluations show that the performance of current method for GRN inference is not satisfying even with synthetic data.…”