Computational enhancer prediction: evaluation and improvements

Abstract

Background Identifying transcriptional enhancers and other cis -regulatory modules (CRMs) is an important goal of post-sequencing genome annotation. Computational approaches provide a useful complement to empirical methods for CRM discovery, but it is critical that we develop effective means to evaluate their performance in terms of estimating their sensitivity and specificity. Results We introduce here pCRMeval , a pipeline for in silico evaluation of any enhancer prediction tools that are flexible enough to be applied to the Drosophila melanogaster genome. pCRMeval compares the result of predictions with the extensive existing knowledge of experimentally-validated Drosophila CRMs in order to estimate the precision and relative sensitivity of the prediction method. In the case of supervised prediction methods—when training data composed of validated CRMs are used— pCRMeval can also assess the sensitivity of specific training sets. We demonstrate the utility of pCRMeval through evaluation of our SCRMshaw CRM prediction method and training data. By measuring the impact of different parameters on SCRMshaw performance, as assessed by pCRMeval, we develop a more robust version of SCRMshaw, SCRMshaw_HD, that improves the number of predictions while maintaining sensitivity and specificity. Our analysis also demonstrates that SCRMshaw_HD, when applied to increasingly less well-assembled genomes, maintains its strong predictive power with only a minor drop-off in performance. Conclusion Our pCRMeval pipeline provides a general framework for evaluation that can be applied to any CRM prediction method, particularly a supervised method. While we make use of it here primarily to test and improve a particular method for CRM prediction, SCRMshaw, pCRMeval should provide a valuable platform to the research community not only for evaluating individual methods, but also for comparing between competing methods. Electronic supplementary material The online version of this article (10.1186/s12859-019-2781-x) contains supplementary material, which is available to authorized users.

Identification of new Anopheles gambiae transcriptional enhancers using a cross‐species prediction approach

“…(Not all of the 11 lines could be evaluated for each criterion, either due to possible vector‐dependent expression or because expression of orthologs is not known; the range for predicted expression matching represents the lower/upper bounds depending on whether observed expression is enhancer‐driven or vector‐driven.) Although these numbers are moderately weaker than what we have observed in previous applications of SCRMshaw (Kantorovitz et al ., 2009; Kazemian et al ., 2011, 2014), they are well above random expectation (Asma and Halfon, 2019). SCRMshaw performance is highly dependent on the training data used, and a relatively narrow selection of training sets were used here.…”

Annotating the Insect Regulatory Genome

“…STARR-seq, which elegantly converts CRMs into their own reporters by cloning them downstream of a core promoter and sequencing the output, is one increasingly popular method [ 14 , 16 , 17 , 18 , 19 , 20 ]. Although reporter-based methods have long been viewed as a gold standard, due to the fact that they provide a direct functional readout of regulatory activity, there is growing recognition that these methods can lead to both false-positive and false-negative results [ 7 , 8 , 21 ]. However, the overall accuracy of reporter gene assays is believed to be high, and these remain the most definitive assays for regulatory function.…”

“…REDfly has also played a dramatic role in developing methods for computational CRM discovery. Its extensive collection of experimentally verified CRMs provides a ready source of validation data for assessing CRM predictions and for comparing among methods [ 21 , 112 , 113 , 114 , 115 , 116 ]. Perhaps more importantly, REDfly’s advanced search and filtering features make it an unmatched source for compiling training data for machine-learning approaches [ 10 , 11 , 61 , 117 ].…”

How to study enhancers in non-traditional insect models

“…Although methods demonstrated to work using vertebrate genomes are expected to function equally well in insects, comparing efficacies is difficult given the different training and validation regimens applied. An evaluation platform for assessing methods using a uniform set of Drosophila training and validation data has recently been described (Asma and Halfon, 2019), and a critical comparison of various approaches would be a valuable addition to the field.…”

“…How effective is enhancer discovery when genome assemblies are highly incomplete? Testing SCRMshaw with simulated dis-assembly of the Drosophila genome has revealed that contig N50s of at least 23,000 bp (which encompasses the upper 50% of current insect assemblies) are sufficient for effective SCRMshaw prediction, with minor loss of sensitivity and negligible increase in false-positive rates (Asma and Halfon, 2019). Therefore, highly complete genome assembly does not appear to be a prerequisite for successful enhancer prediction by SCRMshaw.…”