Towards More Realistic Simulated Datasets for Benchmarking Deep Learning Models in Regulatory Genomics

Abstract

Deep neural networks and support vector machines have been shown to accurately predict genomewide signals of regulatory activity from raw DNA sequences. These models are appealing in part because they can learn predictive DNA sequence features without prior assumptions. Several methods such as in-silico mutagenesis, GradCAM, DeepLIFT, Integrated Gradients and GkmExplain have been developed to reveal these learned features. However, the behavior of these methods on regulatory genomic data remains an area of active research. Although prior work has benchmarked these methods on simulated datasets with known ground-truth motifs, these simulations employed highly simplified regulatory logic that is not representative of the genome. In this work, we propose a novel pipeline for designing simulated data that comes closer to modeling the complexity of regulatory genomic DNA. We apply the pipeline to build simulated datasets based on publicly-available chromatin accessibility experiments and use these datasets to benchmark different interpretation methods based on their ability to identify ground-truth motifs. We find that a GradCAM-based method, which was reported to perform well on a more simplified dataset, does not do well on this dataset (particularly when using an architecture with shorter convolutional kernels in the first layer), and we theoretically show that this is expected based on the nature of regulatory genomic data. We also show that Integrated Gradients sometimes performs worse than gradient-times-input, likely owing to its linear interpolation path. We additionally explore the impact of user-defined settings on the interpretation methods, such as the choice of "reference"/"baseline", and identify recommended settings for genomics. Our analysis suggests several promising directions for future research on these model interpretation methods. Code and links to data are available at https://github.com/kundajelab/interpret-benchmark.

Unconstrained generation of synthetic antibody–antigen structures to guide machine learning methodology for antibody specificity prediction

“…Simulation allows the generation of synthetic complete-knowledge ground-truth datasets (i.e., datasets whose generation rules are known and therefore contain validated properties to be learned) containing desired levels of signal and noise that reflect experimental settings and biological mechanisms [58][59][60] . Simulated datasets have been used in methodological development and calibration before large-scale datasets become available, to disentangle machine learning hypotheses and to prioritize the design of future experiments 61,62 . For antibody-antigen binding prediction, simulations may help precisely and meaningfully define different real-world antibody-antigen binding problems, which requires levels of annotation that are not yet available in experimental data.…”

Hold out the genome: a roadmap to solving the cis-regulatory code

“…Further development of proteinstructure/affinity prediction models that can predict protein-protein 156 and protein-DNA complexes 157 that form on DNA in a sequence-dependent manner will also enable us to encode priors for TF-TF interactions and other cis-regulatory parameters, increasing the models' predictive power. Since determining what the models have learned is critical to many applications, developing and benchmarking model interpretation frameworks will be important 158,159 . Finally, having standardized datasets and modeling competitions (e.g.…”

Robustness and Usefulness in AI Explanation Methods

“…Moreover, decisionmakers often defer to algorithmic decision support systems [15] and struggle to use the algorithms effectively -often underperforming compared to both humans who are not assisted and the algorithms themselves [20]. Model-agnostic methods are not alone here, as Grad-CAM [31] has been shown to perform quite poorly on tasks it was specifically designed to excel at [27] under benchmark conditions that are not sufficiently realistic.…”