GeneDisco: A Benchmark for Experimental Design in Drug Discovery

Abstract

In vitro cellular experimentation with genetic interventions, using for example CRISPR technologies, is an essential step in early-stage drug discovery and target validation that serves to assess initial hypotheses about causal associations between biological mechanisms and disease pathologies. With billions of potential hypotheses to test, the experimental design space for in vitro genetic experiments is extremely vast, and the available experimental capacity -even at the largest research institutions in the world -pales in relation to the size of this biological hypothesis space. Machine learning methods, such as active and reinforcement learning, could aid in optimally exploring the vast biological space by integrating prior knowledge from various information sources as well as extrapolating to yet unexplored areas of the experimental design space based on available data. However, there exist no standardised benchmarks and data sets for this challenging task and little research has been conducted in this area to date. Here, we introduce GeneDisco, a benchmark suite for evaluating active learning algorithms for experimental design in drug discovery. GeneDisco contains a curated set of multiple publicly available experimental data sets as well as open-source implementations of state-of-the-art active learning policies for experimental design and exploration.

A Framework and Benchmark for Deep Batch Active Learning for Regression

“…However, while our benchmark contains many large data sets, it cannot cover all possible application scenarios that the considered BMDAL methods could be applied to. For example, it is unclear whether our insights can be transferred to applications like drug discovery (Mehrjou et al, 2021) or atomistic ML (Zaverkin and Kästner, 2021), where other types of data and other NNs are employed. Even in the tabular data setting, the relevance of our results for smaller data sets or recently proposed NN architectures (e.g.…”

“…Zaverkin and Kästner (2021) work with atomistic data sets, which require specialized NN architectures and longer training times, and are therefore less well-suited for a large-scale benchmark. Recently, a benchmark for BMDAL for drug discovery has been proposed, which uses four counterfactual regression data sets (Mehrjou et al, 2021).…”

“…For classification, many BMDAL approaches have been proposed (Ren et al, 2021), and there exist at least some standard benchmark data sets like CIFAR-10 (Krizhevsky, 2009) on which methods are usually evaluated. On the other hand, the regression setting has been studied less frequently, and no common benchmark has been established to the best of our knowledge, except for a specialized benchmark in drug discovery (Mehrjou et al, 2021). We expect that the regression setting will gain popularity, not least due to the increasing interest in NNs for surrogate modeling (Behler, 2016;Kutz, 2017;Raissi et al, 2019;Mehrjou et al, 2021;Lavin et al, 2021).…”

PyRelationAL: A Library for Active Learning Research and Development

“…Note that φ is simply the identity function when computing distances in the ambient space. The coreset approach, as defined in [12], would be classified in this category in PyRelationAL, where the latent space representation of a point is defined by the associated penultimate activations in a neural network.…”

“…Recently, there has been a renewed interest in AL, especially in combination with advances in deep learning, to address various real-world applications [8,9]. Notably, applications in medicine have seen growing interest both from academic and industrial institutes [2,3,10,11,12].…”