Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery

Abstract

Knowledge Graphs (KG) and associated Knowledge Graph Embedding (KGE) models have recently begun to be explored in the context of drug discovery and have the potential to assist in key challenges such as target identification. In the drug discovery domain, KGs can be employed as part of a process which can result in lab-based experiments being performed, or impact on other decisions, incurring significant time and financial costs and most importantly, ultimately influencing patient healthcare. For KGE models to have impact in this domain, a better understanding of not only of performance, but also the various factors which determine it, is required. In this study we investigate, over the course of many thousands of experiments, the predictive performance of five KGE models on two public drug discoveryoriented KGs. Our goal is not to focus on the best overall model or configuration, instead we take a deeper look at how performance can be affected by changes in the training setup, choice of hyperparameters, model parameter initialisation seed and different splits of the datasets. Our results highlight that these factors have significant impact on performance and can even affect the ranking of models. Indeed these factors should be reported along with model architectures to ensure complete reproducibility and fair comparisons of future work, and we argue this is critical for the acceptance of use, and impact of KGEs in a biomedical setting. To aid reproducibility of our own work, we release all experimentation code.Preprint. Under review.

Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study

“…6 Statistics for all datasets and a sample of popular general domain benchmark KGs can be found in Table 1. While Hetionet has previously been explored for the task of KG completion as link prediction using KGE models (though not LMs) [Alshahrani et al, 2021, Bonner et al, 2021b, to our knowledge neither RepoDB nor MSI have been represented as KGs and used for evaluating KG completion models despite the potential benefits of this representation 6. More information on each dataset is available in Appendix A.1.…”

Implications of Topological Imbalance for Representation Learning on Biomedical Knowledge Graphs

“…Throughout this work we primarily make use of the publicly available Hetionet biomedical knowledge graph [19]. Hetionet was originally created as part of a project focusing on drug repurposing and integrates 29 public data sources, however its use has been explored in other areas such as target prediction [6]. Hetionet contains information on diseases, human protein-coding genes and compounds, among others, which are represented as entities within the graph and total over 47K.…”

“…Since its introduction, numerous other approaches have been proposed in the literature which build on or alter the TransE approach including TransH [49], ComplEx [46], RotatE [42] & DistMult [51], all with different strengths and weaknesses regarding relation types that can be captured [39]. However TransE has proven to still be highly competitive, when tuned appropriately, and can outperform more recent approaches [6].…”

“…Additionally, we kept the software environment consistent throughout all experimentation using python 3.8, CUDA 10.1, PyTorch 1.9, and PyKEEN 1.5.0. All models were trained using optimal hyperparameters and training setups [6].…”

“…Different Models. Figure 4 highlights that this relationship between degree and score is not just a TransE specific issue but is prevalent across many knowledge graph embedding models used in the literature (including TransH [49], ComplEx [46], RotatE [42] & DistMult [51] which were trained using optimised hyper-parameters for Hetionet as detailed previously [6]). The figure also displays the Hits@10 score on a hold-out testset comprising 10% of the triples for all models.…”