Collaborative Profile-QSAR: A Natural Platform for Building Collaborative Models among Competing Companies

Martin, Éric; Zhu, Xiangwei

doi:10.1021/acs.jcim.0c01342

Cited by 19 publications

(17 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As discussed, the predictive confidence from different unlabelled datasets is fairly consistent and exceeds that of the labelled datasets. The observation that both external and internal chemistry gain confidence, suggests, perhaps contrary to the expectation, 68 that it is possible to acquire knowledge about one's own compounds on one's own assays through the other partners' data in the federated multipartner learning. Despite this general picture, differences between the unlabelled datasets exist.…”

Section: Discussionmentioning

confidence: 78%

Conformal efficiency as a metric for comparative model assessment befitting federated learning

Heyndrickx

Arany

Simm

et al. 2022

Preprint

View full text Add to dashboard Cite

As training volume increases predictive model quality, leveraging existing external data sources holds the promise of time- and cost-efficiency. In a drug discovery setting, pharmaceutical companies all own substantial but confidential datasets. The MELLODDY project develops a privacy-preserving federated machine learning solution and deploys it at an unprecedented scale (more than 100,000 tasks across ten major pharmaceutical companies), while ensuring the security and privacy of each partner’s sensitive data. Each partner builds models that benefit from a shared representation, for their own private assays. Established predictive performance metrics such as AUC ROC or AUC PR are constrained to unseen labelled chemical space. However, they cannot gauge performance gains in unlabelled chemical space. Federated learning indirectly extends labelled space, but in a privacy-preserving context, a partner cannot use this label extension for performance assessment. Metrics that estimate uncertainty on a prediction can be calculated even where no label is known. Practically, the chemical space covered with predictions of sufficient confidence, reflects the applicability domain of a model. After establishing a link to established performance metrics, we propose the efficiency from the conformal prediction framework (‘conformal efficiency’) as a proxy to the applicability domain size. A documented extension of the applicability domain would qualify as a tangible benefit from federated learning. In interim assessments, MELLODDY partners report a median increase in conformal efficiency of the federated over the single-partner model of 5.5% (with increases up to 9.7%). Subject to distributional conditions, that efficiency increase can be directly interpreted as the expected increase in conformal i.e. high confidence predictions. In conclusion, we present the first evidence that privacy-preserving federated machine learning across massive drug-discovery datasets from ten pharma partners indeed extends the applicability domain of property prediction models.

show abstract

Section: Discussionmentioning

confidence: 78%

Conformal efficiency as a metric for comparative model assessment befitting federated learning

Heyndrickx

Arany

Simm

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It is common for profiles generated in industry to have compound overlap [18], so limiting pQSAR to actual compounds from industrial compound libraries is not a set back. Additional steps should be taken when building profiles using disjoint assays and targets, whether that be collecting more assays for the profile, searching for more targets, or building better single-task models.…”

Section: Discussionmentioning

confidence: 99%

“…Each new target only requires training a new PLS model, which is significantly more computationally efficient than training a neural network (NN) model. This approach also lends itself to combining data from different sources, such as pharmaceutical companies, without having to explicitly share the proprietary structures [18].…”

Section: Collaborative Filteringmentioning

confidence: 99%

Model Choice Metrics to Optimize Profile-QSAR Performance

Kim

McLoughlin

et al. 2022

Preprint

View full text Add to dashboard Cite

Predicting molecular activity against protein targets is difficult because of the paucity of experimental data. Approaches like multitask modeling and collaborative filtering seek to improve model accuracy by leveraging results from multiple targets, but are limited because different compounds are measured with different assays, leading to sparse data matrices. Profile-QSAR (pQSAR) 2.0 addresses this problem by fitting a series of partial least squares models for each target, using as features the predictions from single-task models on the remaining targets. This method has been shown to produce better results than single task and multitask models. However, the factors determining the success of pQSAR 2.0 have as yet not been characterized. In this paper we examine the experimental conditions that lead to better pQSAR models. We limit the amount of data available to the method by retraining with decreasing amounts of data and explore the model's ability to generalize to compounds that have never been assayed. Finally, we look at the properties of training data needed to demonstrate pQSAR improvement. We apply pQSAR 2.0 on a collection of GPCR and safety targets collected from Drug Target Commons, ExcapeDB, and ChEMBL. We found that pQSAR improved models on 34 of the 149 assays selected. In the other 115 assays, single task random forests offered better performance. There are many factors that contribute to an increase in performance, but the main factor is compound assay coverage. The pQSAR model improves when more compounds are measured in multiple assays. It is necessary to consider the available data before applying pQSAR. Successful pQSAR models require a profile made of correlated targets that share compounds with other assays. This technique is best used when experimental data is available as random forest regressors often do not generalize well enough for virtual drug search applications.

show abstract

“…To overcome this, new avenues are pursued. One particularly promising approach is collaborative efforts between otherwise competing companies, e.g., Martin and Zhu [1], leveraging artificial intelligence (AI) methods [2,3]. Here, we describe a part of the MELLODDY project, a collaborative effort of different pharma companies (referred to as "partner" throughout this article) in the field of multi-task learning [4].…”

Section: Introductionmentioning

confidence: 99%

“…In general, multi-task models have been shown to be beneficial in drug discovery [5][6][7][8]. Furthermore, increasing the amount of (diverse) high quality data is supposed to increase Molecules 2021, 26, 6959 2 of 15 model performance and applicability domains [1,5,9]. Nevertheless, detailed investigations to leverage the full potential of the new federated multi-task learning approach are needed.…”

Section: Introductionmentioning

confidence: 99%

Don’t Overweight Weights: Evaluation of Weighting Strategies for Multi-Task Bioactivity Classification Models

et al. 2021

View full text Add to dashboard Cite

Machine learning models predicting the bioactivity of chemical compounds belong nowadays to the standard tools of cheminformaticians and computational medicinal chemists. Multi-task and federated learning are promising machine learning approaches that allow privacy-preserving usage of large amounts of data from diverse sources, which is crucial for achieving good generalization and high-performance results. Using large, real world data sets from six pharmaceutical companies, here we investigate different strategies for averaging weighted task loss functions to train multi-task bioactivity classification models. The weighting strategies shall be suitable for federated learning and ensure that learning efforts are well distributed even if data are diverse. Comparing several approaches using weights that depend on the number of sub-tasks per assay, task size, and class balance, respectively, we find that a simple sub-task weighting approach leads to robust model performance for all investigated data sets and is especially suited for federated learning.

show abstract

Collaborative Profile-QSAR: A Natural Platform for Building Collaborative Models among Competing Companies

Cited by 19 publications

References 16 publications

Conformal efficiency as a metric for comparative model assessment befitting federated learning

Conformal efficiency as a metric for comparative model assessment befitting federated learning

Model Choice Metrics to Optimize Profile-QSAR Performance

Don’t Overweight Weights: Evaluation of Weighting Strategies for Multi-Task Bioactivity Classification Models

Contact Info

Product

Resources

About