Don’t Overweight Weights: Evaluation of Weighting Strategies for Multi-Task Bioactivity Classification Models

Humbeck, Lina; Morawietz, Tobias; Sturm, Noé; Zalewski, Adam; Harnqvist, Simon; Heyndrickx, Wouter; Holmes, Matthew R.; Beck, Bernd

doi:10.3390/molecules26226959

Cited by 7 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GlaxoSmithKline released raw clinical data on new drugs in 2012, which was welcomed as the first step toward information disclosure. The movement to follow this lead is spreading worldwide, and there has been a large range of open biomedical datasets available for training new machine learning algorithms developed by governments, medical societies, and international research collaborations. − The possibility of learning from a large amount of data containing similar structures is increasing. By using this method to build a prediction model using in-company data, we expect to develop a prediction model with a prediction accuracy higher than that of in vitro tests.…”

Section: Discussionmentioning

confidence: 99%

Prediction of the Contribution Ratio of a Target Metabolic Enzyme to Clearance from Chemical Structure Information

et al. 2022

View full text Add to dashboard Cite

The contribution ratio of metabolic enzymes such as cytochrome P450 to in vivo clearance (fraction metabolized: fm) is a pharmacokinetic index that is particularly important for the quantitative evaluation of drug–drug interactions. Since obtaining experimental in vivo fm values is challenging, those derived from in vitro experiments have often been used alternatively. This study aimed to explore the possibility of constructing machine learning models for predicting in vivo fm using chemical structure information alone. We collected in vivo fm values and chemical structures of 319 compounds from a public database with careful manual curation and constructed predictive models using several machine learning methods. The results showed that in vivo fm values can be obtained from structural information alone with a performance comparable to that based on in vitro experimental values and that the prediction accuracy for the compounds involved in CYP induction or inhibition is significantly higher than that by using in vitro values. Our new approach to predicting in vivo fm values in the early stages of drug discovery should help improve the efficiency of the drug optimization process.

show abstract

Section: Discussionmentioning

confidence: 99%

Prediction of the Contribution Ratio of a Target Metabolic Enzyme to Clearance from Chemical Structure Information

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Task weighting is a strategy that addresses this by assigning weights to the task loss values as they are summed, effectively suppressing antagonistic noise and allowing more synergistic training signals to be backpropagated. 20,21 Task grouping is another "informed" multitask learning strategy which involves clustering a given set of tasks into explicit subsets of synergistic tasks that are used to train separate multitask neural networks in a constrained manner free from antagonistic signals. 22−25 This work aims to investigate how toxicology domain knowledge can be used to handcraft task groupings and introduce "rule hints" that better guide the training of multitask neural networks.…”

Section: Bi)mentioning

confidence: 99%

“…These effects may be most noticeable when a given set of tasks is naïvely combined into one multitask neural network in which the loss values of antagonistic tasks are jointly backpropagated with other tasks. Task weighting is a strategy that addresses this by assigning weights to the task loss values as they are summed, effectively suppressing antagonistic noise and allowing more synergistic training signals to be backpropagated. , Task grouping is another “informed” multitask learning strategy which involves clustering a given set of tasks into explicit subsets of synergistic tasks that are used to train separate multitask neural networks in a constrained manner free from antagonistic signals. − …”

Section: Introductionmentioning

confidence: 99%

Mechanistic Task Groupings Enhance Multitask Deep Learning of Strain-Specific Ames Mutagenicity

Lui

Guan

Matthews

2023

Chem. Res. Toxicol.

View full text Add to dashboard Cite

The Ames test is a gold standard mutagenicity assay that utilizes various Salmonella typhimurium strains with and without S9 fraction to provide insights into the mechanisms by which a chemical can mutate DNA. Multitask deep learning is an ideal framework for developing QSAR models with multiple end points, such as the Ames test, as the joint training of multiple predictive tasks may synergistically improve the prediction accuracy of each task. This work investigated how toxicology domain knowledge can be used to handcraft task groupings that better guide the training of multitask neural networks compared to a naïve ungrouped multitask neural network developed on a complete set of tasks. Sixteen S. typhimurium ± S9 strain tasks were used to generate groupings based on mutagenic and metabolic mechanisms that were reflected in correlation data analyses. Both grouped and ungrouped multitask neural networks predicted the 16 strain tasks with a higher balanced accuracy compared with single task controls, with grouped multitask neural networks consistently featuring incremental increases in predictivity over the ungrouped approach. We conclude that the main variable driving these performance improvements is the general multitask effect with mechanistic task groupings acting as an enhancement step to further concentrate synergistic training signals united by a common biological mechanism. This approach enables incorporation of toxicology domain knowledge into multitask QSAR model development allowing for more transparent and accurate Ames mutagenicity prediction.

show abstract

“…To avoid any dominance of assays with a higher number of derived tasks during training, MELLODDY-TUNER produces a task weighting file for further machine learning which assigns 1/N weights to all tasks, with N being the number of derived tasks from that assay. 39…”

Section: Training Datasetmentioning

confidence: 99%

Conformal efficiency as a metric for comparative model assessment befitting federated learning

Heyndrickx

Arany

Simm

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

As training volume increases predictive model quality, leveraging existing external data sources holds the promise of time- and cost-efficiency. In a drug discovery setting, pharmaceutical companies all own substantial but confidential datasets. The MELLODDY project develops a privacy-preserving federated machine learning solution and deploys it at an unprecedented scale (more than 100,000 tasks across ten major pharmaceutical companies), while ensuring the security and privacy of each partner’s sensitive data. Each partner builds models that benefit from a shared representation, for their own private assays. Established predictive performance metrics such as AUC ROC or AUC PR are constrained to unseen labelled chemical space. However, they cannot gauge performance gains in unlabelled chemical space. Federated learning indirectly extends labelled space, but in a privacy-preserving context, a partner cannot use this label extension for performance assessment. Metrics that estimate uncertainty on a prediction can be calculated even where no label is known. Practically, the chemical space covered with predictions of sufficient confidence, reflects the applicability domain of a model. After establishing a link to established performance metrics, we propose the efficiency from the conformal prediction framework (‘conformal efficiency’) as a proxy to the applicability domain size. A documented extension of the applicability domain would qualify as a tangible benefit from federated learning. In interim assessments, MELLODDY partners report a median increase in conformal efficiency of the federated over the single-partner model of 5.5% (with increases up to 9.7%). Subject to distributional conditions, that efficiency increase can be directly interpreted as the expected increase in conformal i.e. high confidence predictions. In conclusion, we present the first evidence that privacy-preserving federated machine learning across massive drug-discovery datasets from ten pharma partners indeed extends the applicability domain of property prediction models.

show abstract

Don’t Overweight Weights: Evaluation of Weighting Strategies for Multi-Task Bioactivity Classification Models

Cited by 7 publications

References 18 publications

Prediction of the Contribution Ratio of a Target Metabolic Enzyme to Clearance from Chemical Structure Information

Prediction of the Contribution Ratio of a Target Metabolic Enzyme to Clearance from Chemical Structure Information

Mechanistic Task Groupings Enhance Multitask Deep Learning of Strain-Specific Ames Mutagenicity

Conformal efficiency as a metric for comparative model assessment befitting federated learning

Contact Info

Product

Resources

About