Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning

Minixhofer, Benjamin; Gritta, Milan; Iacobacci, Ignacio

doi:10.48550/arxiv.2105.03791

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous studies have demonstrated benefits of utilizing learned representations from Transformer Networks as inputs for gradient boosting models, leading to improved outcomes compared to directly using the predictions of a Transformer Network 19,36 . We thus follow a similar approach here.…”

Section: Prosmith Feeds the Learned Representations To Gradient Boost...mentioning

confidence: 99%

Drug-target interaction prediction using a multi-modal transformer network demonstrates high generalizability to unseen proteins

Kroll,

Ranjan,

Lercher

2023

Preprint

View full text Add to dashboard Cite

The activities of most enzymes and drugs rely on interactions between proteins and small molecules. Accurate predictions of these interactions could accelerate pharmaceutical and biotechnological research massively. Machine learning models designed for this task are currently limited by the lack of information exchange between the protein and the small molecule during the creation of the required numerical representations. Here, we introduce ProSmith, a machine learning framework that employs a multimodal Transformer Network to simultaneously process protein amino acid sequences and small molecule strings in the same input. This approach facilitates the exchange of all relevant information between the two molecule types during the calculation of their numerical representations, allowing the model to account for their structural and functional interactions. Our final model combines gradient boosting predictions based on the resulting multimodal Transformer Network with independent predictions based on separate deep learning representations of the proteins and small molecules. The corresponding predictions outperform all previous models for predicting protein-small molecule interactions across three diverse tasks: predicting Michaelis constantsKM; inferring potential substrates for enzymes; and predicting protein-drug affinities. The Python code provided can be used to easily implement and improve machine learning predictions involving arbitrary protein-small molecule interactions.

show abstract

Section: Prosmith Feeds the Learned Representations To Gradient Boost...mentioning

confidence: 99%