Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning

Abstract

Transfer learning has become the dominant paradigm for many natural language processing tasks. In addition to models being pretrained on large datasets, they can be further trained on intermediate (supervised) tasks that are similar to the target task. For small Natural Language Inference (NLI) datasets, language modelling is typically followed by pretraining on a large (labelled) NLI dataset before fine-tuning with each NLI subtask. In this work, we explore Gradient Boosted Decision Trees (GBDTs) as an alternative to the commonly used Multi-Layer Perceptron (MLP) classification head. GBDTs have desirable properties such as good performance on dense, numerical features and are effective where the ratio of the number of samples w.r.t the number of features is low. We then introduce FreeGBDT, a method of fitting a GBDT head on the features computed during finetuning to increase performance without additional computation by the neural network. We demonstrate the effectiveness of our method on several NLI datasets using a strong baseline model (RoBERTa-large with MNLI pretraining). The FreeGBDT shows a consistent improvement over the MLP classification head.

Drug-target interaction prediction using a multi-modal transformer network demonstrates high generalizability to unseen proteins

“…Previous studies have demonstrated benefits of utilizing learned representations from Transformer Networks as inputs for gradient boosting models, leading to improved outcomes compared to directly using the predictions of a Transformer Network 19,36 . We thus follow a similar approach here.…”

A multimodal Transformer Network for protein-small molecule interactions enhances predictions of kinase inhibition and enzyme-substrate relationships

“…Previous studies have demonstrated benefits of utilizing learned representations from Transformer Networks as inputs for gradient boosting models, leading to improved outcomes compared to directly using the predictions of a Transformer Network [18,37]. We thus follow a similar approach here.…”

“…The final ProSmith model does not use these predictions directly, but instead uses the learned joint protein-small molecule representations to train gradient boosting models. This strategy was motivated by previous studies that showed superior results when adding a gradient boosting step [18,37]. To investigate whether this additional step indeed contributed to the superior performance of ProSmith, we re-examined the enzyme-substrate prediction task, comparing the model performance of directly using the end-to-end trained multimodal Transformer Network with that of a gradient boosting model that takes the learned joint protein-small molecule representation from this Network as input.…”