TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection

Abstract

We propose TandA, an effective technique for fine-tuning pre-trained Transformer models for natural language tasks. Specifically, we first transfer a pre-trained model into a model for a general task by fine-tuning it with a large and high-quality dataset. We then perform a second fine-tuning step to adapt the transferred model to the target domain. We demonstrate the benefits of our approach for answer sentence selection, which is a well-known inference task in Question Answering. We built a large scale dataset to enable the transfer step, exploiting the Natural Questions dataset. Our approach establishes the state of the art on two well-known benchmarks, WikiQA and TREC-QA, achieving the impressive MAP scores of 92% and 94.3%, respectively, which largely outperform the the highest scores of 83.4% and 87.5% of previous work. We empirically show that TandA generates more stable and robust models reducing the effort required for selecting optimal hyper-parameters. Additionally, we show that the transfer step of TandA makes the adaptation step more robust to noise. This enables a more effective use of noisy datasets for fine-tuning. Finally, we also confirm the positive impact of TandA in an industrial setting, using domain specific datasets subject to different types of noise.

Reranking for Efficient Transformer-based Answer Selection

“…Yang et al [23] applied it to Ad Hoc Document Retrieval, obtaining significant improvement. Garg et al [8] fine-tuned BERT for AS2, achieving the state of the art. However, BERT's high computational cost prevents its use in most real-word applications.…”

“…AS2, given a question and a set of answer sentence candidates, consists in selecting sentences (e.g., retrieved by a search engine) that correctly answer the question. Neural models have significantly contributed with new techniques, e.g., [8,11] to AS2. More recently, neural language models, e.g., ELMO [13], GPT [14], BERT [5], RoBERTa [10], XLNet [3] have led to major advancements in NLP.…”

“…To carry out our experiments, we created three different AS2 datasets, all including a large number of answer sentence candidates. Two of them are built using different samples of the anonymized questions from Alexa Information Traffic, while the third, ASNQ [8], is a sample from the Google Natural Question dataset [6], adapted for the AS2 task, which we further extended. We tested the combinations of fast rerankers, Jaccard similarity, Rel-CNN 1 and CA with an accurate BERT selector, and compared them with the upper bound, obtained with the expensive approach of classifying all candidates with the BERT model.…”

A Survey of Quantum Theory Inspired Approaches to Information Retrieval

“…However, more recent neural models, such as BERT based models [Garg et al 2019;Raffel et al 2019] have achieved a largely improved performance on the same tasks. It is worth exploring the integration of quantum models with the new BERT architecture [Devlin et al 2019] in the future.…”

“…Note that the current state-of-the-art BERT-based neural model for TREC-QA has achieved MAP and MRR of 0.943 and 0.974 respectively[Garg et al 2019].4 The TANDA model mentioned above currently gives the best performance on WikiQA dataset (MAP and MRR of 0.92 and 0.933 respectively, as compared to 0.695 and 0.71 by[Zhang et al 2018d]). …”