The Cascade Transformer: an Application for Efficient Answer Sentence Selection

Abstract

Large transformer-based language models have been shown to be very effective in many classification tasks. However, their computational complexity prevents their use in applications requiring the classification of a large set of candidates. While previous works have investigated approaches to reduce model size, relatively little attention has been paid to techniques to improve batch throughput during inference. In this paper, we introduce the Cascade Transformer, a simple yet effective technique to adapt transformer-based models into a cascade of rankers. Each ranker is used to prune a subset of candidates in a batch, thus dramatically increasing throughput at inference time. Partial encodings from the transformer model are shared among rerankers, providing further speed-up. When compared to a state-of-the-art transformer model, our approach reduces computation by 37% with almost no impact on accuracy, as measured on two English Question Answering datasets.

Pretrained Transformers for Text Ranking: BERT and Beyond

“…One key limitation of BERT is its inability to handle long input sequences and hence difficulty in ranking texts beyond a certain length (e.g., "full-length" documents such as news articles). This limitation is addressed by a number of models (Nogueira and Cho, 2019;Akkalyoncu Yilmaz et al, 2019;Dai and Callan, 2019b;MacAvaney et al, 2019;, and a simple retrieve-then-rerank approach can be elaborated into a multi-stage architecture with reranker pipelines (Nogueira et al, 2019a;Matsubara et al, 2020;Soldaini and Moschitti, 2020) that balance effectiveness and efficiency. On top of multi-stage ranking architectures, researchers have proposed additional innovations, including query expansion , document expansion (Nogueira et al, 2019b;Nogueira and Lin, 2019) and term importance prediction Callan, 2019a, 2020).…”

A Study on Efficiency, Accuracy and Document Structure for Answer Sentence Selection

“…Finally, our word-relatedness encoder can replace the standard attention to enhance the speed of the fast attention-based approaches, resulting in a fast and accurate network in the class of fast methods. Our research will gain more and more important also in the light of improving the efficiency of large architecture using our models for sequential re-ranking (Matsubara et al, 2020;Soldaini and Moschitti, 2020).…”

Early Exiting BERT for Efficient Document Ranking

“…We can see that in both datasets, early exiting is able to accelerate inference by ∼2.5× while maintaining the original model effectiveness. It is worth noting that in Cascade Transformer (CT) (Soldaini and Moschitti, 2020), only a part of the development set is used for evaluation, and therefore the scores are not directly comparable. However, in terms of relative performance, our model appears to achieve a bit higher inference speedup with a comparable score degradation.…”

“…Our work differs from them by using an early exiting strategy that specializes for document ranking. Another related work that focuses on retrieval is Cascade Transformer (Soldaini and Moschitti, 2020), where a fixed proportion of samples are dropped after each layer. In contrast, our work drops samples based on their scores, and empirically we are able to achieve higher inference speedups.…”

“…We conduct experiments on BERT BASE with two document ranking datasets, MS MARCO passage (Bajaj et al, 2016) and ASNQ (Garg et al, 2019). We compare against Cascade Transformer (Soldaini and Moschitti, 2020), a recently proposed technique to accelerate inference in BERT-based document ranking. Results show that our method can reduce inference latency by up to 2.5× with minimal effectiveness degradation.…”