SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval

Abstract

We introduce SPARTA, a novel neural retrieval method that shows great promise in performance, generalization, and interpretability for open-domain question answering. Unlike many neural ranking methods that use dense vector nearest neighbor search, SPARTA learns a sparse representation that can be efficiently implemented as an Inverted Index. The resulting representation enables scalable neural retrieval that does not require expensive approximate vector search and leads to better performance than its dense counterpart. We validated our approaches on 4 opendomain question answering (OpenQA) tasks and 11 retrieval question answering (ReQA) tasks. SPARTA achieves new state-of-the-art results across a variety of open-domain question answering tasks in both English and Chinese datasets, including open SQuAD, CMRC and etc. Analysis also confirms that the proposed method creates human interpretable representation and allows flexible control over the trade-off between performance and efficiency.

Continual Learning of Long Topic Sequences in Neural Information Retrieval

“…Moreover, recent works addressed ranking with sequence-to-sequence transformers based approach as the Mono-T5 model [31] for re-ranking documents returned by a BM25 ranker. Using a weak initial ranker such as BM25 may be the bottleneck of reaching higher performances, some approaches are thus reconsidering dense retrieval [14,15,7,44]. All these models are data-dependent, relying on word/topic/query distribution in the training dataset and their application to new domains is not always straightforward [28,32].…”

Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering

“…Along with the success of deep learning that offers remarkable semantic representation, various deep retrieval models have been developed in the past few years, greatly enhancing retrieval effectiveness and thus lifting final QA performance. According to the different ways of encoding the question and document as well as of scoring their similarity, dense retrievers in existing OpenQA systems can be roughly divided into three types: Representation-based Retriever [16], [29], [36], [72], Interaction-based Retriever [15], [31], and Representation-interaction Retriever [17], [81], as illustrated in Fig. 5.…”

“…Representation-interaction Retriever: In order to achieve both high accuracy and efficiency, some recent systems [17], [81] combine representation-based and interaction-based methods. For instance, ColBERT-QA [17] develops its retriever based on ColBERT [82], which extends the dual-encoder architecture by performing a simple token-level interaction step over the question and document representations to calculate the similarity score.…”

“…Then, ColBERT computes the score of each token embedding of the question over all those of the document first, and then sums all these scores as the final relevance score between q and d. As another example, SPARTA [81] develops a neural ranker to calculate the token-level matching score using dot product between a non-contextualized encoded (e.g., BERT word embedding) question and a contextualized encoded (e.g., BERT encoder) document. Concretely, given the representations of the question and document, the weight of each question token is computed with max-pooling, ReLU and log sequentially; the final relevance score is the sum of each question token weight.…”

SEQ2SEQ++: A Multitasking-Based Seq2seq Model to Generate Meaningful and Relevant Answers

“…There are two popular approaches in conversational chatbot modeling namely Transformer network-based models such as [13]- [15] and recurrent neural network (RNN)-based sequence to sequence learning (Seq2Seq) models such as [16]- [22]. The Transformer network is based on the feed-forward network [11], wherein sentences are processed as a whole rather than word by word by utilizing a self-attention mechanism, which can be highly parallelized.…”