Learning to Re-Rank with Contextualized Stopwords

Abstract

The use of stopwords has been thoroughly studied in traditional Information Retrieval systems, but remains unexplored in the context of neural models. Neural re-ranking models take the full text of both the query and document into account. Naturally, removing tokens that do not carry relevance information provides us with an opportunity to improve the effectiveness by reducing noise and lower document representation caching-storage requirements. In this work we propose a novel contextualized stopword detection mechanism for neural re-ranking models. This mechanism consists of training a sparse vector in order to filter out document tokens from the ranking decision. This vector is learned end-to-end based on the contextualized document representations, allowing the model to filter terms on a per occurrence basis. This leads to a more explainable model, as it reduces noise. We integrate our component into the state-of-the-art interaction-based TK neural re-ranking model. Our experiments on the MS MARCO passage collection and queries from the TREC 2019 Deep Learning Track show that filtering out traditional stopwords prior to the neural model reduces its effectiveness, while learning to filter out contextualized representations improves it.

Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

“…To further reduce the number of passage tokens to store, we adopt a simplified version of Hofstätter et al [17]'s contextualized stopwords (CS), which was first introduced for the TK-Sparse model. CS learns a removal gate of tokens solely based on their contextdependent vector representations.…”

“…The original implementation [17] masks scores after TK's kernelactivation, meaning the non-zero gates have to be saved as well, which increases the systems' complexity. In contrast, we directly apply the gate to the representation vectors.…”

“…Applying the stopword gate directly to the representation vector allows us to observe much more stable training than the authors of TK-Sparse observed -we do not need to adapt the training procedure with special mechanisms to keep the model from collapsing. Following Hofstätter et al [17] we train the removal gate with a regularization loss, which forces the stopword removal gate to become active during training as described in §4.1.…”

“…In order to further reduce the number of stored vectors we learn to reduce the BOW 2 representations with simplified contextualized stopwords (CS) [17]. To maximally reduce the dimensionality of the token vectors to one, we also employ an Exact Matching (EM) component, which matches only the vector representations of whole-words which are exact matches from the query.…”

On Approximate Nearest Neighbour Selection for Multi-Stage Dense Retrieval

“…However, allowing each query embedding the same chance to contribute to the candidate set may be sub-optimal. Indeed, consider a query embedding representing a stopword appearing in the query -retrieving many nearest neighbours to that query embedding is unlikely to retrieve as many relevant documents as a more discriminative query embedding [4,21] 1 .…”