An Axiomatic Approach to Diagnosing Neural IR Models

Rennings, Daniel; Moraes, Felipe; Hauff, Claudia

doi:10.1007/978-3-030-15712-8_32

Cited by 29 publications

(41 citation statements)

References 41 publications

(64 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even though we can demonstrate promising first steps to axiomatically explain retrieval systems' result rankings, the addition of further well-grounded axiomatic constraints capturing other retrieval aspects seems to be needed to further improve the explanations. Its current limitations notwithstanding, we consider our approach a promising complement to the more tightly-controlled studies from previous work [7,32,44]. While the latter shed light on the general principles under which complex relevance scoring models operate, our axiomatic reconstruction framework could help IR system designers-or even end users-make sense of a concrete ranking for a real-world query.…”

Section: Discussionmentioning

confidence: 97%

Towards Axiomatic Explanations for Neural Ranking Models

Völske

Bondarenko

Fröbe

et al. 2021

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

Recently, neural networks have been successfully employed to improve upon state-of-the-art effectiveness in ad-hoc retrieval tasks via machine-learned ranking functions. While neural retrieval models grow in complexity and impact, little is understood about their correspondence with well-studied IR principles. Recent work on interpretability in machine learning has provided tools and techniques to understand neural models in general, yet there has been little progress towards explaining ranking models.We investigate whether one can explain the behavior of neural ranking models in terms of their congruence with well understood principles of document ranking by using established theories from axiomatic IR. Axiomatic analysis of information retrieval models has formalized a set of constraints on ranking decisions that reasonable retrieval models should fulfill. We operationalize this axiomatic thinking to reproduce rankings based on combinations of elementary constraints. This allows us to investigate to what extent the ranking decisions of neural rankers can be explained in terms of the existing retrieval axioms, and which axioms apply in which situations. Our experimental study considers a comprehensive set of axioms over several representative neural rankers. While the existing axioms can already explain the particularly confident ranking decisions rather well, future work should extend the axiom set to also cover the other still "unexplainable" neural IR rank decisions. CCS CONCEPTS• Information systems → Retrieval models and ranking.

show abstract

Section: Discussionmentioning

confidence: 97%

Towards Axiomatic Explanations for Neural Ranking Models

Völske

Bondarenko

Fröbe

et al. 2021

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

show abstract

“…Model agnostic approaches for rankings include [22] which constructed surrogate models on a pre-selected feature subset, [23,25] adapted Lime to the problem of explaining textual document relevance with respect to a query for a pointwise ranker, [19,26] used axioms as a diagnostic tool and [24] which addresses the pointwise limitation by locally explaining the list output of any pure text based ranker via a set of intent terms. Unlike [23][24][25] we focus on explaining LtR models with arbitrary features such as TF-IDF scores, page rank, etc as well as being applicable to pairwise and listwise models.…”

Section: Related Workmentioning

confidence: 99%

Extracting per Query Valid Explanations for Blackbox Learning-to-Rank Models

Singh

Khosla

Wang

et al. 2021

Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

Learning-to-rank (LTR) is a class of supervised learning techniques that apply to ranking problems dealing with a large number of features. The popularity and widespread application of LTR models in prioritizing information in a variety of domains makes their scrutability vital in today's landscape of fair and transparent learning systems. However, limited work exists that deals with interpreting the decisions of learning systems that output rankings. In this paper we propose a model agnostic local explanation method that seeks to identify a small subset of input features as explanation to the ranked output for a given query. We introduce new notions of validity and completeness of explanations specifically for rankings, based on the presence or absence of selected features, as a way of measuring goodness. We devise a novel optimization problem to maximize validity directly and propose greedy algorithms as solutions. In extensive quantitative experiments we show that our approach outperforms other model agnostic explanation approaches across pointwise, pairwise and listwise LTR models in validity while not compromising on completeness. CCS CONCEPTS• Information systems → Retrieval models and ranking;

show abstract

“…For completeness, we first state the original axiom and then outline how we extend and relax it in order to create a diagnostic dataset from it. For axioms TFC1, TFC2, LNC2 and M-TDC we follow the process described in [25]. We make use of the following notation: Q is a query and consists of terms q 1 , q 2 , ...; D i is a document of length |D i | containing terms d i1 , d i2 , ...; the count of term w in document D is c(w, D); lastly, S(Q, D) is the retrieval score the model assigns to D for a given Q.…”

Section: Diagnostic Datasetsmentioning

confidence: 99%

Diagnosing BERT with Retrieval Heuristics

Câmara

Hauff

2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Word embeddings, made widely popular in 2013 with the release of word2vec, have become a mainstay of NLP engineering pipelines. Recently, with the release of BERT, word embeddings have moved from the term-based embedding space to the contextual embedding space-each term is no longer represented by a single lowdimensional vector but instead each term and its context determine the vector weights. BERT's setup and architecture have been shown to be general enough to be applicable to many natural language tasks. Importantly for Information Retrieval (IR), in contrast to prior deep learning solutions to IR problems which required significant tuning of neural net architectures and training regimes, "vanilla BERT" has been shown to outperform existing retrieval algorithms by a wide margin, including on tasks and corpora that have long resisted retrieval effectiveness gains over traditional IR baselines (such as Robust04). In this paper, we employ the recently proposed axiomatic dataset analysis technique-that is, we create diagnostic datasets that each fulfil a retrieval heuristic (both term matching and semantic-based)-to explore what BERT is able to learn. In contrast to our expectations, we find BERT, when applied to a recently released large-scale web corpus with ad-hoc topics, to not adhere to any of the explored axioms. At the same time, BERT outperforms the traditional query likelihood retrieval model by 40%. This means that the axiomatic approach to IR (and its extension of diagnostic datasets created for retrieval heuristics) may in its current form not be applicable to large-scale corpora. Additional-different-axioms are needed.

show abstract

An Axiomatic Approach to Diagnosing Neural IR Models

Cited by 29 publications

References 41 publications

Towards Axiomatic Explanations for Neural Ranking Models

Towards Axiomatic Explanations for Neural Ranking Models

Extracting per Query Valid Explanations for Blackbox Learning-to-Rank Models

Diagnosing BERT with Retrieval Heuristics

Contact Info

Product

Resources

About