Analytics Methods to Understand Information Retrieval Effectiveness—A Survey

Abstract

Information retrieval aims to retrieve the documents that answer users’ queries. A typical search process consists of different phases for which a variety of components have been defined in the literature; each one having a set of hyper-parameters to tune. Different studies focused on how and how much the components and their hyper-parameters affect the system performance in terms of effectiveness, others on the query factor. The aim of these studies is to better understand information retrieval system effectiveness. This paper reviews the literature of this domain. It depicts how data analytics has been used in IR for a better understanding system effectiveness. This review concludes lack of full understanding of system effectiveness according to the context although it has been possible to adapted the query processing to some contexts successfully. This review also concludes that, even if it is possible to distinguish effective from non effective system on average on a query set, nor the system component analysis nor the query features analysis were successful in explaining when and why a particular system fails on a particular query.

Preface to the Special Issue “Natural Language Processing (NLP) and Machine Learning (ML)—Theory and Applications”

“…The paper authored by Mothe [1] concentrates on better understanding information retrieval system effectiveness when taking into account the system and the query, while the other existing dimensions (document collection, effectiveness measures) are left in the background. The paper reviews the literature of the field from this perspective and provides a clear negative answer to the basic but essential question: "Can we design a transparent model in terms of its performance on a query?"…”