Harmony Assumptions in Information Retrieval and Social Networks

Every information retrieval (IR) model embeds in its scoring function a form of term frequency (TF) quantification. The contribution of the term frequency is determined by the properties of the function of the chosen TF quantification, and by its TF normalization. The first defines how independent the occurrences of multiple terms are, while the second acts on mitigating the a priori probability of having a high term frequency in a document (estimation usually based on the document length). New test collections, coming from different domains (e.g. medical, legal), give evidence that not only document length, but in addition, verboseness of documents should be explicitly considered. Therefore we propose and investigate a systematic combination of document verboseness and length. To theoretically justify the combination, we show the duality between document verboseness and length. In addition, we investigate the duality between verboseness and other components of IR models. We test these new TF normalizations on four suitable test collections. We do this on a well defined spectrum of TF quantifications. Finally, based on the theoretical and experimental observations, we show how the two components of this new normalization, document verboseness and length, interact with each other. Our experiments demonstrate that the new models never underperform existing models, while sometimes introducing statistically significantly better results, at no additional computational cost.

show abstract

“…The following definition formalizes the well-defined spectrum of quantifications (Roelleke et al. 2015 ).…”

Section: Probabilistic Derivation Of Ir Modelsmentioning

confidence: 99%

“…2 quantifications when This spectrum is well-defined because each of these s correspond to an assumption regarding term events (Roelleke et al. 2015 ). corresponds to assuming independence, and the and variants assume the occurrences of an event to be dependent.…”

Section: Probabilistic Derivation Of Ir Modelsmentioning

confidence: 99%

A systematic approach to normalization in probabilistic models

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…Many applications apply the heuristics by topping IR models off with novel parameters to deliver burstiness into the retrieval process. However, these heuristics are not generalizable, and their theoretical explanations are rarely published [16]. Hence in this paper, we use multinomial language modelling to leverage entity burstiness.…”

Section: Introductionmentioning

confidence: 99%

Semantic-aware Retrieval and Recommendation based on the Dirichlet Compound Language Model

Bahrani

Roelleke

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Traditional bag-of-word Information Retrieval (IR) has led to quasi-standard models (e.g. TF-IDF, BM25) that are applied in the context of recommendation. While the benefits of semantic-aware models are clear, there is no standard semantic IR model, and this prevents the usage of semantic retrieval for tasks such as recommendation. We propose a ranking framework by quantifying parameters of the Dirichlet Compound Model (DCM) and using query performance prediction to aggregate scores of semantic extensions of DCM. Instead of employing heuristics, we introduce reasonable aggregation standards which are expandable to the recommendation task by considering user interaction (e.g. rating) as a semantic event. This approach helps to bridge the gap between advanced IR and recommendation. One of the main findings of this paper is that the concept-based extension of DCM consistently outweighed both terms-only and concept-only baselines concerning biomedical data. The DCM-based recommender system is also shown to be effective when applied on MovieLens, Book Crossing and LastFM. By bringing together recommendation (prediction) and retrieval, we lay the grounds for generalizable and hybrid algorithms. This paper shows how advanced but well-established probability distributions can be transparently leveraged in the standardization of semantic ranking methods, especially for applications in which recommendation (notification) is critical.

show abstract

FDCM

Bahrani

Roelleke

2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Harmony Assumptions in Information Retrieval and Social Networks

Cited by 3 publications

References 63 publications

A systematic approach to normalization in probabilistic models

A systematic approach to normalization in probabilistic models

Semantic-aware Retrieval and Recommendation based on the Dirichlet Compound Language Model

FDCM

Contact Info

Product

Resources

About