Using Collection Shards to Study Retrieval Performance Effect Sizes

Ferro, Nicola; Kim, Dae Young; Sanderson, Mark

doi:10.1145/3310364

Cited by 21 publications

(10 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different work considered different statistical analysis and techniques, but always breaking down the run and topic effect. More recently, in [13] and [12], GLMM and ANOVA have been used to formalise the sub-corpora effect. GLMM and ANOVA were applied in [14] and [15] to the composition of the run components and their effect, using a Grid of Points (GoP) setting, borrowed from [11], to study all the possible run configurations.…”

Section: Related Workmentioning

confidence: 99%

Leveraging Behavioral Heterogeneity Across Markets for Cross-Market Training of Recommender Systems

Roitero

Carterrete²,

Mehrotra³

et al. 2020

Companion Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

Modern recommender systems are optimised to deliver personalised recommendations to millions of users spread across different geographic regions exhibiting various forms of heterogeneity, including behavioural-, content-and trend specific heterogeneity. System designers often face the challenge of deploying either a single global model across all markets, or developing custom models for different markets. In this work, we focus on the specific case of music recommendation across 21 different markets, and consider the trade-off between developing global model versus market specific models. We begin by investigating behavioural differences across users of different markets, and motivate the need for considering market as an important factor when training models. We propose five different training styles, covering the entire spectrum of models: from a single global model to individual market specific models, and in the process, propose ways to identify and leverage users abroad, and data from similar markets. Based on a large scale experimentation with data for 100M users across 21 different markets, we present insights which highlight that markets play a key role, and describe models that leverage market specific data in serving personalised recommendations.

show abstract

Section: Related Workmentioning

confidence: 99%

Leveraging Behavioral Heterogeneity Across Markets for Cross-Market Training of Recommender Systems

Roitero

Carterrete²,

Mehrotra³

et al. 2020

Companion Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

show abstract

“…In similar work, Voorhees et al [184] used sharding to produce the replicates necessary for estimating topic*system interaction effect sizes. Ferro et al [59] developed the first exhaustive model of topics, systems, shards, as well as all their interactions, and ran extensive experiments using several sharding schemes, including a selection of randomized and deterministic methods. They found that the topic factor has a large effect, the system factor is a small to medium-size effect, and the shard factor is a medium-to-large effect (roughly half of the topic factor).…”

Section: Modeling Ir Systems Performance: Linear Modelsmentioning

confidence: 99%

“…Model (MD5 ma ) is a combination of several models that have been used recently [198,59,62,61,184], and also extends the model by Zampieri et al [198] in order to cover all of the new interactions that are created when nesting query formulations. The models of Ferro et al [59], Ferro and Sanderson [62,61], Voorhees et al [184] were also extended so that the query formulations can be included in addition to all of the resulting cross-factor interactions. The design of experiments underlying this model is depicted in Figure 4.3b.…”

Section: Multiple Topic Formulations -Multiple Corpora Anova Modelsmentioning

confidence: 99%

Modelling and Explaining IR System Performance Towards Predictive Evaluation

Faggioli

2023

SIGIR Forum

View full text Add to dashboard Cite

Information Retrieval (IR) systems play a fundamental role in many modern commodities, including search engines (SEs), digital libraries, recommender systems, and social networks. The IR task is particularly challenging because of the volatility of IR systems performance: users' information needs change daily, and so do the documents to be retrieved and the concept of what is relevant to a given information need. Therefore, the empirical offline evaluation of an IR system is a costly and slow post-hoc procedure, that happens after the system deployment. Given the challenges linked to empirical IR evaluation, predicting a system's performance before its deployment would add significant value to the development of an IR system. In this manuscript, we place the cornerstone for the prediction of IR performance, by considering two closely related areas: the modeling of IR systems performance and the Query Performance Prediction (QPP). The former area allows us to identify those features that impact the most on the performance and that can be used as predictors, while the latter provides us with a starting point to instantiate the predictive task in IR. Concerning the modeling of IR performance, we first investigate one of the most popular statistical tools, ANOVA. In particular, we compare traditional ANOVA with a recent approach, bootstrap ANOVA, and observe the different conclusions that can be achieved using these two different statistical tools [Faggioli and Ferro, 2021]. Secondly, using ANOVA, we study the concept of topic difficulty and observe that the topic difficulty is not an intrinsic property of the information need but stems from the formulation used to represent the topic [Culpepper et al., 2022]. Finally, we show how to use Generalized Linear Models (GLMs) as an alternative to the traditional linear modeling of IR performance [Faggioli et al., 2022]. We show how GLMs provide more powerful inference with comparable stability. Our analyses on the QPP domain start with developing a predictor used to select among a set of reformulations for the same information need, the best-performing one for the systematic review task [Di Nunzio and Faggioli, 2021]. Secondly, we investigate how to classify queries as either semantic or lexical to predict whether neural models will perform better than lexical ones. Finally, given the challenges shown in the evaluation of the previous approaches, we devise a new evaluation procedure, dubbed sMARE [Faggioli et al., 2021]. sMARE allows moving from a single point estimation of the performance to a distributional one, allowing us to achieve improved comparisons between QPP models and more precise analyses. Awarded by: University of Padova, Padova, Italy on 20 March 2023. Supervised by: Nicola Ferro. Available at: https://www.research.unipd.it/handle/11577/3472979.

show abstract

“…In more detail, we considered the ANOVA effect size ω 2 , an unbiased index used to provide insights of the population-wide relationship between a set of factors and the studied outcomes [15,14,70,50,16]. With such setting, we fitted a linear model with which we measured the effect of the age, school, and all other possible answers to the questions in the questionnaire (B) w.r.t.…”

Section: Experimental Settingmentioning

confidence: 99%

Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19

Roitero,

Soprano,

Portelli

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of non-expert is exploited. We study whether crowdsourcing is an effective and reliable method to assess truthfulness during a pandemic, targeting statements related to COVID-19, thus addressing (mis)information that is both related to a sensitive and personal issue and very recent as compared to when the judgment is done. In our experiments, crowd workers are asked to assess the truthfulness of statements, and to provide evidence for the assessments. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we report results on workers behavior, agreement among workers, effect of aggregation functions, of scales transformations, and of workers background and bias. We perform a longitudinal study by re-launching the task multiple times with both novice and experienced workers, deriving important insights on how the behavior and quality change over time. Our results show that: workers are able to detect and objectively categorize online (mis)information related to COVID-19; both crowdsourced and expert judgments can be transformed and aggregated to improve quality; worker background and other signals (e.g., This is a preprint of an article accepted for publication in Personal and Ubiquitous Computing (Special Issue on Intelligent Systems for Tackling Online Harms).

show abstract

Using Collection Shards to Study Retrieval Performance Effect Sizes

Cited by 21 publications

References 54 publications

Leveraging Behavioral Heterogeneity Across Markets for Cross-Market Training of Recommender Systems

Leveraging Behavioral Heterogeneity Across Markets for Cross-Market Training of Recommender Systems

Modelling and Explaining IR System Performance Towards Predictive Evaluation

Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19

Contact Info

Product

Resources

About