Query sampling for learning data fusion

Abstract

Data fusion is to merge the results of multiple independent retrieval models into a single ranked list. Several earlier studies have shown that the combination of different models can improve the retrieval performance better than using any of the individual models. Although many promising results have been given by supervised fusion methods, training data sampling has attracted little attention in previous work of data fusion. By observing some evaluations on TREC and NTCIR datasets, we found that the performance of one model varied largely from one training example to another, so that not all training examples were equivalently effective. In this paper, we propose two novel approaches: greedy and boosting approaches, which select effective training data by query sampling to improve the performance of supervised data fusion algorithms such as BayesFuse, probFuse and MAPFuse. Extensive experiments were conducted on five data sets including TREC-3,4,5 and NTCIR-3,4. The results show that our sampling approaches can significantly improve the retrieval performance of those data fusion methods.

Intelligent topic selection for low-cost information retrieval evaluation: A New perspective on deep vs. shallow judging

“…Further studies investigated topic selection for other purposes, such as creating low-cost datasets for training learning-to-rank algorithms (Mehrotra & Yilmaz, 2015), system rank estimation (Hauff, Hiemstra, De Jong, & Azzopardi, 2009), and selecting training data to improve supervised data fusion algorithms (Lin & Cheng, 2011). These studies do not consider topic selection for low-cost evaluation of IR systems.…”