Extending Probabilistic Data Fusion Using Sliding Windows

Lillis, David; Toolan, Fergus; Collier, Rem W.; Dunnion, John

doi:10.1007/978-3-540-78646-7_33

Cited by 15 publications

(14 citation statements)

References 20 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Precision at n documents (P@n) [3,[5][6][7]9,11,12,14,15,17,18]. • Average Precision (AP) or Mean Average Precision (MAP) [2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18]. • Precision/Recall curves [4,10,12].…”

Section: A Use Of Standard Ir Metricsmentioning

confidence: 99%

“…In addition to comparing a proposed fusion algorithm with the component systems' outputs, evaluation typically also includes a comparison with competing data fusion algorithms. This can be seen in [2,[4][5][6][7][8][9][10][11][12][13][14][15][15][16][17]20].…”

Section: Comparison With Other Fusion Algorithmsmentioning

confidence: 99%

“…Without this, a few outlier queries may cause the mean scores to increase even if the more common scenario is for performance to disimprove, or vice versa. For this purpose, a two-tailed paired t-test is most commonly employed [3,[5][6][7][8][11][12][13][14][15][16]. The Wilcoxon signed-rank test has also been used [9,17], although it has been stated that this test is not suitable for IR [22].…”

Section: Statistical Significance Testsmentioning

confidence: 99%

“…Some data fusion approaches rely on knowledge of the past performance of the component systems. Where this is the case, evaluations generally follow the lead of the machine learning community by dividing the set of available queries into a training set and a test set [2,[6][7][8]12,16,23]. The proportion that is used for training varies across experiments, with no particular approach appearing dominant.…”

Section: E Training Datamentioning

confidence: 99%

See 3 more Smart Citations

Towards an open science platform for the evaluation of data fusion

Huang

Chen

Meng

et al. 2018

2018 IEEE 3rd International Conference on Cloud Computing and Big Data Analysis (ICCCBDA)

Self Cite

View full text Add to dashboard Cite

Combining the results of different search engines in order to improve upon their performance has been the subject of many research papers. This has become known as the "Data Fusion" task, and has great promise in dealing with the vast quantity of unstructured textual data that is a feature of many Big Data scenarios. However, no universally-accepted evaluation methodology has emerged in the community. This makes it difficult to make meaningful comparisons between the various proposed techniques from reading the literature alone. Variations in the datasets, metrics, and baseline results have all contributed to this difficulty.This paper argues that a more unified approach is required, and that a centralised software platform should be developed to aid researchers in making comparisons between their algorithms and others. The desirable qualities of such a system have been identified and proposed, and an early prototype has been developed. Re-implementing algorithms published by other researchers is a great burden on those proposing new techniques. The prototype system has the potential to greatly reduce this burden and thus encourage more comparable results being generated and published more easily.

show abstract

Section: A Use Of Standard Ir Metricsmentioning

confidence: 99%

Section: Comparison With Other Fusion Algorithmsmentioning

confidence: 99%

Section: Statistical Significance Testsmentioning

confidence: 99%

Section: E Training Datamentioning

confidence: 99%

See 2 more Smart Citations

Towards an open science platform for the evaluation of data fusion

Huang

Chen

Meng

et al. 2018

2018 IEEE 3rd International Conference on Cloud Computing and Big Data Analysis (ICCCBDA)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Unsupervised fusion methods such as BordaFuse [3], Condorect Fusion [17] and CombMNZ [7] heuristically determine such a score based merely on the evidences provided by the search results like similarity, rank, and the number of the models retrieving the document. Supervised methods, for example, SegFuse [20], SlideFuse [14], PosFuse [13] and PLQA [23], get a better approximation of the score based on the given training data, which shows information about how well the models performed for some given queries before. With such training data, supervised methods can further estimate the quality of the evidences such as the accuracy of a rank position for a specific model [13] and the weight of a model [15], and then adjust the ranking score accordingly.…”

Section: Introductionmentioning

confidence: 99%

Query sampling for learning data fusion

Lin

Cheng

2011

Proceedings of the 20th ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Data fusion is to merge the results of multiple independent retrieval models into a single ranked list. Several earlier studies have shown that the combination of different models can improve the retrieval performance better than using any of the individual models. Although many promising results have been given by supervised fusion methods, training data sampling has attracted little attention in previous work of data fusion. By observing some evaluations on TREC and NTCIR datasets, we found that the performance of one model varied largely from one training example to another, so that not all training examples were equivalently effective. In this paper, we propose two novel approaches: greedy and boosting approaches, which select effective training data by query sampling to improve the performance of supervised data fusion algorithms such as BayesFuse, probFuse and MAPFuse. Extensive experiments were conducted on five data sets including TREC-3,4,5 and NTCIR-3,4. The results show that our sampling approaches can significantly improve the retrieval performance of those data fusion methods.

show abstract