Using statistical testing in the evaluation of retrieval experiments

Hull, David A.

doi:10.1145/160688.160758

Cited by 357 publications

(209 citation statements)

References 11 publications

Supporting

Mentioning

204

Contrasting

Unclassified

Order By: Relevance

“…19 We also indicate significance of the results using the standard Wilcoxon signed rank test (p < 0.05) [134]. For comparison with the MT experiments presented in previous section, the IR results are also tested against the M0 P0 baseline (of the respective language pair); those that are significantly better are typed in bold and those which are significantly worse are typed in italics.…”

Section: Information Retrieval Qualitymentioning

confidence: 99%

Adaptation of machine translation for multilingual information retrieval in the medical domain

Pecina¹,

Dušek²,

Goeuriot³

et al. 2014

Artificial Intelligence in Medicine

View full text Add to dashboard Cite

Objective. We investigate machine translation (MT) of user search queries in the context of cross-lingual information retrieval (IR) in the medical domain. The main focus is on techniques to adapt MT to increase translation quality; however, we also explore MT adaptation to improve effectiveness of cross-lingual IR.Methods and Data. Our MT system is Moses, a state-of-the-art phrase-based statistical machine translation system. The IR system is based on the BM25 retrieval model implemented in the Lucene search engine. The MT techniques employed in this work include in-domain training and tuning, intelligent training data selection, optimization of phrase table configuration, compound splitting, and exploiting synonyms as translation variants. The IR methods include morphological normalization and using multiple translation variants for query expansion. The experiments are performed and thoroughly evaluated on three language pairs: Czech-English, German-English, and French-English. MT quality is evaluated on data sets created within the Khresmoi project and IR effectiveness is tested on the CLEF eHealth 2013 data sets.Results. The search query translation results achieved in our experiments are outstanding -our systems outperform not only our strong baselines, but also Google Translate and Microsoft Bing Translator in direct comparison carried out on all the language pairs. The baseline BLEU scores increased from 26.59 to 41.45 for Czech-English, from 23.03 to 40.82 for German-English, and from 32.67 to 40.82 for French-English. This is a 55% improvement on average. In terms of the IR performance on this particular test collection, a significant improvement over the baseline is achieved only for French-English. For Czech-English and German-English, the increased MT quality does not lead to better IR results.Conclusions. Most of the MT techniques employed in our experiments improve MT of medical search queries. Especially the intelligent training data selection proves to be very successful for domain adaptation of MT. Certain improvements are also obtained from German compound splitting on the source language side. Translation quality, however, does not appear to correlate with the IR performance -better translation does not necessarily yield better retrieval. We discuss in detail the contribution of the individual techniques and state-of-the-art features and provide future research directions.

show abstract

Section: Information Retrieval Qualitymentioning

confidence: 99%

Adaptation of machine translation for multilingual information retrieval in the medical domain

Pecina¹,

Dušek²,

Goeuriot³

et al. 2014

Artificial Intelligence in Medicine

View full text Add to dashboard Cite

show abstract

“…The table shows that searching with knowledge (i.e., a generalized lattice) obtained better evaluation scores for each measure, except for precision. To see if these differences can be considered statistically significant (Hull, 1993) we performed a paired t-test for each measure.…”

Section: Experimental Evaluation Of Galois As a Browsing Retrieval Symentioning

confidence: 99%

A lattice conceptual clustering system and its application to browsing retrieval

1996

View full text Add to dashboard Cite

Abstract. The theory of concept (or Galois) lattices provides a simple and formal approach to conceptual clustering. In this paper we present GALOIS, a system that automates and applies this theory. The algorithm utilized by GALOIS to build a concept lattice is incremental and efficient, each update being done in time at most quadratic in the number of objects in the lattice. Also, the algorithm may incorporate background information into the lattice, and through clustering, extend the scope of the theory. The application we present is concerned with information retrieval via browsing, for which we argue that concept lattices may represent major support structures. We describe a prototype user interface for browsing through the concept lattice of a document-term relation, possibly enriched with a thesaurus of terms. An experimental evaluation of the system performed on a medium-sized bibliographic database shows good retrieval performance and a significant improvement after the introduction of background knowledge.

show abstract

“…This search engine extracts semantic information from image content features, such as colour, shape, texture, spatial location of objects in images, etc [4]- [8]. The extracted visual information is natural and objective, but completely ignores the role of human knowledge in the interpretation process.…”

Section: Content-based Approachesmentioning

confidence: 99%

“…With the huge amounts of indexed web pages, we expect top-ranked documents will be more representative, and relevance model estimation will be more accurate and reliable for each query, we send the same keywords to Google Web Search and obtain a list of relevant documents via Google Web APIs [11]. Before calculating the statistics from these top-ranked HTML documents, we remove all HTML tags, filter out words appearing in the INQUERY [4] stop word list, and stem words using Porter algorithm [19], which are all common pre processing in the Information Retrieval systems [2], and usually improve retrieval performance. The relevance model is estimated in the same way described before.…”

Section: Relevance Model Estimationmentioning

confidence: 99%