OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research

Hersh, William R.; Buckley, Chris; Leone, Tullia; Hickam, David H.

doi:10.1007/978-1-4471-2099-5_20

Cited by 446 publications

(348 citation statements)

References 7 publications

Supporting

Mentioning

329

Contrasting

Unclassified

Order By: Relevance

“…The first such evaluation campaign using medical data for evaluation of IR was OHSUMED [100]. The test collection contained around 350,000 abstracts from medical journals taken from the MEDLINE database over a period of five years and two sets of topics: a manually created one and another one based on the controlled vocabulary thesaurus of the Medical Subject Headings (MeSH) [68].…”

Section: Optimizing Query Translation For Cross-lingual Information Rmentioning

confidence: 99%

Adaptation of machine translation for multilingual information retrieval in the medical domain

Pecina¹,

Dušek²,

Goeuriot³

et al. 2014

Artificial Intelligence in Medicine

View full text Add to dashboard Cite

Objective. We investigate machine translation (MT) of user search queries in the context of cross-lingual information retrieval (IR) in the medical domain. The main focus is on techniques to adapt MT to increase translation quality; however, we also explore MT adaptation to improve effectiveness of cross-lingual IR.Methods and Data. Our MT system is Moses, a state-of-the-art phrase-based statistical machine translation system. The IR system is based on the BM25 retrieval model implemented in the Lucene search engine. The MT techniques employed in this work include in-domain training and tuning, intelligent training data selection, optimization of phrase table configuration, compound splitting, and exploiting synonyms as translation variants. The IR methods include morphological normalization and using multiple translation variants for query expansion. The experiments are performed and thoroughly evaluated on three language pairs: Czech-English, German-English, and French-English. MT quality is evaluated on data sets created within the Khresmoi project and IR effectiveness is tested on the CLEF eHealth 2013 data sets.Results. The search query translation results achieved in our experiments are outstanding -our systems outperform not only our strong baselines, but also Google Translate and Microsoft Bing Translator in direct comparison carried out on all the language pairs. The baseline BLEU scores increased from 26.59 to 41.45 for Czech-English, from 23.03 to 40.82 for German-English, and from 32.67 to 40.82 for French-English. This is a 55% improvement on average. In terms of the IR performance on this particular test collection, a significant improvement over the baseline is achieved only for French-English. For Czech-English and German-English, the increased MT quality does not lead to better IR results.Conclusions. Most of the MT techniques employed in our experiments improve MT of medical search queries. Especially the intelligent training data selection proves to be very successful for domain adaptation of MT. Certain improvements are also obtained from German compound splitting on the source language side. Translation quality, however, does not appear to correlate with the IR performance -better translation does not necessarily yield better retrieval. We discuss in detail the contribution of the individual techniques and state-of-the-art features and provide future research directions.

show abstract

Section: Optimizing Query Translation For Cross-lingual Information Rmentioning

confidence: 99%

Adaptation of machine translation for multilingual information retrieval in the medical domain

Pecina¹,

Dušek²,

Goeuriot³

et al. 2014

Artificial Intelligence in Medicine

View full text Add to dashboard Cite

show abstract

“…OHSUMED (Hersh, Buckley, Leone, & Hickam, 1994) contains relatively short abstracts of references from medical journals in the MEDLINE database. The average length of sentences is similar to AP88-90.…”

Section: Setupmentioning

confidence: 99%

Sentence-based relevance flow analysis for high accuracy retrieval

Lee

Seo

Jeon

et al. 2011

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

Traditional ranking models for information retrieval lack the ability to make a clear distinction between relevant and nonrelevant documents at top ranks if both have similar bag-of-words representations with regard to a user query. We aim to go beyond the bag-of-words approach to document ranking in a new perspective, by representing each document as a sequence of sentences. We begin with an assumption that relevant documents are distinguishable from nonrelevant ones by sequential patterns of relevance degrees of sentences to a query. We introduce the notion of relevance flow, which refers to a stream of sentence-query relevance within a document. We then present a framework to learn a function for ranking documents effectively based on various features extracted from their relevance flows and leverage the output to enhance existing retrieval models. We validate the effectiveness of our approach by performing a number of retrieval experiments on three standard test collections, each comprising a different type of document: news articles, medical references, and blog posts. Experimental results demonstrate that the proposed approach can improve the retrieval performance at the top ranks significantly as compared with the state-of-the-art retrieval models regardless of document type.

show abstract

“…Other collections have also frequently been used such as the OHSUMED collection (Hersh, Buckley et al 1994) used in (Joachims 1996;Baker and McCallum 1998;Lam and Ho 1998;Ruiz and Srinivasan 1999), the 20-newsgroups (Lang 1995) and in (Baker and McCallum 1998;Joachims 1998;McCallum and Nigam 1998;Nigam and Ghani 2000;Schapire and Singer 2000).…”

Section: Benchmarksmentioning

confidence: 99%

Text and Hypertext Categorization

Benbrahim

Bramer

2009

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Automatic categorization of text documents has become an important area of research in the last two decades, with features that make it significantly more difficult than the traditional classification tasks studied in machine learning. A more recent development is the need to classify hypertext documents, most notably web pages. These have features that add further complexity to the categorization task but also offer the possibility of using information that is not available in standard text classification, such as metadata and the content of the web pages that point to and are pointed at by a web page of interest. This chapter surveys the state of the art in text categorization and hypertext categorization, focussing particularly on issues of representation that differentiate them from 'conventional' classification tasks and from each other.

show abstract

OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research

Cited by 446 publications

References 7 publications

Adaptation of machine translation for multilingual information retrieval in the medical domain

Adaptation of machine translation for multilingual information retrieval in the medical domain

Sentence-based relevance flow analysis for high accuracy retrieval

Text and Hypertext Categorization

Contact Info

Product

Resources

About