A novel probabilistic retrieval model is presented. It forms a basis to interpret the TF-IDF term weights as making relevance decisions. It simulates the local relevance decision-making for every location of a document, and combines all of these "local" relevance decisions as the "documentwide" relevance decision for the document. The significance of interpreting TF-IDF in this way is the potential to: (1) establish a unifying perspective about information retrieval as relevance decision-making; and (2) develop advanced TF-IDF-related term weights for future elaborate retrieval models. Our novel retrieval model is simplified to a basic ranking formula that directly corresponds to the TF-IDF term weights. In general, we show that the term-frequency factor of the ranking formula can be rendered into different term-frequency factors of existing retrieval systems. In the basic ranking formula, the remaining quantity − log p(r|t ∈ d ) is interpreted as the probability of randomly picking a nonrelevant usage (denoted byr) of term t. Mathematically, we show that this quantity can be approximated by the inverse document-frequency (IDF). Empirically, we show that this quantity is related to IDF, using four reference TREC ad hoc retrieval data collections.

In an ad-hoc retrieval task, the query is usually short and the user expects to find the relevant documents in the first several result pages. We explored the possibilities of using Wikipedia's articles as an external corpus to expand ad-hoc queries. Results show promising improvements over measures that emphasize on weak queries.

No abstract

No abstract

We studied the effectiveness of a new class of contextdependent term weights for information retrieval. Unlike the traditional term frequency-inverse document frequency (TF-IDF ), the new weighting of a term t in a document d depends not only on the occurrence statistics of t alone but also on the terms found within a text window (or "document-context") centered on t. We introduce a Boost and Discount (B&D) procedure which utilizes partial relevance information to compute the contextdependent term weights of query terms according to a logistic regression model. We investigate the effectiveness of the new term weights compared with the contextindependent BM25 weights in the setting of relevance feedback. We performed experiments with title queries of the TREC-6, -7, -8, and 2005 collections, comparing the residual Mean Average Precision (MAP) measures obtained using B&D term weights and those obtained by a baseline using BM25 weights. Given either 10 or 20 relevance judgments of the top retrieved documents, using the new term weights yields improvement over the baseline for all collections tested. The MAP obtained with the new weights has relative improvement over the baseline by 3.3 to 15.2%, with statistical significance at the 95% confidence level across all four collections.

XML holds the promise to yield (1) a more precise search by providing additional information in the elements, (2) a better integrated search of documents from heterogeneous sources, (3) a powerful search paradigm using structural as well as content specifications, and (4) data and information exchange to share resources and to support cooperative search. We survey several indexing techniques for XML documents, grouping them into flatfile, semistructured, and structured indexing paradigms. Searching techniques and supporting techniques for searching are reviewed, including full text search and multistage search. Because searching XML documents can be very flexible, various search result presentations are discussed, as well as database and information retrieval system integration and XML query languages. We also survey various retrieval models, examining how they would be used or extended for retrieving XML documents. To conclude the article, we discuss various open issues that XML poses with respect to information retrieval and database research. IntroductionAn Internet search engine (e.g., Altavista or Infoseek) returns thousands of so-called matched documents from a single query, some of which are relevant and others irrelevant to the query. End users typically have problems with organizing and digesting such vast quantities of information, in which much (i.e., 75% as pointed out by Selberg and Etzioni, 1997) of the information retrieved is likely to be irrelevant. XML holds the promise that searching can be done more precisely because structural, self-describing information and meta-data (e.g., RDF) is available, to allow for context-based and/or category-based search. XML also holds the promise to model heterogeneous data, generated from databases (DBs) or from word processors, thereby enabling search engines to locate and process heterogeneous documents or records.An XML document consists of a set of elements, which are hierarchically structured, as defined by the user. Each element has a name (e.g., p for paragraph), which is defined by the user. Data of an element (say, p) can be stored inside the element delimited by its start tag (i.e., ͗p͘) and its end tag (i.e., ͗/p͘), or it can be stored as values in its attribute (e.g., ͗p idϭ "1"͘). Certain attribute value types are reserved for referencing (e.g., IDREF). An XML element is accessed typically using the XPath language. Child elements and their parent element are separated by a slash. For example, the XPath /header/author/first accesses the first element from the root element header, and then the author element.It is possible to use other mark-up languages (e.g., HTML) or proprietary formats but XML appears to be suitable for a wide variety of information retrieval (IR) tasks, specific enough to reduce modeling complexity and open enough for easy and rapid adoption. A major advantage of XML over HTML is that users can define their own tags. Tag names are typically chosen to incorporate some relationship to the semantics of the contents or the type of co...

No abstract

No abstract