Information filtering and query indexing for an information retrieval model

Abstract

In the information filtering paradigm, clients subscribe to a server with continuous queries or profiles that express their information needs. Clients can also publish documents to servers. Whenever a document is published, the continuous queries satisfying this document are found and notifications are sent to appropriate clients. This article deals with the filtering problem that needs to be solved efficiently by each server: Given a database of continuous queries db and a document d , find all queries q ∈ db that match d . We present data structures and indexing algorithms that enable us to solve the filtering problem efficiently for large databases of queries expressed in the model AWP. AWP is based on named attributes with values of type text, and its query language includes Boolean and word proximity operators.

A Methodology for the Automatic Creation of Massive Continuous Query Datasets from Real Life Corpora

“…The robustness of the proposed methodology is highlighted not only by the publications in top-class venues that utilize it (e.g., [11,12,16,18,22]), but also by the different document corpora it was applied on (TREC .gov, TREC ClueWeb09, OHSUMED, NN, and others). Finally, interesting directions for future work include the design and implementation of modules for creating realistic vector space and semi-structured continuous queries.…”

“…Some of these approaches include the systems XFilter [1], YFilter [6], DFA [10], the Boolean version of SIFT [17], and the agent-based DIAS [11]. Other approaches focus more on the algorithmic aspect by providing efficient treebased data structures such as [12,16,18,19,20] for dealing with documents that are free text and profiles that are conjunctions of keywords. To the best of our knowledge the only work that is somewhat relevant to ours is [15], where a corpus of documents (but no continuous queries) is built for adaptive filtering tasks.…”

“…To this end, a number of systems and algorithms that try to solve the filtering problem efficiently for different data models and query languages have been proposed [1,6,10,17,11,12,16,18,19,20]. However, despite all the research in the area, there is an apparent lack of a benchmarking mechanism (in the form of a large-scale standarised test collection of continuous queries and the relevant document publications) specifically created for evaluating filtering tasks.…”

“…Notice also that one-time queries, such as those obtained from public releases of major search engines' query logs (like Google BigQuery, Zeitgeist, or the AOL query set) are inappropriate for filtering tasks as they typically express a one-time information need, contrary to continuous queries that are used to express recurrent and long-standing information needs. Finally, other efforts, such as the TREC Filtering Track, are insufficient as they contain only a few dozens of manually created and curated continuous queries, and cannot live up to the need of modern benchmarking that is in the order of millions (e.g., as in [12,16,18,19,20]). …”

MOVE: A Large Scale Keyword-Based Content Filtering and Dissemination System

“…We opt to use the data traces of traditional web search services, which are truly representative of the end user behavior to use keywords in the real world. Similar approaches are used by many previous works [23] We use an input query history file (containing 4,000,000 pre-processed queries) collected from the Microsoft MSN search engine (MSN in short). On average, the number of terms per query is 2.843 in MSN.…”