Filtered document retrieval with frequency-sorted indexes

Abstract

Ranking techniques are effective at finding answers in document collections but can be expensive to evaluate. We propose an evaluation technique that uses early recognition of which documents are likely to be highly ranked to reduce costs; for our test data, queries are evaluated in 2% of the memory of the standard implementation without degradation in retrieval effectiveness. Cpu time and disk traffic can also be dramatically reduced by designing inverted indexes explicitly to support the technique. The principle of the index design is that inverted lists are sorted by decreasing within‐document frequency rather than by document number, and this method experimentally reduces cpu time and disk traffic to around one third of the original requirement. We also show that frequency sorting can lead to a net reduction in index size, regardless of whether the index is compressed. © 1996 John Wiley & Sons, Inc.

Using Information Retrieval techniques for supporting data mining

“…In the list above the first and second chunk contain one pointer, and the third one two. The final frequency-sorted index size is rarely larger than when generated the classic way [29].…”

Algebraic query optimization for distributed top-k queries

“…Top-k query processing has received much attention in a variety of settings such as similarity search on multimedia data [7,24,29,30,45,46], ranked retrieval on text and semistructured documents in digital libraries and on the Web [3,6,36,40,48,52,55], network and stream monitoring [4,14] collaborative recommendation and preference queries on ecommerce product catalogs [17,31,42,56], and ranking of SQL-style query results on structured data sources in general [1,11,18]. Among the ample work on top-k query processing, the TA family of algorithms for monotonic score aggregation [25,30,46] stands out as an extremely efficient and highly versatile method.…”

Retrieval quality vs. effectiveness of specificity-oriented search in XML collections

“…By modifying the output criterion, also incremental-tunable algorithms are possible. As a specific instance of this class of algorithms, the Nosferatu algorithm (Pfeifer and Pennekamp 1997) assumes that inverted list entries are sorted by decreasing indexing weights (a similar algorithm has been described in Persin et al 1996). In this case, inverted lists are processed in parallel, and entries are read in the order of decreasing RSV increments.…”