Deterministic Algorithm for the t-Threshold Set Problem

Abstract

Abstract. Given k sorted arrays, the t-Threshold problem, which is motivated by indexed search engines, consists of finding the elements which are present in at least t of the arrays. We present a new deterministic algorithm for it and prove that, asymptotically in the sizes of the arrays, it is optimal in the alternation model used to study adaptive algorithms. We define the Opt-Threshold problem as finding the smallest non empty t-threshold set, which is equivalent to find the largest t such that the t-threshold set is non empty, and propose a naive algorithm to solve it.

Alternation and redundancy analysis of the intersection problem

“…The t-threshold set and opt-threshold set problems [Barbay and Kenyon 2003] are natural generalizations of the intersection problem, which could be useful in indexed search engines. The redundancy seems to be important in the complexity of these problems as well, but a proper measure is harder to define in this context.…”

Compressed bitmap indexes: beyond unions and intersections

“…Our work has considered N values up to a few thousand (at most 11,115). Yet datasets whose indexes have millions of bitmaps are not out of the question.…”

“…A similar algorithm was earlier presented by Barbay and Kenyon . Any input may appear in their heap, but at any time, there will be T − 1 inputs that are not in the heap.…”

“…If each such observation triggers Ω(1) work (as it does with SCANCOUNT , Section 5.1, when a counter is incremented), this implies an Ω( T X ) lower bound. Barbay and Kenyon have established a data‐dependent lower bound for the problem, assuming the data are presented in sorted arrays and using a model where comparisons are the only allowed operations on array elements. However, both bounds leave open the possibility of using parallelism.…”

“…Algorithm Source Section Main idea SCANCOUNT [9] 5.1 Allocate array of counters, scan values while incrementing counters and, finally, scan array of counters for matching counts MGOPT [10,11] 5.2 Set aside the largest T 1 inputs, merge the remaining N T C 1 inputs using a heap and then look up matching values in the largest inputs DSK [9] 5.2 Similar to MGOPT, but during the merger of the small inputs, some values are skipped; requires a tuning parameter BSTM Modified from [8,12] 5.3.1 Transforms the query into a Boolean circuit to be evaluated on the bitmaps W2CTI Novel 6.1 Merge inputs two by two starting with lowest-cardinality inputs while maintaining counters and prune results as early as possible LOOPED Novel 6.2 Allocates T temporary bitmaps corresponding to the count values 1; 2; : : : ; T ; the first bitmap updates the first temporary bitmap, the second bitmap updates the first two temporary bitmaps, and so on RBMRG Novel, inspired by [13] 6.3 Using a heap, merge Run-length-encoding-compressed words unions, which are well understood, we assume that 2 6 T 6 N 1. These queries are often called T -overlap [15,16], T -occurrence [9,17] or T -threshold [11,18] queries. We can map a T -overlap query to a query over bitmaps using a Boolean threshold function: given N bits, the T -threshold function #.T; ¹b 1 ; : : : ; b N º/ returns true if at least T bits are true; it returns false otherwise.…”