TT-Join: Efficient Set Containment Join

Yang, Jianye; Zhang, Wenjie; Yang, Shiyu; Zhang, Ying; Lin, Xuemin

doi:10.1109/icde.2017.107

Cited by 19 publications

(23 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…al introduce a OIF index combined the inverted index with Btree to tackle three kinds of set-containment queries: subset queries, equality queries and superset queries. In a recent work ( [30]), J. Yang et. al propose a TT-join method for the set containment join problem, which is based on prefix tree structure and utilize the element frequency information; they also present a detailed summary of the existing set-containment join methods.…”

Section: Related Workmentioning

confidence: 99%

“…For instance, such set-valued attributes may correspond to the profile of a person, the tags of a post, the domain information of a webpage, and the tokens or q-grams of a document. In the literature, there has been a variety of interests in the computation of set-valued records including set containment search (e.g., [6], [18], [24], [32]), set similarity joins (e.g., [27], [29]), and set containment joins (e.g., [10], [21], [22], [30]).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Selectivity Estimation on Set Containment Search

Yang

Zhang

et al. 2019

Database Systems for Advanced Applications

Self Cite

View full text Add to dashboard Cite

In this paper, we study the problem of selectivity estimation on set containment search. Given a query record Q and a record dataset S, we aim to accurately and efficiently estimate the selectivity of set containment search of query Q over S. The problem has many important applications in commercial fields and scientific studies. To the best of our knowledge, this is the first work to study this important problem. We first extend existing distinct value estimating techniques to solve this problem and develop an inverted list and G-KMV sketch based approach IL-GKMV. We analyse that the performance of IL-GKMV degrades with the increase of vocabulary size. Motivated by limitations of existing techniques and the inherent challenges of the problem, we resort to developing effective and efficient sampling approaches and propose an ordered trie structure based sampling approach named OT-Sampling. OT-Sampling partitions records based on element frequency and occurrence patterns and is significantly more accurate compared with simple random sampling method and IL-GKMV. To further enhance performance, a divide-and-conquer based sampling approach, DC-Sampling, is presented with an inclusion/exclusion prefix to explore the pruning opportunities. We theoretically analyse the proposed techniques regarding various accuracy estimators. Our comprehensive experiments on 6 real datasets verify the effectiveness and efficiency of our proposed techniques.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Selectivity Estimation on Set Containment Search

Yang

Zhang

et al. 2019

Database Systems for Advanced Applications

Self Cite

View full text Add to dashboard Cite

show abstract

“…In one of the representative works on set containment search [28], Terrovitis et al introduce a OIF index combined the inverted index with B-tree to tackle three kinds of set containment queries: subset queries, equality queries and superset queries. In a recent work [34], Yang et al propose a TT-join method for the set containment join problem, which is based on prefix tree structure and utilize the element frequency information; they also present a detailed summary of the existing set containment join methods. The containment queries can also be modeled as range searching problem in computational geometry [8]; nevertheless, the performance is exponentially dependent on dimension n which is unsuitable in practice for our problem.…”

Section: Searching Set-valued Datamentioning

confidence: 99%

“…For instance, such set-valued attributes may correspond to the profile of a person, the tags of a post, the domain information of a webpage and the tokens or q-grams of a document. In the literature, there has been a variety of interests in the computation of set-valued records including set containment search (e.g., [9,21,28,36]), set similarity joins (e.g., [31,33]) and set containment joins (e.g., [13,24,25,34]).…”

Section: Introductionmentioning

confidence: 99%

Selectivity Estimation on Set Containment Search

Yang

Zhang

et al. 2019

Data Sci. Eng.

Self Cite

View full text Add to dashboard Cite

In this paper, we study the problem of selectivity estimation on set containment search. Given a query record Q and a record dataset S , we aim to accurately and efficiently estimate the selectivity of set containment search of query Q over S. We first extend existing distinct value estimating techniques to solve this problem and develop an inverted list and G-KMV sketchbased approach IL-GKMV. We analyze that the performance of IL-GKMV degrades with the increase in vocabulary size. Motivated by limitations of existing techniques and the inherent challenges of the problem, we resort to developing effective and efficient sampling approaches and propose an ordered trie structure-based sampling approach named OT-Sampling. OT-Sampling partitions records based on element frequency and occurrence patterns and is significantly more accurate compared with simple random sampling method and IL-GKMV. To further enhance the performance, a divide-and-conquer-based sampling approach, DC-Sampling, is presented with an inclusion/exclusion prefix to explore the pruning opportunities. Meanwhile, we consider weighted set containment selectivity estimation and devise stratified random sampling approach named StrRS. We theoretically analyze the proposed techniques regarding various accuracy estimators. Our comprehensive experiments on nine real datasets verify the effectiveness and efficiency of our proposed techniques.

show abstract

“…For example, a set-valued attribute of a tuple may record the prerequisites of a course, or the labels of a digital image, or the tokens in an email, and so on. With this comes a large body of research interests on efficient algorithms for fundamental operations on such attributes such as containment joins [1][2][3][4][5][6][7][8][9][10][11], containment queries (e.g., [12]), and similarity joins (e.g., [13]). This paper focuses on set containment join (SCJ).…”

Section: Introductionmentioning

confidence: 99%

FreshJoin: An Efficient and Adaptive Algorithm for Set Containment Join

et al. 2019

View full text Add to dashboard Cite

This paper revisits set containment join (SCJ) problem, which uses the subset relationship (i.e., ⊆) as condition to join set-valued attributes of two relations and has many fundamental applications in commercial and scientific fields. Existing in-memory algorithms for SCJ are either signature-based or prefix-tree-based. The former incurs high CPU cost because of the enumeration of signatures, while the latter incurs high space cost because of the storage of prefix trees. This paper proposes a new adaptive parameter-free in-memory algorithm, named as frequency-hash join or in short, to evaluate SCJ efficiently. builds a flat index on-the-fly to record three kinds of signatures (i.e., two least frequent elements and a hash signature whose length is determined adaptively by the frequencies of elements in the universe set). The index consists of two sparse inverted indices and two arrays which record hash signatures of all sets in each relation. The index is well organized such that can avoid enumerating hash signatures. The rationality of this design is explained. And, the time and space cost of the proposed algorithm, which provide a rule to choose from existing algorithms, are analyzed. Experiments on 16 real-life datasets show that usually reduces more than 50% of space cost while remains as competitive as the state-of-the-art algorithms in running time.

show abstract

TT-Join: Efficient Set Containment Join

Cited by 19 publications

References 21 publications

Selectivity Estimation on Set Containment Search

Selectivity Estimation on Set Containment Search

Selectivity Estimation on Set Containment Search

FreshJoin: An Efficient and Adaptive Algorithm for Set Containment Join

Contact Info

Product

Resources

About