Text joins in an RDBMS for web data integration

Gravano, Luis; Ipeirotis, Panagiotis G.; Koudas, Nick; Srivastava, Divesh

doi:10.1145/775165.775166

Cited by 47 publications

(67 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also tried to compare to other recently published methods, such as [5,21], but none of these is available for download. In particular, we acknowledge that comparison against Unix command line tools are not completely satisfactory, as our method first builds an index of the set to be searched.…”

Section: Methodsmentioning

confidence: 99%

“…We could not compare PETER with this algorithm as there is no implementation available. The benefit of using positional q-grams was shown in [5]. As the authors implemented a sampling-based approximation for similarity string joins, one cannot directly compare PETER to their tool.…”

Section: Related Workmentioning

confidence: 99%

“…Applications arise in duplicate detection [15], error correction [13] and data cleansing [5], to name only a few. They are also of uttermost importance in the Life Sciences.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Prefix Tree Indexing for Similarity Search and Similarity Joins on Genomic Data

Rheinländer

Knobloch

Hochmuth

et al. 2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Similarity search and similarity join on strings are important for applications such as duplicate detection, error detection, data cleansing, or comparison of biological sequences. Especially DNA sequencing produces large collections of erroneous strings which need to be searched, compared, and merged. However, current RDBMS offer similarity operations only in a very limited and inefficient form that does not scale to the amount of data produced in Life Science projects. We present PETER, a prefix tree based indexing algorithm supporting approximate search and approimate joins. Our tool supports Hamming and edit distance as similarity measure and is available as C++ library, as Unix command line tool, and as cartridge for a commercial database. It combines an efficient implementation of compressed prefix trees with advanced pre-filtering techniques that exclude many candidate strings early. The achieved speed-ups are dramatic, especially for DNA with its small alphabet. We evaluate our tool on several collections of long strings containing up to 5,000,000 entries of length up to 3,500. We compare its performance to agrep, nrgrep, and user-defined functions inside a relational database. Our experiments reveal that PETER is faster by orders of magnitudes compared to the command-line tools. Compared to RDBMS, it computes similarity joins in minutes for which UDFs did not finish within a day and outperforms the built-in join methods even in the exact case.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Prefix Tree Indexing for Similarity Search and Similarity Joins on Genomic Data

Rheinländer

Knobloch

Hochmuth

et al. 2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Therefore, we adopt the following method, using the "n-gram" technique where n = 3 that has been experimentally shown to work well [11].…”

Section: N and Nmentioning

confidence: 99%

gStore: a graph-based SPARQL query engine

et al. 2013

View full text Add to dashboard Cite

We address efficient processing of SPARQL queries over RDF datasets. The proposed techniques, incorporated into the gStore system, handle, in a uniform and scalable manner, SPARQL queries with wildcards and aggregate operators over dynamic RDF datasets. Our approach is graph-based. We store RDF data as a large graph, and also represent a SPARQL query as a query graph. Thus the query answering problem is converted into a subgraph matching problem. To achieve efficient and scalable query processing, we develop an index, together with effective pruning rules and efficient search algorithms. We propose techniques that use this infrastructure to answer aggregation queries. We also propose an effective maintenance algorithm to handle online updates over RDF repositories. Extensive experiments confirm the efficiency and effectiveness of our solutions.

show abstract

“…There are many ways to evaluate the similarity between textual fields in literature [13], [7], [5]. What we apply in our solution is the TF.IDF method which is effectively used in the WHIRL system [5].…”

Section: Definition 4 Given a Distinctness Threshold D T A Column Cmentioning

confidence: 99%

Information Integration

SpringerReference

View full text Add to dashboard Cite

Abstract-Information integration has been widely addressed over the last several decades. However, it is far from solved due to the complexity of resolving schema and data heterogeneities. In this paper, we propose out attempt to alleviate such difficulty by realizing keyword search functionality for integrating information from heterogeneous databases. Our solution does not require predefined global schema or any mappings between databases. Rather, it relies on an operator called keyword join to take a set of lists of partial answers from different data sources as input, and output a list of results that are joined by the tuples from input lists based on predefined similarity measures as integrated results. Our system allows source databases remain autonomous and the system to be dynamic and extensible. We have tested our system with real dataset and benchmark, which shows that our proposed method is practical and effective.

show abstract

Text joins in an RDBMS for web data integration

Cited by 47 publications

References 12 publications

Prefix Tree Indexing for Similarity Search and Similarity Joins on Genomic Data

Prefix Tree Indexing for Similarity Search and Similarity Joins on Genomic Data

gStore: a graph-based SPARQL query engine

Information Integration

Contact Info

Product

Resources

About