Semi-Stream Similarity Join Processing in a Distributed Environment

Kim, Hong-Ji; Lee, Kihoon

doi:10.1109/access.2020.3009414

Cited by 6 publications

(2 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DSim-Join [18] was another algorithm which was proposed to address the problem of semi-stream join. DSim-Join aimed to minimize cache-based database accessed in a distributed stream processing system, equalized the load between parallel join threads, counterbalanced join processing, and decreased the data transmission.…”

Section: Semi-stream Join Algorithmsmentioning

confidence: 99%

TinyLFU-based semi-stream cache join for near-real-time data warehousing

et al. 2022

View full text Add to dashboard Cite

Semi-stream join is an emerging research problem in the domain of near-real-time data warehousing. A semi-stream join is basically a join between a fast stream (S) and a slow disk-based relation (R). In the modern era of technology, huge amounts of data are being generated swiftly on a daily basis which needs to be instantly analyzed for making successful business decisions. Keeping this in mind, a famous algorithm called CACHEJOIN (Cache Join) was proposed. The limitation of the CACHEJOIN algorithm is that it does not deal with the frequently changing trends in a stream data efficiently. To overcome this limitation, in this paper we propose a TinyLFU-CACHEJOIN algorithm, a modified version of the original CACHEJOIN algorithm, which is designed to enhance the performance of a CACHEJOIN algorithm. TinyLFU-CACHEJOIN employs an intelligent strategy which keeps only those records of R in the cache that have a high hit rate in S. This mechanism of TinyLFU-CACHEJOIN allows it to deal with the sudden and abrupt trend changes in S. We developed a cost model for our TinyLFU-CACHEJOIN algorithm and proved it empirically. We also assessed the performance of our proposed TinyLFU-CACHEJOIN Article Title algorithm with the existing CACHEJOIN algorithm on a skewed synthetic dataset. The experiments proved that TinyLFU-CACHEJOIN algorithm significantly outperforms the CACHEJOIN algorithm.

show abstract

Section: Semi-stream Join Algorithmsmentioning

confidence: 99%

TinyLFU-based semi-stream cache join for near-real-time data warehousing

et al. 2022

View full text Add to dashboard Cite

show abstract

“…A number of algorithms have been proposed to process realtime stream in an efficient way [41]- [46] for RTDW [47]. During processing stream with disk data, the data is coming from two sources.…”

Section: Literature Reviewmentioning

confidence: 99%

An Efficient Data Access Approach With Queue and Stack in Optimized Hybrid Join

2021

View full text Add to dashboard Cite

As rapid decision making in business organizations gain in popularity, the complexity and adaptability of extract, transform, and load (ETL) process of near real-time data warehousing has dramatically increased. The most important part of near real-time data warehouse is to feed new data from different data sources on near-real-time basis. However, this new data is not in the format of the data warehouse therefore, it needs to be transformed into the required format by using transformation algorithms which is essential part of ETL process. A semi-stream join algorithm is required to implement this transformation, for this purpose a HYBRIDJOIN (hybrid join) algorithm has been presented in the literature. However, major design issue with this algorithm is that it uses a single buffer to load the disk partitions and therefore, the algorithm has to wait until the next disk partition overwrites the exiting partition in the disk buffer. As the cost of loading disk partition into disk buffer is the major cost of overall algorithm processing cost, this leaves the performance of algorithm sub-optimal. Moreover, existing approaches only considering the oldest key join attributes for finding the matches with master data and maintaining the Queue of key join attribute. However, performance can be improved if recent and oldest attributes process in parallel. This article addresses the limitation of HYBRIDJOIN by presenting two optimized new algorithms named: Parallel-Hybrid Join (P-HYBRIDJOIN) and Hybrid Join with Queue and Stack (QaS-HYBRIDJOIN). Proposed algorithms aim to reduce major processing cost that is disk I/O as well as to increase number of matching stream tuples. Both of these algorithms perform significantly better in terms of throughput and number of matching tuples as compared to existing approaches. Performance analysis and cost model for proposed algorithms show the best performance using intermittent stream data under limited resources.

show abstract

Distributed real-time ETL architecture for unstructured big data

Mehmood

Anees

2022

Knowl Inf Syst

View full text Add to dashboard Cite

Semi-Stream Similarity Join Processing in a Distributed Environment

Cited by 6 publications

References 37 publications

TinyLFU-based semi-stream cache join for near-real-time data warehousing

TinyLFU-based semi-stream cache join for near-real-time data warehousing

An Efficient Data Access Approach With Queue and Stack in Optimized Hybrid Join

Distributed real-time ETL architecture for unstructured big data

Contact Info

Product

Resources

About