A partition-based approach to support streaming updates over persistent data in an active datawarehouse

Lutteroth

et al. 2017

Information Systems

Semi-stream join algorithms join a fast data stream with a disk-based relation. This is important, for example, in real-time data warehousing where a stream of transactions is joined with master data before loading it into a data warehouse. In many important scenarios, the stream input has a skewed distribution, which makes certain performance optimisations possible.We propose two such optimisation techniques: 1) a caching technique for frequently used master data, and 2) a technique for selective load shedding of stream tuples. The caching technique is fine-grained, operating on a tuple-level. Furthermore, it is generic in the sense that it can be applied to different semistream join algorithms to deal with high loads. We analyze it by combining it with various well-known semi-stream joins, and show that it improves the service rate by more than 40% for typical data with skewed distributions. The load shedding technique sheds the fraction of the stream that is most expensive to join. In contrast to existing approaches, the service rate improves under load shedding. We present experimental data showing significant improvements as compared to related approaches and perform a sensitivity analysis for various internal parameters.

Section: Related Workmentioning

confidence: 99%

“…The generic caching technique differs from other approaches such as [7,8] in two aspects. 1) Granularity: the technique is a tuple-level cache, i.e.…”

Section: Cachingmentioning

confidence: 99%

mentioning

confidence: 97%

See 1 more Smart Citation

Skewed distributions in semi-stream joins: How much can caching help?

Lutteroth

et al. 2017

Information Systems

“…Most recently a partition-based approach (Chakraborty & Singh, 2009) was introduced that focuses on minimizing the disk overhead in the MESHJOIN algorithm. However, a switch operator is introduced to switch between the Index Nested Loop Join (INLJ) and MESHJOIN.…”

Section: Related Workmentioning

confidence: 99%

HYBRIDJOIN for Near-Real-Time Data Warehousing

Developments in Data Extraction, Management, and Analysis

Weber

2013

“…One approach for improving MESHJOIN has been a partition-based join algorithm [10] which can also deal with stream intermittence. It uses a two-level hash table in order to attempt to join stream tuples as soon as they arrive, and uses a partition-based waiting area for other stream tuples.…”

Section: Related Workmentioning

confidence: 99%

X-HYBRIDJOIN for Near-Real-Time Data Warehousing

Lecture Notes in Computer Science

Weber

2011

Abstract. In order to make timely and effective decisions, businesses need the latest information from data warehouse repositories. To keep these repositories up-to-date with respect to end user updates, nearreal-time data integration is required. An important phase in near-realtime data integration is data transformation where the stream of updates is joined with disk-based master data. The stream-based algorithm Mesh Join (MESHJOIN) has been proposed to amortize disk access. MESHJOIN makes no assumptions about the data distribution. In real world applications, however, skewed distributions can be found, e.g, certain products are sold more frequently than the remainder of the products. The question arises, how much does MESHJOIN loose in terms of performance by not adapting to data skew. In this paper we perform a rigorous experimental study analyzing the possible performance improvements while considering typical data distributions. For this purpose we design an algorithm Extended Hybrid Join (X-HYBRIDJOIN) that is complementary to MESHJOIN in that it can adapt to data skew and stores parts of the master data in memory permanently, reducing the disk access overhead significantly. We compare the performance of X-HYBRIDJOIN against the performance of MESHJOIN. We take several precautions to make sure the comparison is adequate and focuses on the utilization of data skew. The experiments show that considering data skew offers substantial room for performance gains that cannot be used by non-adaptive approaches such as MESHJOIN.