The algorithm of the join data stream with diskresident relation

Abstract

Current data integration approaches are moving towards real-time updates. One important element in real-time data integration is the join of a continuous incoming data stream with a disk-resident relation. Because data stream is infinite, it is impossible to adopt blocking join algorithms such as sort-merge and hash join. The novel algorithm MESHJOIN has been proposed for joining a continuous stream with a disk-resident relation. The crux of MESHJOIN algorithm is that the whole memory block of disk-based relation is replaced at each iteration. We propose that the memory block is divided into a number of logical partitions, and then only one logical partition of memory block is replaced at each iteration. The experimental results show that the service rate of the join is increased because I/O cost for one loop iteration is decreased. Keywords-relational data stream; join; disk-resident relation; logical partition I.0 B INTRODUCTION Active data warehousing has emerged as a new business intelligence (BI) paradigm where updates from the operational stores are propagated in real-time to the repository. This shift of practices significantly affects the ETL (extract transfer load) process as the type of joins between an infinite stream of incoming records and some stored data warehouse table. The core operation is identified byR where S is viewed as append-only sequences of data tuples where all tuples of the same stream have the same schema, R is a large, disk-resident, warehouse relation, and the join condition C depends on the semantics of the transformation. An active data warehouse requires the evaluation of this expression on line as the tuples of S are streamed from the operational sources, in order to ensure that the updates are propagated in a timely fashion. The major challenge is to handle the fast arrival rate of S tuples relative to the slow I/O access of R.While there is a lot of research on how to join relations that are either both stored or streaming, there is little work devoted to the problem of joining a streaming relation with a store one. Whether we can adapt blocking join algorithms such as sort-merge [15] and hash join to the join C S R , it will require a considerable amount of disk buffering for S tuples which would slow down the join operation [13,14]. The indexed nested-loop join also satisfy the setting, but it may require the maintenance of an additional index on R and probe the index incurs expensive random I/Os. The work in [1] first drew attention to the need to support streaming updates in an active data warehouse. The proposed MESHJOIN algorithm utilizes sequential reads from the stored relation in a round robin fashion in order to amortize the I/O look up cost for a large number of stream tuples. The work in [3,4,5] proposes some revised algorithm.In this paper we contribute by introducing block stream algorithm for joining a relational stream with a disk-resident relation. We increase the stream buffer so that memory can accommodate more windows, thereby reducing disk block I/O. In t...

Optimizing Semi-Stream CACHEJOIN for Near-Real- Time Data Warehousing

“…The problem of joining a streaming data with a stored data was first introduced in (Neoklis Polyzotis, Skiadopoulos, Vassiliadis, Simitsis, & Frantzell, 2008) and as a solution a seminal algorithm called MESHJOIN (Mesh Join) was presented. Later, various optimizations in MESHJOIN have been proposed (Bornea et al, 2011), (Naeem et al, 2012a), (Naeem, Dobbie, Weber, & Alam, 2010), (Naeem, Weber, Dobbie, & Lutteroth, 2013) , (Du & Zou, 2013), (Naeem, Dobbie, & Weber, 2012b). Since the concept of long tail is very common in sales data (Kleinberg, 2002), CACHEJOIN (Naeem et al, 2012a), one of these algorithms, was particularly designed for irregular streams by caching the frequent records of stored data.…”

“…Significance of real-time business data devalues, as it gets older. At the same time, the traditional working hours for global enterprises are not germane as they continue to serve customers around the globe and around the clock every day (Golfarelli & Rizzi, 2009), (Vassiliadis, 2009) and (Thomsen & Pedersen, 2005). For uninterrupted global services, continuous real-time data availability for in time business decisions and actions is crucial and indispensable.…”