Striving towards Near Real-Time Data Integration for Data Warehouses

Abstract

Abstract. The amount of information available to large-scale enterprises is growing rapidly. While operational systems are designed to meet well-specified (short) response time requirements, the focus of data warehouses is generally the strategic analysis of business data integrated from heterogeneous source systems. The decision making process in traditional data warehouse environments is often delayed because data cannot be propagated from the source system to the data warehouse in time. A real-time data warehouse aims at decreasing the time it takes to make business decisions and tries to attain zero latency between the cause and effect of a business decision. In this paper we present an architecture of an ETL environment for real-time data warehouses, which supports a continual near real-time data propagation. The architecture takes full advantage of existing J2EE (Java 2 Platform, Enterprise Edition) technology and enables the implementation of a distributed, scalable, near real-time ETL environment. Instead of using vendor proprietary ETL (extraction, transformation, loading) solutions, which are often hard to scale and often do not support an optimization of allocated time frames for data extracts, we propose in our approach ETLets (spoken "et-lets") and Enterprise Java Beans (EJB) for the ETL processing tasks.

A Unified View of Data-Intensive Flows in Business Intelligence Systems: A Survey

“…This brought the idea of (near) real-time data warehousing systems. Several approaches discussed the main requirements of such systems and proposed architectural changes to traditional DW systems for satisfying these new requirements (e.g., [8,97]). For instance, besides the main requirement of data freshness, [97] has also indicated minimal overhead of the source system and scalability in terms of input data sources, user queries, and data volumes, as relevant for these systems.…”

“…They however pointed out the contradiction between users need for maximal data freshness and completeness, and the high overhead of the traditional DW workflows that often require costly data cleaning and transformations. To this end, the approaches in [8] and [97] discuss both conceptual and technological changes that would balance the delays in traditional ETL processes. In practice, SAP Business Warehouse [66] is an example of such system.…”

A distributed tree data structure for real-time OLAP on cloud architectures

“…Practitioners have therefore called for some time for an integrated OLAP/OLTP approach with a real-time OLAP system that gets updated instantaneously as new data arrives and always provides an upto-date data warehouse for the decision support process (e.g. [9]). Some recent publications have begun to address this problem by providing "quasi real-time" incremental maintenance schemes and loading procedures for static data cubes (e.g.…”

“…Some recent publications have begun to address this problem by providing "quasi real-time" incremental maintenance schemes and loading procedures for static data cubes (e.g. [9], [10], [11], [12]). However, these approaches are not fully realtime.…”

“…However, these systems are not designed for real-time (OLTP style) operation. Instead, they use batch processing similar to [9], [10], [11], [12]. The situation is similar for BigTable [21], BigQuery [22], and Dremel [23].…”