Data Warehousing Process Modeling from Classical Approaches to New Trends: Main Features and Comparisons

Abstract

The extract, transform, and load (ETL) process is at the core of data warehousing architectures. As such, the success of data warehouse (DW) projects is essentially based on the proper modeling of the ETL process. As there is no standard model for the representation and design of this process, several researchers have made efforts to propose modeling methods based on different formalisms, such as unified modeling language (UML), ontology, model-driven architecture (MDA), model-driven development (MDD), and graphical flow, which includes business process model notation (BPMN), colored Petri nets (CPN), Yet Another Workflow Language (YAWL), CommonCube, entity modeling diagram (EMD), and so on. With the emergence of Big Data, despite the multitude of relevant approaches proposed for modeling the ETL process in classical environments, part of the community has been motivated to provide new data warehousing methods that support Big Data specifications. In this paper, we present a summary of relevant works related to the modeling of data warehousing approaches, from classical ETL processes to ELT design approaches. A systematic literature review is conducted and a detailed set of comparison criteria are defined in order to allow the reader to better understand the evolution of these processes. Our study paints a complete picture of ETL modeling approaches, from their advent to the era of Big Data, while comparing their main characteristics. This study allows for the identification of the main challenges and issues related to the design of Big Data warehousing systems, mainly involving the lack of a generic design model for data collection, storage, processing, querying, and analysis.

Optimization of Transaction Processing System (TPS) Using RAD With FAST Method

“…Inputs capture or collect raw data from within the organization or from the external environment. Processing transfers the lines of input into a more meaningful format [4]. The output transfers the processed information to the people who will use it or to the activities that need it.…”

“…Reports can be generated by simply inputting the period of the desired report and reports can also be made in detail according to the categories provided. This purchasing, inventory and sales information system processes inventory data only to incoming and outgoing goods data derived from purchases, sales, purchase returns, and sales returns [14] [15]. while supervision of the quantity of stock that has reached the reorder point still has to be done manually.…”

Design and implementation of the web (extract, transform, load) process in data warehouse application

“…The ETL process refers to a sequential set of operations that includes extracting, transforming and loading data [18]. ETL process is the core and backbone of the DW architecture [19]. It aims to standardize the processing of the source data and make the data available and appropriate for the intended use [7].…”

“…Extraction: external data are extracted from various source systems to an intermediate stage [3], [19]. In this stage, transformations are conducted to ensure that the performance of the source system does not deteriorate.…”

Comparative Analysis of ETL Tools in Big Data Analytics

“…The Extract, Transform, and Load (ETL) method is the most typical method for collecting data from many sources and integrating it into a single source of information, i.e., a data warehouse [2]. During the ETL process, data is extracted from such a resource as a relational database, XML files, flat files, or spreadsheets, transformed to data warehouse standards, and then loaded into the warehouse [3].…”