The challenges of Extract, Transform and Loading (ETL) system implementation for near real-time environment

Sabtu, Adilah; Azmi, Nurulhuda Firdaus Mohd; Sjarif, Nilam Nur Amir; Ismail, Saiful Adli; Yusop, Othman Mohd; Sarkan, Haslina Md; Chuprat, Suriayati

doi:10.1109/icriis.2017.8002467

Cited by 25 publications

(11 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Tahap selanjutnya atau tahap ketiga adalah proses Extract Transform Load (ETL). Proses ETL merupakan proses mentransformasikan data transaksi agar dapat tersimpan pada datawahouse yang telah dirancang sebelumnya [10]. Data yang diambil pada data transaksi merupakan data yang benar-benar dibutuhkan untuk proses analisa dan pengembilan keputusan [11].…”

Section: Metode Penelitianunclassified

Penerapan Teknologi Datawarehouse Nosql Dan Business Intelligence Untuk Analisa Transaksi Penjualan

Handika¹,

Tama²,

Krisnayanti³

2020

jurnalresistor

View full text Add to dashboard Cite

Business process are increasingly developing, causing data generated will be even greater. The application of datawarehouse technology and MongoDB as one of the NoSQL technologies can solve large historical data processing problems. In addition, business intelligence technology that is able to visualize data in into graphics and tables will make it easier for stakeholders to analyze sales transactions. The process for transforming transaction data into a datawarehouse is the Extract Transform Load (ETL) process. The data generated from the ETL process is quality data and is really needed for analysis. Black box testing results and data validity indicate the system can be accepted and can produce the right information for decision support.

show abstract

Section: Metode Penelitianunclassified

Penerapan Teknologi Datawarehouse Nosql Dan Business Intelligence Untuk Analisa Transaksi Penjualan

Handika¹,

Tama²,

Krisnayanti³

2020

jurnalresistor

View full text Add to dashboard Cite

show abstract

“…They also enable ETL to always improve performance when needed by adding more servers to the cluster. Sabtu et al [27] also point out the challenges that near real-time solutions faces but focus on multiple and heterogeneous data sources, data backups, inconsistency, performance degradation, data source overload and master data overhead. Table 1 sums up these solutions and related problems, also described in detail below.…”

Section: Related Workmentioning

confidence: 99%

DOD-ETL: distributed on-demand ETL for near real-time business intelligence

Machado

Cunha

Pereira

et al. 2019

J Internet Serv Appl

View full text Add to dashboard Cite

The competitive dynamics of the globalized market demand information on the internal and external reality of corporations. Information is a precious asset and is responsible for establishing key advantages to enable companies to maintain their leadership. However, reliable, rich information is no longer the only goal. The time frame to extract information from data determines its usefulness. This work proposes DOD-ETL, a tool that addresses, in an innovative manner, the main bottleneck in Business Intelligence solutions, the Extract Transform Load process (ETL), providing it in near real-time. DOD-ETL achieves this by combining an on-demand data stream pipeline with a distributed, parallel and technology-independent architecture with in-memory caching and efficient data partitioning. We compared DOD-ETL with other Stream Processing frameworks used to perform near real-time ETL and found DOD-ETL executes workloads up to 10 times faster. We have deployed it in a large steelworks as a replacement for its previous ETL solution, enabling near real-time reports previously unavailable.

show abstract

“…According to Muddasir e Mohammed [11], by using the loading time window period is just suitable to organizations which allow this feature or that focus on long run goals, which data can be updated in a less frequency. So, the usage of loading time window period to update data is enough to the majority applications, as it had been widely studied and applied in OLAP applications with low cost when is used the incremental data updating approach [12].…”

Section: Introductionmentioning

confidence: 99%

An Innovative Method to Extract Data in a Real-time Data Warehousing Environment

Vilela¹,

Ciferri²

2021

Computing Advances &Amp; Trends

View full text Add to dashboard Cite

ETL (Extract, Transform, and Load) is an essential process required to perform data extraction in knowledge discovery in databases and in data warehousing environments. The ETL process aims to gather data that is available from operational sources, process and store them into an integrated data repository. Also, the ETL process can be performed in a real-time data warehousing environment and store data into a data warehouse. This paper presents a new and innovative method named Data Extraction Magnet (DEM) to perform the extraction phase of ETL process in a real-time data warehousing environment based on non-intrusive, tag and parallelism concepts. DEM has been validated on a dairy farming domain using synthetic data. The results showed a great performance gain in comparison to the traditional trigger technique and the attendance of real-time requirements.

show abstract

The challenges of Extract, Transform and Loading (ETL) system implementation for near real-time environment

Cited by 25 publications

References 12 publications

Penerapan Teknologi Datawarehouse Nosql Dan Business Intelligence Untuk Analisa Transaksi Penjualan

Penerapan Teknologi Datawarehouse Nosql Dan Business Intelligence Untuk Analisa Transaksi Penjualan

DOD-ETL: distributed on-demand ETL for near real-time business intelligence

An Innovative Method to Extract Data in a Real-time Data Warehousing Environment

Contact Info

Product

Resources

About