Performance Comparison of Hadoop Based Tools with Commercial ETL Tools – A Case Study

Misra, Sumit; Saha, Sanjoy Kumar; Mazumdar, Chandan

doi:10.1007/978-3-319-03689-2_12

Cited by 10 publications

(7 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ETLMR was outlined in (Liu, Thomsen & Pedersen, 2012) for a demonstration purpose. (Misra et al, 2013) shows that ETL solutions based on MapReduce frameworks, such as Apache Hadoop, are very efficient and less costly compared to ETL tools market. Recently, the authors in (Liu et al, 2014) have proposed the CloudETL framework which uses Apache Hadoop to parallelize ETL processes and Apache Hive to process data.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, the emergence of big data has generated much interest in the research community. Some authors such as (Liu, Thomsen & Pedersen, 2011), (Liu, Thomsen & Pedersen, 2014), and (Misra, Saha & Mazumdar, 2013) have proposed interesting ETL approaches. Indeed, our study is motivated by the fact that the existing conceptual modeling approaches such as (El Akkaoui & Zimányi, 2009), (Trujillo & Luján-Mora, 2003), and (Vassiliadis et al, 2002) are not suitable for big data environments.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Extracting-Transforming-Loading Modeling Approach for Big Data Analytics

Bala

Boussaïd

Alimazighi

2016

International Journal of Decision Support System Technology

View full text Add to dashboard Cite

Due to their widespread use, Internet, Web 2.0 and digital sensors create data in non-traditional volumes (at terabytes and petabytes scale). The big data characterized by the four V's has brought with it new challenges given the limited capabilities of traditional computing systems. This paper aims to provide solutions which can cope with very large data in Decision-Support Systems (DSSs). In the data integration phase, specifically, the authors propose a conceptual modeling approach for parallel and distributed Extracting-Transforming-Loading (ETL) processes. Among the complexity dimensions of big data, this study focuses on the volume of data to ensure a good performance for ETL processes. The authors' approach allows anticipating on the parallelization/distribution issues at the early stage of Data Warehouse (DW) projects. They have implemented an ETL platform called Parallel-ETL (P-ETL for short) and conducted some experiments. Their performance analysis reveals that the proposed approach enables to speed up ETL processes by up to 33% with the improvement rate being linear.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Extracting-Transforming-Loading Modeling Approach for Big Data Analytics

Bala

Boussaïd

Alimazighi

2016

International Journal of Decision Support System Technology

View full text Add to dashboard Cite

show abstract

“…The conventional ETL system is typically operated on a single machine that cannot effectively handle huge volumes of big data [12]. To deal with the considerable quantity of big data in the ETL process, there have been several attempts in recent years to utilize a parallelized data processing concept [13][14][15].…”

Section: Introductionmentioning

confidence: 99%

“…This study conducted an experimental evaluation assessing system scalability based on different scales of jobs and data to compare with other MapReduce-based tools. Another study [15] compared Hadoop-based ETL solutions with commercial ETL solutions in terms of cost and performance. They concluded that Hadoop-based ETL solutions are better in comparison to existing commercial ETL solutions.…”

Section: Introductionmentioning

confidence: 99%

MapReduce-Based D_ELT Framework to Address the Challenges of Geospatial Big Data

Lee

2019

IJGI

View full text Add to dashboard Cite

The conventional extracting-transforming-loading (ETL) system is typically operated on a single machine not capable of handling huge volumes of geospatial big data. To deal with the considerable amount of big data in the ETL process, we propose D_ELT (delayed extracting-loading -transforming) by utilizing MapReduce-based parallelization. Among various kinds of big data, we concentrate on geospatial big data generated via sensors using Internet of Things (IoT) technology. In the IoT environment, update latency for sensor big data is typically short and old data are not worth further analysis, so the speed of data preparation is even more significant. We conducted several experiments measuring the overall performance of D_ELT and compared it with both traditional ETL and extracting-loading-transforming (ELT) systems, using different sizes of data and complexity levels for analysis. The experimental results show that D_ELT outperforms the other two approaches, ETL and ELT. In addition, the larger the amount of data or the higher the complexity of the analysis, the greater the parallelization effect of transform in D_ELT, leading to better performance over the traditional ETL and ELT approaches.

show abstract

“…Pitfall of this effort, needed for data cleaning during extraction and integration, is increasing response times but is necessary to achieve query-optimization and data quality (Misra, Saha et al 2013). Thus, data cleaning is an on-going process that requires awareness of underlying fundamental principles that are subjected to performance improvement (Chaturvedi, Faruquie et al 2015).…”

Section: Introductionmentioning

confidence: 99%

Data Cleaning In Data Warehouse: A Survey of Data Pre-processing Techniques and Tools

Fatima¹,

Nazir²,

Khan³

2017

IJITCS

View full text Add to dashboard Cite

Abstract-A Data Warehouse is a computer system designed for storing and analyzing an organization's historical data from day-to-day operations in Online Transaction Processing System (OLTP). Usually, an organization summarizes and copies information from its operational systems to the data warehouse on a regular schedule and management performs complex queries and analysis on the information without slowing down the operational systems. Data need to be pre-processed to improve quality of data, before storing into data warehouse. This survey paper presents data cleaning problems and the approaches in use currently for preprocessing. To determine which technique of preprocessing is best in what scenario to improve the performance of Data Warehouse is main goal of this paper. Many techniques have been analyzed for data cleansing, using certain evaluation attributes and tested on different kind of data sets. Data quality tools such as YALE, ALTERYX, and WEKA have been used for conclusive results to ready the data in data warehouse and ensure that only cleaned data populates the warehouse, thus enhancing usability of the warehouse. Results of paper can be useful in many future activities like cleansing, standardizing, correction, matching and transformation. This research can help in data auditing and pattern detection in the data.

show abstract

Performance Comparison of Hadoop Based Tools with Commercial ETL Tools – A Case Study

Cited by 10 publications

References 1 publication

Extracting-Transforming-Loading Modeling Approach for Big Data Analytics

Extracting-Transforming-Loading Modeling Approach for Big Data Analytics

MapReduce-Based D_ELT Framework to Address the Challenges of Geospatial Big Data

Data Cleaning In Data Warehouse: A Survey of Data Pre-processing Techniques and Tools

Contact Info

Product

Resources

About