Distributed real-time ETL architecture for unstructured big data

Mehmood, Erum; Anees, Tayyaba

doi:10.1007/s10115-022-01757-7

Cited by 12 publications

(4 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This system could suggest the regions to focus on for drivers depending on competitor presence and historical pickups. Mehmood and Anees [3] focused on designing distributed real-time ETL architecture for unstructured big data. They proposed an architecture using Apache Kafka, MongoDB, and Apache Spark.…”

Section: Literature Reviewmentioning

confidence: 99%

Towards a Trustworthy and Efficient ETL Pipeline for ATM Transaction Data

Muhammad Ahmad Ashfaq,

Nimra Haq,

Usman Arshad

et al. 2024

SJCMS

View full text Add to dashboard Cite

ATMs generate vast amounts of data daily, which needs to be analyzed and stored. Dealing with this data, also termed big data, is a complex task, and here comes the role of ETL pipelines. ETL pipelines need extensive resources for operations, and their performance optimization is necessary as data must be dealt with in near or even real-time. If the pipeline deals with financial data such as ATM transactions, steps should be taken to ensure the data's security, privacy, confidentiality, and integrity. This can be achieved using Blockchain technology. It is a distributed ledger technology having an immutable nature. It has significant advantages in terms of providing security, but it has disadvantages as well, such as low throughput and transactional latency. If blockchain is used in an ETL pipeline, it will affect the overall performance. So, to prevent the decline in performance, steps should be taken to optimize it. In this paper, we are using parallelization and partitioning as techniques to optimize performance. The primary goal here is to achieve maximum security while maintaining performance.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Towards a Trustworthy and Efficient ETL Pipeline for ATM Transaction Data

Muhammad Ahmad Ashfaq,

Nimra Haq,

Usman Arshad

et al. 2024

SJCMS

View full text Add to dashboard Cite

show abstract

“…There is no cost model provided by the authors' solutions to quantify the performance gain. Mehmood and Anees in [33], propose a real time ETL architecture for unstructured big data during the transformation phase, focusing on accelerating stream disk joins and mitigating disk overhead and data loss. Finally in order to improve the performance of the ETL process in the era of big data and by pre-processing the data beforehand, Monika Patel and Dhiren B.…”

Section: Related Workmentioning

confidence: 99%

Towards a Scalable and Efficient ETL

Gueddoudj,

Chikh

2023

IJCDS

View full text Add to dashboard Cite

Extract, transform, and load (ETL) processes are crucial for building repositories of data from a variety of self-contained sources. Despite their complexity and cost, ETL processes have demonstrated some maturity for traditional, XML, and graph data sources. However the main challenge for ETL processes is double: (1) they do not scale when brought down to managing large and highly varied data sources, involving web-data. (2) the deployment of the target data warehouse in a polystore. The paper reviews various research efforts along this line of research. The paper then proposes a conceptual modeling of these processes using BPMN (Business Process Modeling Notation). These processes are automatically converted to scripts to be implemented within Spark framework. The solution is packaged according a new distributed architecture (Open ETL) that supports both batch and stream processing. To make our new approach more concrete and evaluable, a real case study using the LUBM benchmark, which involves heterogeneous data sources is considered.

show abstract

“…Although ETL is required for data warehousing and analytics, not all ETL software packages are created equal. The best ETL tool may vary depending on the circumstances and use cases [4]. This document compares the most common ETL 2022 tools [5].…”

Section: Introductionmentioning

confidence: 99%

Comparative Analysis of ETL Tools in Big Data Analytics

Qaiser

Farooq

Mustafa

et al. 2023

PakJET

View full text Add to dashboard Cite

Massive amounts of data are being generated these days all around the world. The majority of social networking platforms, ecommerce sites, the public and private sectors, healthcare institutions, cloud networks, and various servers are all generating immense amount of data. Collected data from various sites could be in a structured or unstructured format. Extract Transform Load (ETL) is a crucial component of the growing demand for quicker business decisions aimed at many contemporary applications. Due to the volume and speed of data, real-time ETL is built on the foundation of multi-source, unstructured data stream extraction and transformation employing disc data in dispersed environments. The entire procedure is pipelined while processing so that the final resultant data can provide some essential and useful conclusions to work on. Some analytical findings are once again helpful in making decisions. However, the produced results may differ in some numbers, graphs, and figures in many circumstances. This occurs as a result of the usage of some unrealistic tools for big data processing. This paper proposed several methods and appropriate ETL (extract, transform, and load) tools for the big data processing, which may result in appropriate analytics and conclusions from the data.

show abstract

Distributed real-time ETL architecture for unstructured big data

Cited by 12 publications

References 46 publications

Towards a Trustworthy and Efficient ETL Pipeline for ATM Transaction Data

Towards a Trustworthy and Efficient ETL Pipeline for ATM Transaction Data

Towards a Scalable and Efficient ETL

Comparative Analysis of ETL Tools in Big Data Analytics

Contact Info

Product

Resources

About