Big Data Architectures: A Detailed and Application Oriented Analysis

Abstract

Big Data refers to huge amounts of heterogeneous data from both traditional and new sources, growing at a higher rate than ever. Due to their high heterogeneity, it is a challenge to build systems to centrally process and analyze efficiently such data which are internal and external to organizations. A Big data architecture describes the blueprint of a system handling massive volume of data during its storage, processing, analysis and visualization. Several architectures belonging to different categories have been proposed by academia and industry but the field is still lacking benchmarks. Therefore, a detailed analysis of the characteristics of the existing architectures is required in order to ease the choice between architectures for specific use cases or industry requirements. The types of data sources, the hardware requirements, the maximum tolerable latency, the fitment to industry, the amount of data to be handled are some of the factors that need to be considered carefully before making the choice of an architecture of a Big Data system. However, the wrong choice of architecture can result in huge decline for a company reputation and business. This paper reviews the most prominent existing Big Data architectures, their advantages and shortcomings, their hardware requirements, their open source and proprietary software requirements and some of their realworld use cases catering to each industry. The purpose of this body of work is to equip Big Data architects with the necessary resources to make better informed choices to design optimal Big Data systems.

Architecting Data-Driven Applications: From Architectural Design to the Data Quality

“…Additionally, They explained potential application scenarios for IoT dataflow management, such as smart cities, transportation, and manufacturing. Kalipe G. [62] examined the major existing architectures for Big Data (Lambda, Kappa, Zeta, ...). And explored the benefits and drawbacks of each architecture, as well as their hardware requirements, open source and proprietary software needs, and examples of real-world applications in various industries.…”

Big Data Architectures and Concepts

“…5, No. 2, December 2023 DOI: doi.org/10.35970/jinita.v5i2.1876 Data in business, it is necessary to push back the limits of the systems, particularly in terms of the volume of data to be analyzed, the processing speed, and the variety of data to be managed [2]. The implementation of a big data architecture within an enterprise allows for batch processing of data sources in real-time, giving the possibility of exploiting voluminous data, while transforming unstructured data into structured data to facilitate its exploitation, and also to centralize existing data and those from different sources in different formats to promote predictive analysis and which allows for tasks based on machine learning and artificial intelligence technologies [1].…”

“…This architecture is the most commonly used for processing and managing large data in real-time and batch mode simultaneously. It allows functional separation of storage, consumption, and complex real-time processing with the ability to store and process large volumes of data (batch) while integrating the most recent data into the results [2], [6]. However, it is the most widely used because of the parameters it offers for successful processing, such as resistance to failures, fast response time during processing, scaling, and the possibility of merging block data processing (batch) and new data input (realtime).…”

“…Any new data stream that arrives at the Batch layer is calculated and processed using MapReduce or machine learning. The result of this processing is stored as a batch view [1], [2].…”

“…It is based on the streaming architecture in which a series of incoming data is first stored in a messaging engine such as Apache Kafka. From there, a streaming engine reads and transforms the data into an analyzable format and then stores it in an analytical database that end users can query [1], [2]. Kappa is a simplified, dedicated data processing architecture used in streaming layer deployment models where data sources are both batch and real-time and where end-to-end latency requirements are very strict.…”