Towards a unified storage and ingestion architecture for stream processing

Abstract

Abstract-Big Data applications are rapidly moving from a batch-oriented execution model to a streaming execution model in order to extract value from the data in real-time. However, processing live data alone is often not enough: in many cases, such applications need to combine the live data with previously archived data to increase the quality of the extracted insights. Current streaming-oriented runtimes and middlewares are not flexible enough to deal with this trend, as they address ingestion (collection and pre-processing of data streams) and persistent storage (archival of intermediate results) using separate services. This separation often leads to I/O redundancy (e.g., write data twice to disk or transfer data twice over the network) and interference (e.g., I/O bottlenecks when collecting data streams and writing archival data simultaneously). In this position paper, we argue for a unified ingestion and storage architecture for streaming data that addresses the aforementioned challenge. We identify a set of constraints and benefits for such a unified model, while highlighting the important architectural aspects required to implement it in real life. Based on these aspects, we briefly sketch our plan for future work that develops the position defended in this paper.

Real Time Big Data Analysis by Using Apache Kudu and NoSQL Redis in Web Applications

“…With the wide spread of big data technology this highspeed data processing become more popular, but there are limitations on bandwidth and latency for real time or near real time processing. That is why there are some gaps between an analytics system and a real end-to-end system (Wang et al, 2017;Marcu et al, 2017;Petrov&Valov, 2019).…”

A Scalable and Robust Framework for Data Stream Ingestion

“…Failures can cause the loss of large amounts of data streams which may lead to erroneous analytic results. Ingestion systems should incorporate high-availability mechanisms that allow to operate continuously for a long time in spite of failures [5]. It is, therefore, desirable for ingestion systems to offer the desired degree of robustness in handling failures while minimizing data loss [7].…”

“…A typical streaming analytics system is built on top of a three layers stack that include ingestion, processing, and storage components [5]. The ingestion layer is the entry point to the streaming architecture.…”

“…The challenge of data explosion has generated a lot of interest in low-latency in-memory frameworks and data stream processing systems in recent years [4]. In order to keep up with the high rate and volume of data, modern data processing systems must deliver insights with minimal latency and high throughput [5]. Streaming data analytics is a new programming paradigm designed to incorporate continuous data into a decision-making process [2].…”

KerA: Scalable Data Ingestion for Stream Processing

“…None of the state-of-art ingestion systems are designed to leverage data locality optimizations as envisioned with KerA in a unified storage and ingestion architecture [19]. Moreover, thanks to its network agnostic implementation [7], KerA can benefit from emerging fast networks and RDMA, providing more efficient reads and writes than using TCP/IP.…”