KerA: Scalable Data Ingestion for Stream Processing

Marcu, Ovidiu-Cristian; Costan, Alexandru; Antoniu, Gabriel; Pérez-Hernández, María S.; Nicolae, Bogdan; Tudoran, Radu; Bortoli, Stefano

doi:10.1109/icdcs.2018.00152

Cited by 12 publications

(10 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the next section, after presenting a background on KerA's dynamic partitioning mechanism (KerA as described in [7] does not implement replication), we describe the implementation of the virtual log replication in KerA.…”

Section: The Virtual Log Design Principlementioning

confidence: 99%

Virtual Log-Structured Storage for High-Performance Streaming

Marcu

Costan

Nicolae

et al. 2021

2021 IEEE International Conference on Cluster Computing (CLUSTER)

Self Cite

View full text Add to dashboard Cite

Over the past decade, given the higher number of data sources (e.g., Cloud applications, Internet of things) and critical business demands, Big Data transitioned from batchoriented to real-time analytics. Stream storage systems, such as Apache Kafka, are well known for their increasing role in real-time Big Data analytics. For scalable stream data ingestion and processing, they logically split a data stream topic into multiple partitions. Stream storage systems keep multiple data stream copies to protect against data loss while implementing a stream partition as a replicated log. This architectural choice enables simplified development while trading cluster size with performance and the number of streams optimally managed. This paper introduces a shared virtual log-structured storage approach for improving the cluster throughput when multiple producers and consumers write and consume in parallel data streams. Stream partitions are associated with shared replicated virtual logs transparently to the user, effectively separating the implementation of stream partitioning (and data ordering) from data replication (and durability). We implement the virtual log technique in the KerA stream storage system. When comparing with Apache Kafka, KerA improves the cluster ingestion throughput (for replication factor three) by up to 4x when multiple producers write over hundreds of data streams.

show abstract

Section: The Virtual Log Design Principlementioning

confidence: 99%

Virtual Log-Structured Storage for High-Performance Streaming

Marcu

Costan

Nicolae

et al. 2021

2021 IEEE International Conference on Cluster Computing (CLUSTER)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Important features when considering data stream ingestion tools include the ease of installing, publishing, transporting, consuming, and archiving streams to disk. Data ingestion system should be able to support high throughput, low latency and must scale to a large number of both data stream producers and consumers [12]. We categorized these requirements to include source integration and pre-processing, fault tolerance and message delivery guarantees, provenance and security, backpressure and routing, scalability, and extensibility.…”

Section: Requirements Of a Stream Ingestion Systemmentioning

confidence: 99%

“…It is, therefore, desirable for ingestion systems to offer the desired degree of robustness in handling failures while minimizing data loss [7]. Data ingestion needs to support high throughput, low latency and must scale to a large number of both data producers and consumers [12].…”

Section: B Fault Tolerance and Message Delivery Guaranteesmentioning

confidence: 99%

“…Other similar systems include Scribe [20], a messaging system as Facebook; Siphon [21], a messaging system in Microsoft Azure HDInsight that utilize Kafka. Marcu et al [12] developed KerA, a data ingestion framework that alleviate the limitations of Kafka and other ingestion systems. The study focused on improving throughput, latency and scalability but do not consider issues such as provenance and extensibility.…”

Section: Dataflow Output and Performance Improvementmentioning

confidence: 99%

See 1 more Smart Citation

A Scalable and Robust Framework for Data Stream Ingestion

Isah

Zulkernine

2018

2018 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

An essential part of building a data-driven organization is the ability to handle and process continuous streams of data to discover actionable insights. The explosive growth of interconnected devices and the social Web has led to a large volume of data being generated on a continuous basis. Streaming data sources such as stock quotes, credit card transactions, trending news, traffic conditions, time-sensitive patient's data are not only very common but can rapidly depreciate if not processed quickly. The ever-increasing volume and highly irregular nature of data rates pose new challenges to data stream processing systems. One such challenging but important task is how to accurately ingest and integrate data streams from various sources and locations into an analytics platform. These challenges demand new strategies and systems that can offer the desired degree of scalability and robustness in handling failures. This paper investigates the fundamental requirements and the state of the art of existing data stream ingestion systems, propose a scalable and fault-tolerant data stream ingestion and integration framework that can serve as a reusable component across many feeds of structured and unstructured input data in a given platform, and demonstrate the utility of the framework in a real-world data stream processing case study that integrates Apache NiFi and Kafka for processing high velocity news articles from across the globe. The study also identifies best practices and gaps for future research in developing large-scale data stream processing infrastructure.

show abstract

“…Since they are dynamic, unknown and unlimited, the received information needs to be stored and analyzed without compromising memory and run-time. Handling this type of data requires three fundamentals, which are high-throughput, ultra-low-latency, and low-power [14]- [19].…”

Section: Introductionmentioning

confidence: 99%

Hardware Architecture Proposal for TEDA Algorithm to Data Streaming Anomaly Detection

et al. 2021

View full text Add to dashboard Cite

The amount of data in real-time, such as time series and streaming data, available today continues to grow. Being able to analyze this data the moment it arrives can bring an immense added value. However, it also requires a lot of computational effort and new acceleration techniques. As a possible solution to this problem, this paper proposes a hardware architecture for Typicality and Eccentricity Data Analytic (TEDA) algorithm implemented on Field Programmable Gate Arrays (FPGA) for use in data streaming anomaly detection. TEDA is based on a new approach to outlier detection in the data stream context. The suggested design has a full parallel input of N elements and a 3-stage pipelined architecture to reduce the critical path and thus optimize the throughput. In order to validate the proposals, results of the occupation and throughput of the proposed hardware are presented. The design reached a speed of up to 693x, compared to other software platforms, with a throughput of up to 10.96 MSPs (Mega Sample Per second), using a small portion of the target FPGA resources. Besides, the bit accurate simulation results are also presented. This work is a pioneer in the hardware implementation of the TEDA technique in FPGA. The project aims to Xilinx Virtex-6 xc6vlx240t-1ff1156 as the target FPGA.

show abstract

KerA: Scalable Data Ingestion for Stream Processing

Cited by 12 publications

References 7 publications

Virtual Log-Structured Storage for High-Performance Streaming

Virtual Log-Structured Storage for High-Performance Streaming

A Scalable and Robust Framework for Data Stream Ingestion

Hardware Architecture Proposal for TEDA Algorithm to Data Streaming Anomaly Detection

Contact Info

Product

Resources

About