SamzaSQL: Scalable Fast Data Management with Streaming SQL

Pathirage, Milinda; Hyde, Julian; Pan, Yi; Plale, Beth

doi:10.1109/ipdpsw.2016.141

Cited by 14 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, there are more SQL extensions for stream processing scenarios developed for certain systems, e.g., streaming SQL for Apache Kafka called KSQL [57], Continuous Computation Language (CCL) that is the extended SQL used in SAP HANA Smart Data Streaming [58], or SamzaSQL [59] as extended SQL for the DSPS Samza [60].…”

Section: Related Workmentioning

confidence: 99%

DataEther: Data Exploration Framework For Ethereum

Chen

et al. 2019

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

With the demand to process ever-growing data volumes, a variety of new data stream processing frameworks have been developed. Moving an implementation from one such system to another, e.g., for performance reasons, requires adapting existing applications to new interfaces. Apache Beam addresses these high substitution costs by providing an abstraction layer that enables executing programs on any of the supported streaming frameworks. In this paper, we present a novel benchmark architecture for comparing the performance impact of using Apache Beam on three streaming frameworks: Apache Spark Streaming, Apache Flink, and Apache Apex. We find significant performance penalties when using Apache Beam for application development in the surveyed systems. Overall, usage of Apache Beam for the examined streaming applications caused a high variance of query execution times with a slowdown of up to a factor of 58 compared to queries developed without the abstraction layer. All developed benchmark artifacts are publicly available to ensure reproducible results.

show abstract

Section: Related Workmentioning

confidence: 99%

DataEther: Data Exploration Framework For Ethereum

Chen

et al. 2019

2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

show abstract

“…These requirements include the ability to (i) process continuous data on-the-fly without any requirement to store them, (ii) support high-level languages such as SQL, (iii) handle imperfections such as delayed, missing and out-of-order data, (iv) guarantee predictable and repeatable outcomes, (v) efficiently store, access, modify, and combine (with live streaming data) state information, (vi) ensure that the integrity of the data is maintained at all times and relevant applications are up and available despite failures, (vii) automatically and transparently distribute the data processing load across multiple processors and machines, and (viii) respond to high-volume data processing applications in real-time using a highly optimized execution path. Cutting edge DSMEs include Samza-SQL [32], KSQL, and SQLstream Blaze [30], [33].…”

Section: ) Data Stream Processing Layermentioning

confidence: 99%

A Survey of Distributed Data Stream Processing Frameworks

et al. 2019

View full text Add to dashboard Cite

Big data processing systems are evolving to be more stream oriented where each data record is processed as it arrives by distributed and low-latency computational frameworks on a continuous basis. As the stream processing technology matures and more organizations invest in digital transformations, new applications of stream analytics will be identified and implemented across a wide spectrum of industries. One of the challenges in developing a streaming analytics infrastructure is the difficulty in selecting the right stream processing framework for the different use cases. With a view to addressing this issue, in this paper we present a taxonomy, a comparative study of distributed data stream processing and analytics frameworks, and a critical review of representative open source (Storm, Spark Streaming, Flink, Kafka Streams) and commercial (IBM Streams) distributed data stream processing frameworks. The study also reports our ongoing study on a multilevel streaming analytics architecture that can serve as a guide for organizations and individuals planning to implement a real-time data stream processing and analytics framework. INDEX TERMS Dataflow architectures, data stream architectures, distributed processing systems comparison, survey, taxonomy.

show abstract

“…The core of Samza consists of several fairly low-level abstractions, on top of which highlevel operators have been built (Pathirage et al 2016). However, the core abstractions have been carefully designed for operational robustness, and the scalability of Samza is directly attributable to the choice of these foundational abstractions.…”

Section: Overviewmentioning

confidence: 99%

Apache Samza

Kleppmann¹

2018

Encyclopedia of Big Data Technologies

View full text Add to dashboard Cite

Apache Samza is an open source framework for distributed processing of high-volume event streams. Its primary design goal is to support high throughput for a wide range of processing patterns, while providing operational robustness at the massive scale required by Internet companies. Samza achieves this goal through a small number of carefully designed abstractions: partitioned logs for messaging, fault-tolerant local state, and cluster-based task scheduling.

show abstract

SamzaSQL: Scalable Fast Data Management with Streaming SQL

Cited by 14 publications

References 17 publications

DataEther: Data Exploration Framework For Ethereum

DataEther: Data Exploration Framework For Ethereum

A Survey of Distributed Data Stream Processing Frameworks

Apache Samza

Contact Info

Product

Resources

About