Viper: A module for communication-layer determinism and scaling in low-latency stream processing

Walulya, Ivan; Palyvos-Giannas, Dimitris; Nikolakopoulos, Yiannis; Gulisano, Vincenzo; Papatriantafilou, Marina; Tsigas, Philippas

doi:10.1016/j.future.2018.05.067

Cited by 17 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Interesting future work studies include the possibilities given by the inter-thread deployment function of Haren to elastically adjust threads of an SPE and boost Haren's adaptivity by means of autonomous adjustments of its configuration parameters (e.g., the scheduling period P). Other interesting directions include a more in-depth exploration of Haren's behavior for complex queries that involve parallel branches [26] as well as for runtime changes of the queries or the policies used to schedule their operators.…”

Section: Discussionmentioning

confidence: 99%

Haren

Palyvos-Giannas

Gulisano

Papatriantafilou

2019

Proceedings of the 13th ACM International Conference on Distributed and Event-Based Systems

Self Cite

View full text Add to dashboard Cite

In modern Stream Processing Engines (SPEs), numerous diverse applications, which can differ in aspects such as cost, criticality or latency sensitivity, can co-exist in the same computing node. When these differences need to be considered to control the performance of each application, custom scheduling of operators to threads is of key importance (e.g., when a smart vehicle needs to ensure that safety-critical applications always have access to computational power, while other applications are given lower, variable priorities). Many solutions have been proposed regarding schedulers that allocate threads to operators to optimize specific metrics (e.g., latency) but there is still lack of a tool that allows arbitrarily complex scheduling strategies to be seamlessly plugged on top of an SPE. We propose Haren to fill this gap. More specifically, we (1) formalize the thread scheduling problem in stream processing in a general way, allowing to define ad-hoc scheduling policies, (2) identify the bottlenecks and the opportunities of scheduling in stream processing, (3) distill a compact interface to connect Haren with SPEs, enabling rapid testing of various scheduling policies, (4) illustrate the usability of the framework by integrating it into an actual SPE and (5) provide a thorough evaluation. As we show, Haren makes it is possible to adapt the use of computational resources over time to meet the goals of a variety of scheduling policies. CCS CONCEPTS • Information systems → Online analytical processing engines; • Software and its engineering → Scheduling; Middleware.

show abstract

Section: Discussionmentioning

confidence: 99%

Haren

Palyvos-Giannas

Gulisano

Papatriantafilou

2019

Proceedings of the 13th ACM International Conference on Distributed and Event-Based Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Stream Processing in a nutshell. Stream processing is leveraged in large distributed systems [1,2,7,9,10,13,22,23] to process unbounded streams of tuples. Stream processing applications are defined as Directed Acyclic Graphs of operators that transform the tuples delivered by a set of data sources and produce new streams of tuples that are eventually delivered to end-users.…”

Section: Talk Overviewmentioning

confidence: 99%

Motivations and Challenges for Stream Processing in Edge Computing

Gulisano

2021

Companion of the ACM/SPEC International Conference on Performance Engineering

Self Cite

View full text Add to dashboard Cite

The 2030 Agenda for Sustainable Development of the United Nations General Assembly defines 17 development goals to be met for a sustainable future. Goals such as Industry, Innovation and Infrastructure and Sustainable Cities and Communities depend on digital systems. As a matter of fact, billions of Euros are invested into digital transformation within the European Union, and many researchers are actively working to push state-of-the-art boundaries for techniques/tools able to extract value and insights from the large amounts of raw data sensed in digital systems.Edge computing aims at supporting such data-to-value transformation. In digital systems that traditionally rely on central data gathering, edge computing proposes to push the analysis towards the devices and data sources, thus leveraging the large cumulative computational power found in modern distributed systems. Some of the ideas promoted in edge computing are not new, though. Continuous and distributed data analysis paradigms such as stream processing have argued about the need for smart distributed analysis for basically 20 years. Starting from this observation, this talk covers a set of standing challenges for smart, distributed, and continuous stream processing in edge computing, with real-world examples and use-cases from smart grids and vehicular networks.

show abstract

“…For monitoring applications in CPSs, determinism is crucial to achieve correct and predictable/repeatable analysis, especially when the latter is used to generate sensitive or safety-related alerts [32]. A way to achieve such behavior is to compose queries with base operators that, if fed the same sequence of input tuples, produce the same sequence of output tuples [21,46]. That is, operators that have no randomness in their analysis and whose analysis does not depend on execution-specific aspects [21,4].…”

Section: Correctness Guarantees Enabled By the Data Streaming Processmentioning

confidence: 99%

Intrusion Detection in Industrial Networks via Data Streaming

et al. 2020

Self Cite

View full text Add to dashboard Cite

Given the increasing threat surface of industrial networks due to distributed, Internet-of-Things (IoT) based system architectures, detecting intrusions in Industrial IoT (IIoT) systems is all the more important, due to the safety implications of potential threats. The continuously generated data in such systems form both a challenge but also a possibility: data volumes/rates are high and require processing and communication capacity but they contain information useful for system operation and for detection of unwanted situations. In this chapter we explain that stream processing (a.k.a. data streaming) is an emerging useful approach both for general applications and for intrusion detection in particular, especially since it can enable data analysis to be carried out in the continuum of edge-fog-cloud distributed architectures of industrial networks, thus reducing communication latency and gradually filtering and aggregating data volumes. We argue that usefulness stems also due to facilitating provisioning of agile responses, i.e. due to potentially smaller latency for intrusion detection and hence also improved possibilities for intrusion mitigation. In the chapter we outline architectural features of IIoT networks, potential threats and examples of state-of-the art intrusion detection methodologies. Moreover, we give an overview of how leveraging distributed and parallel execution of streaming applications in industrial setups can influence the possibilities of protecting these systems. In these contexts, we give examples using electricity networks (a.k.a. Smart Grid systems). We conclude that future industrial networks, especially their Intrusion Detection Systems (IDSs), should take advantage of data streaming concept by decoupling semantics from the deployment.

show abstract

Viper: A module for communication-layer determinism and scaling in low-latency stream processing

Cited by 17 publications

References 9 publications

Haren

Haren

Motivations and Challenges for Stream Processing in Edge Computing

Intrusion Detection in Industrial Networks via Data Streaming

Contact Info

Product

Resources

About