Prompt: Dynamic Data-Partitioning for Distributed Micro-batch Stream Processing Systems

Abdelhamid, Ahmed S.; Mahmood, Ahmed; Daghistani, Anas; Aref, Walid G.

doi:10.1145/3318464.3389713

Cited by 14 publications

(15 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several research works investigate load distribution and routing strategies (e.g., [3,23,31,50,92,95,107,143]). For instance, Rivetti et al [143] present a solution to balance load among parallel instances of a stateless operator, accounting for variable tuple processing times.…”

Section: What: Adaptation Actions and Controlled Entitiesmentioning

confidence: 99%

“…Several solutions, especially those acting on data streams (e.g., load distribution, shedding), perform adaptation with iner granularity, at level of single tuples (e.g., [3,23,50,95,168,185]) or batches of tuples (e.g., [38,177]). Solutions acting at the infrastructure level usually work with the granularity of the computing node (e.g., [47,82,176]) or the network link [5].…”

Section: What: Adaptation Actions and Controlled Entitiesmentioning

confidence: 99%

“…A related metric is load imbalance, which measures the load diference among parallel instances of an operator. This metric is especially used by load distribution solutions (e.g., [3,50,92,191]), which often aim at minimizing this metric, as better load balancing leads to better performance.…”

Section: Metricsmentioning

confidence: 99%

“…Other works (e.g., [2,3,35,37,90,199]) consider event-triggered adaptation actions. In order to implement this kind of scheme, one or more types of event must be associated with the execution of an adaptation policy.…”

Section: When: Triggers and Time Horizonmentioning

confidence: 99%

“…It is clear that proactive solutions are harder to realize, as they require predictions about future working conditions, and their eicacy depends on the accuracy of such predictions. It is therefore no surprising that the vast majority of the existing approaches rely on reactive adaptation schemes (e.g., [3,30,57,60,118,124,154,155,165,195]). Examples of reactive policies are given by the aforementioned threshold-based heuristics, where actions are triggered by threshold violations, which are usually evaluated against latest available monitoring information (e.g., average resource utilization in the last minute).…”

Section: Proactivitymentioning

confidence: 99%

See 4 more Smart Citations

Runtime Adaptation of Data Stream Processing Systems: The State of the Art

et al. 2022

View full text Add to dashboard Cite

Data Stream Processing (DSP) has emerged over the years as the reference paradigm for the analysis of continuous and fast information flows, which have often to be processed with low-latency requirements to extract insights and knowledge from raw data. Dealing with unbounded data flows, DSP applications are typically long-running and, thus, likely experience varying workloads and working conditions over time. To keep a consistent service level in face of such variability, a lot of effort has been spent studying strategies for run-time adaptation of DSP systems and applications. In this survey, we review the most relevant approaches from the literature, presenting a taxonomy to characterize the state of the art along several key dimensions. Our analysis allows us to identify current research trends as well as open challenges that will motivate further investigations in this field.

show abstract

Section: What: Adaptation Actions and Controlled Entitiesmentioning

confidence: 99%

Section: What: Adaptation Actions and Controlled Entitiesmentioning

confidence: 99%

Section: Metricsmentioning

confidence: 99%

Section: When: Triggers and Time Horizonmentioning

confidence: 99%

Section: Proactivitymentioning

confidence: 99%

See 3 more Smart Citations

Runtime Adaptation of Data Stream Processing Systems: The State of the Art

et al. 2022

View full text Add to dashboard Cite

show abstract

Self‐adaptation on parallel stream processing: A systematic review

Vogel

Griebler

Danelutto

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

A recurrent challenge in real-world applications is autonomous management of the executions at run-time. In this vein, stream processing is a class of applications that compute data flowing in the form of streams (e.g., video feeds, images, and data analytics), where parallel computing can help accelerate the executions. On the one hand, stream processing applications are becoming more complex, dynamic, and long-running. On the other hand, it is unfeasible for humans to monitor and manually change the executions continuously. Hence, self-adaptation can reduce costs and human efforts by providing a higher-level abstraction with an autonomic/seamless management of executions. In this work, we aim at providing a literature review regarding self-adaptation applied to the parallel stream processing domain. We present a comprehensive revision using a systematic literature review method. Moreover, we propose a taxonomy to categorize and classify the existing self-adaptive approaches.Finally, applying the taxonomy made it possible to characterize the state-of-the-art, identify trends, and discuss open research challenges and future opportunities.

show abstract

Adaptivity in continuous massively parallel distance-based outlier detection

Toliopoulos

Gounaris

2022

Computing

View full text Add to dashboard Cite

Prompt: Dynamic Data-Partitioning for Distributed Micro-batch Stream Processing Systems

Cited by 14 publications

References 34 publications

Runtime Adaptation of Data Stream Processing Systems: The State of the Art

Runtime Adaptation of Data Stream Processing Systems: The State of the Art

Self‐adaptation on parallel stream processing: A systematic review

Adaptivity in continuous massively parallel distance-based outlier detection

Contact Info

Product

Resources

About