Adaptive Provisioning of Stream Processing Systems in the Cloud

Cerviño, Javier; Kalyvianaki, Evangelia; Salvachúa, Joaquín; Pietzuch, Peter

doi:10.1109/icdew.2012.40

Cited by 35 publications

(22 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even though our solution can be run at longer periods (e.g., once per day), it is desirable to adapt in a dynamic and online fashion. In some works such as [15] dynamic approaches are suggested for adaptive provisioning. However, in order to solve our problem there is a need to take into account additional factors such as the effects of dynamic workload on the user satisfaction metric.…”

Section: F Summary and Discussionmentioning

confidence: 99%

“…One relevant area of research is stream processing in the cloud [17], [18]. However, there are only a few works in this category that specifically consider resource provisioning [15], [19]. In [15], the authors propose adaptive resource provisioning for processing stream queries with the goal of optimizing query latency.…”

Section: Related Workmentioning

confidence: 99%

“…However, there are only a few works in this category that specifically consider resource provisioning [15], [19]. In [15], the authors propose adaptive resource provisioning for processing stream queries with the goal of optimizing query latency. On the other hand, this work does not aim at minimizing monetary costs.…”

Section: Related Workmentioning

confidence: 99%

“…In contrast to our work, the idea is specific to the domain of streaming queries. Finally, in both [19] and [15], there is no concept of subscriber satisfaction metric, which is essential in our problem.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Cost-Effective Resource Allocation for Deploying Pub/Sub on Cloud

Setty

Vitenberg

Kreitz³

et al. 2014

2014 IEEE 34th International Conference on Distributed Computing Systems

View full text Add to dashboard Cite

Publish/subscribe (pub/sub) is a popular communication paradigm in the design of largescale distributed systems. A fundamental challenge in deploying pub/sub systems on a data center or a cloud infrastructure is efficient and cost-effective resource allocation that would allow delivery of notifications to all subscribers. In this paper, we provide answers to the following three fundamental questions: Given a pub/sub workload, (1) what is the minimum amount of resources needed to satisfy all the subscribers, (2) what is a cost-effective way to allocate resources for the given workload, and (3) what is the cost of hosting it on a public Infrastructure-as-a-Service (IaaS) provider like Amazon EC2.To answer these questions, we formulate a problem coined Minimum Cost Subscriber Satisfaction (MCSS). We prove MCSS to be NP-hard and provide an efficient heuristic solution based on a combination of optimizations. We evaluate the solution experimentally using real traces from Spotify and Twitter along with a pricing model from Amazon. We show the impact of each optimization using a naive solution as the baseline. Using a variety of practical scenarios for each dataset, we also show that our solution scales well for millions of subscribers and runs fast.

show abstract

Section: F Summary and Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Cost-Effective Resource Allocation for Deploying Pub/Sub on Cloud

Setty

Vitenberg

Kreitz³

et al. 2014

2014 IEEE 34th International Conference on Distributed Computing Systems

View full text Add to dashboard Cite

show abstract

“…It is a common observation that data stream processors face resource limitations during their operation due to unexpected loads [3,9]. There are multiple possible solutions for these issues.…”

Section: Introductionmentioning

confidence: 99%

Latency-Aware Secure Elastic Stream Processing with Homomorphic Encryption

Rodrigo

Dayarathna²,

Jayasena

2019

Data Sci. Eng.

View full text Add to dashboard Cite

Increasingly organizations are elastically scaling their stream processing applications into the infrastructure as a service clouds. However, state-of-the-art approaches for elastic stream processing do not consider the potential threats of exposing their data to third parties in cloud environments. We present the design and implementation of an Elastic Switching Mechanism for data stream processing which is based on homomorphic encryption (HomoESM). The HomoESM not only elastically scales data stream processing applications into public clouds but also preserves the privacy of such applications. Using a real-world test setup, which includes an E-mail Filter benchmark and a Web server access log processor benchmark (EDGAR), we demonstrate the effectiveness of our approach. Experiments on Amazon EC2 indicate that the proposed approach for homomorphic encryption provides a significant result which is 10-17% improvement in average latency in the case of E-mail Filter benchmark and EDGAR benchmark, respectively. Furthermore, EDGAR add/subtract operations, multiplication, and comparison operations showed up to 6.13%, 7.81%, and 26.17% average latency improvements, respectively. Finally, we evaluate the potential of scaling the homomorphic stream processor in the public cloud. These results indicate the potential for real-world deployments of secure elastic data stream processing applications.

show abstract

Resource aware scheduler for distributed stream processing in cloud native environments

Sarathchandra

Karandana

Heenatigala

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

Recently distributed stream processors are increasingly being deployed in cloud computing infrastructures. In this article, we study performance characteristics of distributed stream processing applications in Google Compute Engine which is based on Kubernetes. We identify performance gaps in terms of throughput which appear in such environments when using a round robin (RR) scheduling algorithm. As a solution, we propose resource aware stream processing scheduler called resource aware scheduler for stream processing applications in cloud native environments (RaspaCN).We implement RaspaCN's job scheduler using two-step process. First, we use machine learning to identify the optimal number of worker nodes. Second, we use RR and multiple Knapsack algorithms to produce performance optimal stream processing job schedules. With three application benchmarks called HTTP Log Processor, Nexmark, and Email Processor representing real world stream processing scenarios we evaluate the performance benefits obtained via RaspaCN's scheduling algorithm. RaspaCN could produce percentage increase of average throughput values by at least 37%, 38%, and 10%, respectively, for HTTP Log Processor, Nexmark, and Email Processor benchmarks for fixed input data rates. Furthermore, we conduct experiments with varying input data rates as well and show 7% improved average throughput for HTTP Log Processor. These experiments show the effectiveness of our proposed stream processor job scheduler for producing improved performance.

show abstract

Adaptive Provisioning of Stream Processing Systems in the Cloud

Cited by 35 publications

References 13 publications

Cost-Effective Resource Allocation for Deploying Pub/Sub on Cloud

Cost-Effective Resource Allocation for Deploying Pub/Sub on Cloud

Latency-Aware Secure Elastic Stream Processing with Homomorphic Encryption

Resource aware scheduler for distributed stream processing in cloud native environments

Contact Info

Product

Resources

About