Serverless Data Analytics in the IBM Cloud

Sampé, Josep; Vernik, Gil; Sánchez-Artigas, Marc; Garcı́a-López, Pedro

doi:10.1145/3284028.3284029

Cited by 65 publications

(45 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To demonstrate this point, consider an application built with the Pywren [17,29] runtime, shown in Figure 1. In this experiment, 100 containers start simultaneously, each performing a basic computation before sending an ack to a leader node when completed.…”

Section: Introductionmentioning

confidence: 99%

Particle

Thomas

Voelker

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

Burst-parallel serverless applications invoke thousands of short-lived distributed functions to complete complex jobs such as data analytics, video encoding, or compilation. While these tasks execute in seconds, starting and configuring the virtual network they rely on is a major bottleneck that can consume up to 84% of total startup time. In this paper we characterize the magnitude of this network cold start problem in three popular overlay networks, Docker Swarm, Weave, and Linux Overlay. We focus on end-to-end startup time that encompasses both the time to boot a group of containers as well as interconnecting them. Our primary observation is that existing overlay approaches for serverless networking scale poorly in short-lived serverless environments. Based on our findings we develop Particle, a network stack tailored for multi-node serverless overlay networks that optimizes network creation without sacrificing multi-tenancy, generality, or throughput. When integrated into a serverless burst-parallel video processing pipeline, Particle improves application runtime by 2.4-3× over existing overlays. CCS Concepts • Computer systems organization → Cloud computing.

show abstract

Section: Introductionmentioning

confidence: 99%

Particle

Thomas

Voelker

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

show abstract

“…Serverless platforms thus provide a scalable computation substrate where the amount of computational resources available to a task can be rapidly altered. Prior work has exploited this flexibility for video processing [11,27], numerical computation [15,25,33,34,40,81,96], data analytics [43,44,70,76], machine learning [18,39,83], and parallel compilation [26].…”

Section: Background and Motivationmentioning

confidence: 99%

“…Due to these benefits, serverless computing is being offered by all major cloud providers [7,32,38,55] and rapidly adopted by tenants [23], mainly for event-driven workloads (e.g., file transcoders). Recent work, however, has explored more general applications on serverless including video processing [11,27], numerical computing [40,81], and analytics [44,70,76]. These examples indicate significant interest in exploiting serverless computing beyond event handling, presumably because of its elasticity and ease of deployment.…”

Section: Introductionmentioning

confidence: 99%

Kappa

Zhang

Fang

Panda³

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

Serverless computing has recently emerged as a new paradigm for running software on the cloud. In this paradigm, programs need to be expressed as a set of short-lived tasks, each of which can complete within a short bounded time (e.g., 15 minutes on AWS Lambda). Serverless computing is beneficial to cloud providers-by allowing them to better utilize resources-and to users-by simplifying management and enabling greater elasticity. However, developing applications to run in this environment is challenging, requiring users to appropriately partition their code, develop new coordination mechanisms, and deal with failure recovery. In this paper, we propose Kappa, a framework that simplifies serverless development. It uses checkpointing to handle lambda function timeouts, and provides concurrency mechanisms that enable parallel computation and coordination. CCS Concepts • Computer systems organization → Cloud computing.

show abstract

“…These functions often read or write private data stored in the cloud infrastructure in order to achieve their operational goal (e.g. serverless data analytics applications [86]). This necessitates a robust access control mechanism in the serverless platforms to determine if a function invocation request is properly authenticated and has the required permissions to access a piece of data.…”

Section: Introductionmentioning

confidence: 99%