Kappa

Zhang, Wen; Fang, Vivian; Panda, Aurojit; Shenker, Scott

doi:10.1145/3419111.3421277

Cited by 42 publications

(3 citation statements)

References 42 publications

(69 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For their implementation, authors introduced a new domain-specific language "LAmbdaPACK" [15]. Zhang et al, proposed a new serverless framework "Kappa", that used a check-pointing mechanism to control function execution time-out [33].…”

Section: Related Workmentioning

confidence: 99%

ReactiveFnJ: A choreographed model for Fork-Join Workflow in Serverless Computing

2023

View full text Add to dashboard Cite

Function-as-a-Service (FaaS) is an event-based reactive programming model where functions run in ephemeral stateless containers for short duration. For building complex serverless applications, function composition is crucial to coordinate and synchronize the workflow of an application. Some serverless orchestration systems exist, but they are in their primitive state and do not provide inherent support for non-trivial workflows like, Fork-Join. To address this gap, we propose a fully serverless and scalable design model ReactiveFnJ for Fork-Join workflow. The intent of this work is to illustrate a design which is completely choreographed, reactive, asynchronous, and represents a dynamic composition model for serverless applications based on Fork-Join workflow. Our design uses two innovative patterns, namely, Relay Composition and Master-Worker Composition to solve execution time-out challenges. As a Proof-of-Concept (PoC), the prototypical implementation of Split-Sort-Merge use case, based on Fork-Join workflow is discussed and evaluated. The ReactiveFnJ handles embarrassingly parallel computations, and its design does not depend on any external orchestration services, messaging services, and queue services. ReactiveFnJ facilitates in designing fully automated pipelines for distributed data processing systems, satisfying the Serverless Trilemma in true essence. A file of any size can be processed using our effective and extensible design without facing execution time-out challenges. The proposed model is generic and can be applied to a wide range of serverless applications that are based on the Fork-Join workflow pattern. It fosters the choreographed serverless composition for complex workflows. The proposed design model is useful for software engineers and developers in industry and commercial organizations, total solution vendors and academic researchers.

show abstract

Section: Related Workmentioning

confidence: 99%

ReactiveFnJ: A choreographed model for Fork-Join Workflow in Serverless Computing

2023

View full text Add to dashboard Cite

show abstract

“…Prior work attempts to overcome the missing locality in three main ways: (1) with a fast remote storage [64], [152], [153], (2) by specializing the system (e.g., making a DAG a first-class concept [85], [156], [158]), or (3) by using functions as (short-lived) containers (e.g., gg longlived mode [15], numpywren [13], or Kappa [155]). Unfortunately, these approaches are either inefficient, or give up some of the key benefits of the FaaS paradigm.…”

Section: Lack Of Localitymentioning

confidence: 99%

“…Replacing serverless frameworks with containers (such as AWS Fargate) enables existing frameworks to run. Another alternative is to use serverless functions as containers by setting up long-lived functions with reverse connections to a central controller [13], [15], [155]. Both container variants require provisioning servers and break serverless simplicity.…”

Section: Related Workmentioning

confidence: 99%

Informed optimization of cloud applications

Abdi

View full text Add to dashboard Cite

Cloud applications running in private data centers or public clouds are critical to every aspect of modern society, from entertainment to healthcare to financial services. The rapid growth of cloud applications is because organizations extensively leverage the general-purpose services offered by cloud platforms. Examples are general-purpose storage services such as AWS S3 [4]. These general-purpose services promise to provide high-performance and low-cost data analysis, elastic and scalable resources, and robust and reliable computations. Yet, cloud applications confront challenges to achieving these promises for several reasons: the exponential growth of data, the network limitations, the overhead of accessing the remote storage service, and the scarcity of complete information to make optimized decisions at different layers of the applications.This dissertation focuses on proposing systems to address the challenges mentioned above.We describe two systems that expand the general-purpose service APIs to exchange information critical to optimizing decisions. We also introduce components and algorithms that interpret the exchanged information and make optimized decisions.First, we describe Kariz to address the challenge of providing high-performance and scalable analytic applications with disaggregated storage-compute deployments. Kariz targets cloud applications that leverage general-purpose services such as analytical frameworks and object storage services. Kariz introduces a distributed caching system that extends the general-purpose object storage service and caches and prefetches input data accessed by the analytical framework. It extends the analytic framework APIs to exchange job dependency graphs (DAGs) and scheduling information with the cache, makes I/O scheduling decisions according to the scheduling information, estimates the run-time improvements, and predicts bandwidth to object storage service. Kariz integrates a cross-DAG scheduler and prefers pre-fetching input data shared by multiple DAGs. To build and evaluate Kariz, we characterize the job dependency graphs(DAGs) generated by analytic frameworks in a workload from in-production clusters.Then, we describe the elements of Kariz: a straggler-resistant partial cache system, a cache controller that run alongside the analytic frameworks-i.e. Spark and Pig-, and two graph-aware caching and pre-fetching algorithm for optimizing the caching of individual queries and three policies for optimizing across queries from, potentially, multiple different clusters. With an vi algorithm that fully exploits the rich information available from Kariz, we demonstrate major speedups (as much as 3×) for TPC-H and TPC-DS.Second, we describe Palette, to address the growing challenge of having computation close to data in general purpose Function-as-a-Service (FaaS) serverless computing. FaaS platforms enable a simple programming model with almost unbounded elasticity, making them an exciting choice for most cloud applications. However, current FaaS platforms achieve this...

show abstract