Characterizing serverless platforms with serverlessbench

Yu, Tianyi; Li, Qingyuan; Du, Dong; Xia, Yubin; Zang, Binyu; Lu, Ziqian; Yang, Pingchao; Qin, Chenggang; Chen, Haibo

doi:10.1145/3419111.3421280

Cited by 113 publications

(76 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…vHive adopts dockerized benchmarks from FunctionBench that provides a diverse set of Python benchmarks [32,33]. Server-lessBench contains a number of multi-function benchmarks, focusing on function composition patterns [65]. Researchers and practitioners release a range of systems that combine the FaaS programming model and autoscaling [8-10, 30, 36].…”

Section: Related Work 81 Open-source Serverless Platformsmentioning

confidence: 99%

Benchmarking, analysis, and optimization of serverless function snapshots

Ustiugov

Petrov

Kogias

et al. 2021

Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

View full text Add to dashboard Cite

Serverless computing has seen rapid adoption due to its high scalability and flexible, pay-as-you-go billing model. In serverless, developers structure their services as a collection of functions, sporadically invoked by various events like clicks. High inter-arrival time variability of function invocations motivates the providers to start new function instances upon each invocation, leading to significant cold-start delays that degrade user experience. To reduce cold-start latency, the industry has turned to snapshotting, whereby an image of a fully-booted function is stored on disk, enabling a faster invocation compared to booting a function from scratch.This work introduces vHive, an open-source framework for serverless experimentation with the goal of enabling researchers to study and innovate across the entire serverless stack. Using vHive, we characterize a state-of-the-art snapshot-based serverless infrastructure, based on industry-leading Containerd orchestration framework and Firecracker hypervisor technologies. We find that the execution time of a function started from a snapshot is 95% higher, on average, than when the same function is memoryresident. We show that the high latency is attributable to frequent page faults as the function's state is brought from disk into guest memory one page at a time. Our analysis further reveals that functions access the same stable working set of pages across different invocations of the same function. By leveraging this insight, we build REAP, a light-weight software mechanism for serverless hosts that records functions' stable working set of guest memory pages and proactively prefetches it from disk into memory. Compared to baseline snapshotting, REAP slashes the cold-start delays by 3.7×, on average. CCS CONCEPTS• Computer systems organization → Cloud computing; • Information systems → Computing platforms; Data centers; • Software and its engineering → n-tier architectures.

show abstract

Section: Related Work 81 Open-source Serverless Platformsmentioning

confidence: 99%

Benchmarking, analysis, and optimization of serverless function snapshots

Ustiugov

Petrov

Kogias

et al. 2021

Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

View full text Add to dashboard Cite

show abstract

“…For the former, we use 19 multimedia processing functions, available online (see our code repository in §1). For multi-stage functions, we study four applications: two data analytics applications as in [23] (a MapReduce-based "word count" application; Thousand Island Scanner (THIS) [33], as well as (in §7.2), a distributed video processing benchmark), a cloud-based Illegitimate Mobile App Detector (IMAD) application [45] 4 , and an image thumbnail generator pipeline (Image Processing) from the ServerlessBench suite [47].…”

Section: Discussionmentioning

confidence: 99%

Ofc

Mvondo

Bacou

Nguetchouang

et al. 2021

Proceedings of the Sixteenth European Conference on Computer Systems

View full text Add to dashboard Cite

Cloud applications based on the "Functions as a Service" (FaaS) paradigm have become very popular. Yet, due to their stateless nature, they must frequently interact with an external data store, which limits their performance. To mitigate this issue, we introduce OFC, a transparent, vertically and horizontally elastic in-memory caching system for FaaS platforms, distributed over the worker nodes. OFC provides these benefits cost-effectively by exploiting two common sources of resource waste: (i) most cloud tenants overprovision the memory resources reserved for their functions because their footprint is non-trivially input-dependent and (ii) FaaS providers keep function sandboxes alive for several minutes to avoid cold starts. Using machine learning models adjusted for typical function input data categories (e.g., multimedia formats), OFC estimates the actual memory resources required by each function invocation and hoards the remaining capacity to feed the cache. We build our OFC prototype based on enhancements to the OpenWhisk FaaS platform, the Swift persistent object store, and the RAM-Cloud in-memory store. Using a diverse set of workloads, we show that OFC improves by up to 82 % and 60 % respectively the execution time of single-stage and pipelined functions. CCS Concepts: • Computer systems organization → Cloud computing; • Software and its engineering → n-tier architectures. * A part of work done while Jinho Hwang was at IBM Research.

show abstract

“…Similar to Maissen, they also found different CPU models executing the cloud function, which could also explain the scatter plot in Figure 5 in [55] (respectively Figure 4 in [53]), where two performance ranges for LINPACK were visible. The last included benchmarking paper in the SLR is another workbench, called ServerlessBench [56]. They included an experiment with an multi-threaded application compared to parallel function instances.…”

Section: Related Approachesmentioning

confidence: 99%

“…PaaS where single functionality in the scope of a cloud function is hidden within a larger application/microservice. Therefore, the multithreading aspect when optimizing performance and costs of cloud function is important to consider for FaaS experiments when using multi-core configurations [9,56]. This is different to long running applications in a PaaS area where multi-threaded implementations of pieces of functionality are suspended for code readability and maintenance which is also addressed by static code quality tools 21 .…”

Section: Checklist For Faas Benchmarkingmentioning

confidence: 99%

SeMoDe – Simulation and Benchmarking Pipeline for Function as a Service

Manner¹

2021

Bamberger Beiträge Zur Wirtschaftsinformatik Und Angewandten Informatik

View full text Add to dashboard Cite

Cloud computing started with the promise of delivering computing resources elastically at scale, pay per use and on demand self-service to name a few capabilities. In early 2016, Amazon Web Services (AWS) launched a new product called AWS Lambda which started the so called serverless hype and established a new cloud delivery model, namely Function as a Service (FaaS). FaaS offerings keep the promise of delivering computing resources on demand. They dynamically scale up and down function instances and introduce the most fine-grained billing model across all as-a-service offerings by accounting on a milliseconds basis. Despite this flexibility and the possibility to concentrate on the business functionality, a FaaS user loses operational control. Only a few configuration options remain to tune the functions. The first pay-as-you-go billing model raises new questions for performance-cost trade-offs. In order to choose a suitable configuration dependent on the use case and get a solid understanding of performance impact of FaaS platforms, SeMoDe implements a benchmarking and simulation pipeline. It calibrates a physical developer machine, simulates the function in different settings which are comparable to those of cloud offerings and enables a decision guidance to choose an appropriate configuration when deploying it. Based on a Structured Literature Review (SLR) to show the benchmarking and simulation efforts, I suggest a checklist for conducting fair, repeatable and meaningful benchmarks with a focus on documenting the experiments.

show abstract

Characterizing serverless platforms with serverlessbench

Cited by 113 publications

References 19 publications

Benchmarking, analysis, and optimization of serverless function snapshots

Benchmarking, analysis, and optimization of serverless function snapshots

Ofc

SeMoDe – Simulation and Benchmarking Pipeline for Function as a Service

Contact Info

Product

Resources

About