BeFaaS: An Application-Centric Benchmarking Framework for FaaS Platforms

Abstract

Due to the popularity of the FaaS programming model, there is now a wide variety of commercial and open-source FaaS systems. Hence, for comparison of different FaaS systems and their configuration options, FaaS application developers rely on FaaS benchmarking frameworks. Existing frameworks, however, tend to evaluate only single isolated aspects, a more holistic application-centric benchmarking framework is still missing.In previous work, we proposed BeFaaS, an extensible application-centric benchmarking framework for FaaS environments that focuses on the evaluation of FaaS platforms through realistic and typical examples of FaaS applications. In this extended paper, we (i) enhance our benchmarking framework with additional features for distributed FaaS setups, (ii) design application benchmarks reflecting typical FaaS use cases, and (iii) use them to run extensive experiments with commercial cloud FaaS platforms (AWS Lambda, Azure Functions, Google Cloud Functions) and the tinyFaaS edge serverless platform. BeFaaS now includes four FaaS application-centric benchmarks, is extensible for additional workload profiles and platforms, and supports federated benchmark runs in which the benchmark application is distributed over multiple FaaS systems while collecting fine-grained measurement results for drill-down analysis.Our experiment results show that (i) network transmission is a major contributor to response latency for function chains, (ii) this effect is exacer- * This work extends [1]. bated in hybrid edge-cloud deployments, (iii) the trigger delay between a published event and the start of the triggered function ranges from about 100ms for AWS Lambda to 800ms for Google Cloud Functions, and (iv) Azure Functions shows the best cold start behavior for our workloads.

Let's Trace It: Fine-Grained Serverless Benchmarking using Synchronous and Asynchronous Orchestrated Applications

“…Serverless benchmarks and measurement frameworks: Table 3 compares ServiBench with the most important serverless benchmarks and performance frameworks. Our study (i) has a wider scope with more applications, functions, and services, and in particular with more than one function or service, (ii) adds realistic applications and invocation patterns based on real-world characteristics [22,66,84], (iii) does not rely on low-level, server-side tracing, not available for public serverless platforms, and (iv) enables analysis across a variety of situations common in production, including synchronous Reference Focus Scope Invocation Patterns Insights apps func/app micro langs services concurrent trace-based white box async faas-profiler [65] Server-level overheads 5 1 28 2 0 vHive [73] Cold-start breakdown 9 1 0 1 1 ServerlessBench [86] Diverse test cases 4 1-7 10 4 1 SeBS [14] Memory size impact 10 1 0 2 1 FunctionBench [34] Diverse workloads 8 1 6 1 1 FaaSDom [46] Language comparison 0 -5 4 0 BeFaaS [27] Application-centric 1 17 0 1 1…”

Streaming vs. Functions: A Cost Perspective on Cloud Event Processing

“…Function infras-tructure is completely handled by the cloud service provider, i.e., "serverless", and consumers pay per request based on the resources consumed by a function [12,22]. Functions can be implemented in a number of programming languages and can be invoked by web requests, IoT sensor readings, database updates, and even other functions, so-called function chaining [23,24].…”

“…Although including a cost model in cloud benchmarking studies is considered good scientific practice [37], in existing benchmarking studies on FaaS [24,[38][39][40][41] and DSP [4,42,43], cost evaluations can mainly be found for cloud functions, where the pay-perexecution pricing model has presented a significant paradigm shift.…”

Fusionize: Improving Serverless Application Performance through Feedback-Driven Function Fusion

“…In serverless computing, developers rely on managed cloud platform services as application building blocks which are connected through application-specific glue code. Usually, this glue code is written as sequences of small, stateless functions [3,4] -often invoked in an event-driven modelrunning on top of Function-as-a-Service (FaaS) platforms such as AWS Lambda 1 or Google Cloud Functions 2 .…”

“…In this section, we give a brief overview of FaaS as a paradigm, discuss influence factors on performance and cost, and finally function fusion as a concept. FaaS is a popular paradigm for cloud software development and deployment in which developers write their source code in the form of small, stateless functions 3 and leave all operations aspects to the cloud provider. These functions are usually written in highlevel programming languages such as Node.js [10].…”