Run, actor, run: towards cross-actor language benchmarking

Abstract

The actor paradigm supports the natural expression of concurrency. It has inspired the development of several actorbased languages, whose adoption depends, to a large extent, on the runtime characteristics (i.e., the performance and scaling behaviour) of programs written in these languages. This paper investigates the relative runtime characteristics of Akka, CAF and Pony, based on the Savina benchmarks. We observe that the scaling of many of the Savina benchmarks does not reflect their categorization (into essentially sequential, concurrent and parallel), that many programs have similar runtime characteristics, and that their runtime behaviour may drastically change nature (e.g., go from essentially sequential to parallel) by tweaking some parameters. These observations lead to our proposal of a single benchmark program which we designed so that through tweaking of some knobs (we hope) we can simulate most of the programs of the Savina suite. CCS Concepts • Computing methodologies → Distributed programming languages; • General and reference → Evaluation; Performance.

High-Performance Deterministic Concurrency using Lingua Franca

“…Our evaluation is based on the Savina benchmark suite [34] for actor languages and frameworks. While this suite has several issues, as Blessing et al discuss in [13], Savina covers a wide range of patterns and, to the best of our knowledge, is the most comprehensive benchmark suite for actor frameworks that has been published. The Savina suite includes Akka implementations of all benchmarks.…”

“…We further omit the A*-Search and Logistic Map Series benchmarks from our presentation. A*-Search suffers from a severe race condition that results in wildly varying execution times [13]. Logistic Map Series is omitted as it violates actor semantics and requires explicit synchronization [13].…”

“…A*-Search suffers from a severe race condition that results in wildly varying execution times [13]. Logistic Map Series is omitted as it violates actor semantics and requires explicit synchronization [13]. For this reason, the CAF implementation needs to use a blocking call, which makes it slower than the other implementations by at least two orders of magnitude.…”

Run, agent, run! architecture and benchmarking of actor-based agents

“…These solutions include: Akka [15], a library developed for the JVM environment, enriched by a strong community with multiple complementary tools for distributed environments and stream processing; the C++ Actor Framework (CAF) [7], a library for creating concurrent programs in C++; Pony [8,9], an actor language for building robust parallel systems by providing data-race free isolation for actors. A comprehensive overview and benchmark over these works can be found in [3].…”

“…Access to the Lower-Level Language. As consequence of an approach based on compilation, the DSL provides direct access to any object or function available in the agent's name space 3 . These lower-level access statements, indicated by the token #, are translated literally to the same statement in the underlying language.…”