Cross-language compiler benchmarking: are we fast yet?

Abstract

Comparing the performance of programming languages is difficult because they differ in many aspects including preferred programming abstractions, available frameworks, and their runtime systems. Nonetheless, the question about relative performance comes up repeatedly in the research community, industry, and wider audience of enthusiasts.This paper presents 14 benchmarks and a novel methodology to assess the compiler effectiveness across language implementations. Using a set of common language abstractions, the benchmarks are implemented in Java, JavaScript, Ruby, Crystal, Newspeak, and Smalltalk. We show that the benchmarks exhibit a wide range of characteristics using language-agnostic metrics. Using four different languages on top of the same compiler, we show that the benchmarks perform similarly and therefore allow for a comparison of compiler effectiveness across languages. Based on anecdotes, we argue that these benchmarks help language implementers to identify performance bugs and optimization potential by comparing to other language implementations.

Applying Optimizations for Dynamically-typed Languages to Java

“…We use benchmarks from the SciMark benchmark suite, 4 benchmarks from the Are We Fast Yet suite [18] and others. More speciically, our benchmarks consist of a micro benchmark that uses a bubble-sort algorithm and larger benchmarks including a Fast Fourier Transformation (FFT ), a dense LU matrix factorization (LU ), array permutations (Permute, Fannkuch), an n-queens problem solver (Queens), a Jacobi successive overrelaxation (SOR), and a sparse matrix multiplication (SparseMatMult).…”

Naïve transient cast insertion isn't (that) bad

“…For this work, we rely on the benchmark suite compiled for previous work [36]. It is a collection of 21 benchmarks in total, derived from the Are We Fast Yet benchmark suite [29] and other benchmarks from the gradual-typing literature.…”

Efficient and thread-safe objects for dynamically-typed languages

“…For that purpose, we take a set of twelve benchmarks that have been implemented for Java, JavaScript, and Ruby to enable a comparison of a set of core features of object-oriented languages [20]. This includes objects, closures, arrays, method dispatch, and basic operations.…”