The DaCapo benchmarks

Abstract

Since benchmarks drive computer science research and industry product development, which ones we use and how we evaluate them are key questions for the community. Despite complex runtime tradeoffs due to dynamic compilation and garbage collection required for Java programs, many evaluations still use methodologies developed for C, C++, and Fortran. SPEC, the dominant purveyor of benchmarks, compounded this problem by institutionalizing these methodologies for their Java benchmark suite. This paper recommends benchmarking selection and evaluation methodologies, and introduces the DaCapo benchmarks, a set of open source, client-side Java benchmarks. We demonstrate that the complex interactions of (1) architecture, (2) compiler, (3) virtual machine, (4) memory management, and (5) application require more extensive evaluation than C, C++, and Fortran which stress (4) much less, and do not require (3). We use and introduce new value, time-series, and statistical metrics for static and dynamic properties such as code complexity, code size, heap composition, and pointer mutations. No benchmark suite is definitive, but these metrics show that DaCapo improves over SPEC Java in a variety of ways, including more complex code, richer object behaviors, and more demanding memory system requirements. This paper takes a step towards improving methodologies for choosing and evaluating benchmarks to foster innovation in system design and implementation for Java and other managed languages.

Actor-Based Parallel Dataflow Analysis

“…Our test inputs are the programs luindex, antlr, and jython from the DaCapo Benchmark Suite version 2006-10-MR2 [3]. We analyse the source of the benchmarks only, but not the standard libraries.…”

“…E-IFDS call-site code def CallSiteActor(n) [1] local PathEdge := ∅ [2] local CallEdge := ∅ [3] local CalleePathEdge := ∅ begin (message) switch [4] case message matches AddPathEdge d1, d2 : [5] if d1 → d2 / ∈ PathEdge then [6] Insert d1 → d2 into PathEdge [7] for each p ∈ calledProcs(n) and d3 ∈ flow call (n, d2, p) do [8] Propagate(sp, AddPathEdge d3, d3 ) [9] Insert p, d2 → d3 into CallEdge [10] for each d4 such that p, d3 → d4 ∈ CalleePathEdge do [11] for each d5 ∈ flowret(ep, d4, n, d2) do [12] Propagate(returnSite(n), AddPathEdge d1, d5 ) [13] od [14] od [15] od [16] for each d3 ∈ flow thru (n, d2) do [17] Propagate(returnSite(n), AddPathEdge d1, d3 ) [18] od [19] fi [20] case message matches AddCalleePathEdge p, d1, d2 :…”

“…Each of T threads in the scheduler runs Algorithm 5 until stopped. Each thread executes tasks from its own queue (specified by qid) until a failure occurs, at which time it yields control to any other threads ready to run (line 3) and then executes a task from while true do [2] if ExecuteNext (qid) == false then [3] Yield () [4] while ExecuteNext (rand () % T) == false do Yield () od [5] fi [6] od end static function ExecuteNext (qid: Integer) begin [7] if…”

Practical Extensions to the IFDS Algorithm

“…For clarity of presentation, we have split this function into four separate functions: -flow(n ) computes all intraprocedural edges. 1 -passArgs(n ) computes call-to-start edges when n is at a call site.…”

“…We ran the extended Benchmark Methods Variables Instructions Possible Types antlr 949 10839 16621 257 bloat 3142 33727 46550 623 chart 9419 91280 129850 2292 fop 13556 131901 185129 3400 hsqldb 768 8004 11552 443 jython 5487 56090 74031 1079 luindex 1306 12519 18131 617 lusearch 1633 14850 21368 676 pmd 3643 33945 49640 998 xalan 786 7708 11084 451 Table 1. Benchmark Characteristics algorithm on the DaCapo Benchmark Suite, version 2006-10-MR2 [1]. Since most of the benchmarks use reflection, we provided Soot with summaries of uses of reflection obtained by instrumenting the benchmarks using ProBe [11] and *J [5].…”