AutoBench: Finding Workloads That You Need Using Pluggable Hybrid Analyses

Abstract

Researchers often rely on benchmarks to demonstrate feasibility or efficiency of their contributions. However, finding the right benchmark suite can be a daunting taskexisting benchmark suites may be outdated, known to be flawed, or simply irrelevant for the proposed approach. Creating a proper benchmark suite is challenging, extremely time consuming, and also-unless it becomes widely popular-a thankless endeavor. In this paper, we introduce a novel approach to help researchers find relevant workloads for their experimental evaluation needs. Our approach relies on the huge number of open-source projects available in public repositories, and on unit testing having become best practice in software development. Using a repository crawler employing pluggable static and dynamic analyses for filtering and workload characterization, we allow users to automatically find projects with relevant workloads. Preliminary results presented here show that unit tests can provide a viable source of workloads, and that the combination of static and dynamic analyses improves the ability to identify relevant workloads that can serve as the basis for custom benchmark suites.

Large‐scale characterization of Java streams

“…We also provide a list of publicly available open‐source software projects that heavily use stream processing (see Table 2), which could be considered as potential workload candidates for benchmarking by researchers and tool builders. Indeed, prior work 20,48 has shown promising results on using JUnit tests as workloads for benchmarking. Suitable workload candidates would use stream code from modern open‐source applications, complementing benchmark suites made either from stream code adaptations of traditional MapReduce algorithms 78 or by converting workloads designed for relational databases to stream‐based code 79 …”

“…Some authors have explored unit tests as a source of workloads to create benchmark suites. Zheng et al 48 explore the feasibility of using unit tests available on open‐source projects as workloads for custom benchmarks. They find more than 500 Java and Scala projects containing unit tests that can be considered as good candidate workloads for benchmarking purposes.…”

“…We use Stream-Analyzer to profile all streams used during the execution of JUnit tests available in a software project, if any. Prior work 20,29,48 succeed to analyze multiple runtime characteristics of workloads exercised by unit tests, providing relevant statistics and revealing code patterns. Following this approach, we leverage JUnit tests to automatically exercise code available in publicly available software projects, enabling the otherwise extremely tedious task of locating candidate executable workloads for the massive application of dynamic analysis.…”

Automatically Assessing and Extending Code Coverage for NPM Packages

“…To be effective, it is crucial that DPA is applied to workloads covering a significant portion of application code, as DPA cannot provide any information on code that it is not executed. Previous work has shown [15], [16] that unit tests are a viable source of workloads for discovering runtime properties with DPA. In this section, we show that the test suite provided by TESA can also be used to extend the effectiveness of DPA tools.…”

Analysis and Optimization of Task Granularity on the Java Virtual Machine

“…A possible future direction could be to extend the results presented here by conducting a large-scale characterization of task granularity, performing related optimizations on a broad range of applications. To this end, one could integrate tgp into existing frameworks for large-scale dynamic analysis (such as AutoBench [151]), to automatically collect task-granularity profiles from many publicly available open-source workloads. Moreover, task-granularity analysis could be performed on multiple environments, including Cloud-based ones.…”