On the representativeness of embedded Java benchmarks

Abstract

Abstract-Java has become one of the predominant languages for embedded and mobile platforms due to its architecturally neutral design, portability, and security. But Java execution in the embedded world encompasses Java Virtual Machines (JVMs) specially tuned for the embedded world, with stripped-down capabilities, and configurations for memory-limited environments. While there have been some studies on desktop and server Java, there have been very few studies on embedded Java. The non proliferation of embedded Java benchmarks and the lack of widespread profiling tools and simulators have only exacerbated the problem. While the industry uses some benchmarks such as MorphMark, MIDPMark, and EEMBC Java Grinder Bench, their representativeness in comparison to actual embedded Java applications has not been studied. In order to conduct such a study, we gathered an actual mobile phone application suite and characterized it in detail. We measure several properties of the various applications and benchmarks, perform similarity/dissimilarity analysis and shed light on the representativeness of current industry standard embedded benchmarks against actual mobile Java applications. It was observed that for many characteristics, the applications had a broader range, indicating that the benchmarks were under representing the range of characteristics in the real world. Furthermore, we find that the applications exhibit less code reuse/hotness compared to the benchmarks. We also draw comparisons of the embedded benchmarks against popular desktop/client Java benchmarks, such as the SPECjvm98 and DaCapo. Interestingly, embedded applications spend a significant amount of time in standard library code, on average 65%, suggesting to the usefulness of software and hardware techniques to facilitate pre-compilation with out the real time resource overhead of JIT.

Full-system analysis and characterization of interactive smartphone applications

“…Although that study noted many of the same differences between real workloads and the SPEC benchmarks that we have noted, their results did not indicate that these differences lead to diverging microarchitectural conclusions as we have found in our study. In [20], the authors studied embedded Java applications and compared them against the leading Java benchmarks, SPEC and Decapo. Their results are consistent with ours.…”

A Java Processor IP Design for Embedded SoC

“…For example, for the Logic and String test programs from the Embedded Caffeine Mark test suite, JAIP always outperforms CVM-JIT. Nevertheless, these two test programs are quite synthetic and may not represent the workloads in real usage cases [Isen et al 2008]. One practical application where JAIP always outperforms CVM-JIT is the Pi program used in our test.…”

Don't race the memory bus: taming the GC leadfoot

“…We have ported all eight of the SPECjvm98 applications to Android, but for space reasons focus here on the most "representative" [28] of those eight: javac and jack. Because the full set of Java APIs are not all fully supported on Android and some packages are completely omitted, our porting of DaCapo is restricted to two multithreaded applications from the DaCapo 9.12 Bach release, lusearch and xalan.…”