ArchExplorer.org: A methodology for facilitating a fair Comparison of research ideas

Abstract

While reproducing the experimental results of research articles is standard practice in mature domains of science, such as physics or biology, it has not yet become mainstream in computer architecture. However, recent research shows that the lack of a fair and broad comparison of research ideas can be significantly detrimental to the progress, and thus the productivity, of research. At the same time, the complexity of architecture simulators and the fact that simulators are not systematically disseminated with novel ideas are largely responsible for this situation. While this methodology has a fundamental impact on research, it is by essence a practical issue.In this article, we present and set up an atypical approach to overcome this practical methodology issue, which takes the form of an open and continuous exploration through ArchExplorer, a server-side web infrastructure, that can significantly ease the process of fairly and quantitatively comparing research ideas. The web infrastructure ArchExplorer.org is now publicly open, and we demonstrate the approach with a set of data cache mechanisms. We show that this broad exploration can challenge some earlier assessments about data cache research, and even challenges the conclusions of an earlier but less thorough study on data cache comparison.

Statistical Performance Comparisons of Computers

“…For instance, statistical techniques were used for sampling simulation [36], principal components analysis was used to evaluate the representativeness of benchmarks [3,29], and regression techniques were used to model the design space of processors [10,13,18,22]. Therefore, the computer architecture community is already largely familiar with complex statistical tools, so that embracing a more rigorous performance measurement and comparison process is only a logical extension of the current trend.…”

ArchExplorer for Automatic Design Space Exploration

“…5 ArchExplorer, developed as part of the European SARC integrated project, is a framework for an open and permanent exploration of the architecture design space. This solution takes the form of a website rather than a traditional simulation environment.…”

“…Emer et al present a notable exception to parametric-only designspace exploration. 5 They create efficient branch predictors by decomposing branch prediction algorithms into elementary primitives and then composing them, creating new branch predictors. For both architectural and compiler exploration, many recent works, such as Ipek et al 6 and Lee et al, 7 discuss building statistical models using machine-learning techniques, showing that it is possible to converge rapidly to good solutions in a huge design space.…”