Empirical study design in the area of high-performance computing (HPC)

Abstract

The development of High-Performance Computing (HPC) programs is crucial to progress in many fields of scientific endeavor. We have run initial studies of the productivity of HPC developers and of techniques for improving that productivity, which have not previously been the subject of significant study. Because of key differences between development for HPC and for more conventional software engineering applications, this work has required the tailoring of experimental designs and protocols.A major contribution of our work is to begin to quantify the code development process in a specialized area that has previously not been extensively studied. Specifically, we present an analysis of the domain of High-Performance Computing for the aspects that would impact experimental design; show how those aspects are reflected in experimental design for this specific area; and demonstrate how we are using such experimental designs to build up a body of knowledge specific to the domain. Results to date build confidence in our approach by showing that there are no significant differences across studies comparing subjects with similar experience tackling similar problems, while there are significant differences in performance and effort among the different parallel models applied.

Claims about the use of software engineering practices in science: A systematic literature review

“…We found this claim in four studies [2,3,54,55]. There are two primary reasons given for this claim.…”

“…As was mentioned in the introduction, the cost or danger of performing these physical experiments is often the reason why scientists build software models in the first place, so the physical experiments will not be done before promising software models have been created. Second, experimental validation is frequently impractical since it is usually difficult or impossible to know what the correct result for a piece of software will be until the software is run [2,54,55]. In some cases, scientific software developers treat validation studies as research projects or theses in and of themselves due to the challenge in performing them.…”

Applying empirical software engineering to software architecture: challenges and lessons learned

“…The importance and current lack, of empirical assessment has been demonstrated in many software engineering areas like high performance computing (Shull et al 2005), agile software development (Dybå and Dingsøyr 2008), regression testing (Engstrom et al 2008), variability management (Chen et al 2009), reverse engineering (Tonella et al 2007), and information visualization (Ellis and Dix 2006). The Goal Question Metric (GQM) paradigm is a general approach for the "specification of a measurement system targeting a particular set Falessi, D., Ali Babar, M., Cantone, G., Kruchten, P., Empirically Assessing Software Architecture Research: Challenges and Lessons Learned, Empirical Software Engineering Journal, 15(3): pp.…”

A checklist for integrating student empirical studies with research and teaching goals

“…In this case, the goal of the ESWSs was to understand whether various development approaches could quickly increase the effectiveness of novice programmers. Therefore, because the desired outcome of the ESWSs was to learn about the activities of novice developers, students were exactly the right test population (Shull et al 2005;Hochstein et al 2006;Basili et al 2008). & Testing the feasibility of technologies.…”

“…(Shull et al 2000;Jaccheri 2001;Shull et al 2001;Baresi and Morasca 2002;Baresi et al 2003;Morasca 2003;Shull et al 2005;Walia and Carver 2006). In an earlier paper, we addressed some issues in ESWSs by presenting a framework for assessing student experiments from four points of view: researcher, student, instructor, and professional.…”