A Controlled Experiment for Program Comprehension through Trace Visualization

Abstract

Software maintenance activities require a sufficient level of understanding of the software at hand that unfortunately is not always readily available. Execution trace visualization is a common approach in gaining this understanding, and among our own efforts in this context is EXTRAVIS, a tool for the visualization of large traces. While many such tools have been evaluated through case studies, there have been no quantitative evaluations to the present day. This paper reports on the first controlled experiment to quantitatively measure the added value of trace visualization for program comprehension.We designed eight typical tasks aimed at gaining an understanding of a representative subject system, and measured how a control group (using the Eclipse IDE) and an experimental group (using both Eclipse and EXTRAVIS) performed in terms of correctness and time spent. The results are statistically significant in both regards, showing a 22% decrease in time needed for the given tasks, and a 43% increase in correctness of the results for the group using trace visualization. Index TermsProgram comprehension, dynamic analysis, controlled experiment.B. Cornelissen is with the Software Improvement Group, A.J.

Aiding Software Developers to Maintain Developer Tests

“…Future work. We now list some of the important avenues for future work which both address usability and further research: (1) the possibility to navigate from the tests in the results window (see Figure 2) directly to the actual test code, (2) the incremental build-up of the test mapping, thus avoiding long waiting times, and (3) a more extensive evaluation using a controlled experiment that would give us insight into the time gain of using TestNForce (similar to, e.g., [24], [25], [20]). …”

Object-focused environments revisited

“…For example, Cornelissen et al evaluated a tool for visualizing execution traces [65], Stasko et al contrasted two visualization tools for depicting hierarchies using different space-filing layouts [66], Marcus et al analyzed the support for program comprehension tasks of their sv3D tool [67], and Wettel et al assessed the validity of the city metaphor for visualizing software systems [68].…”

Understanding JavaScript event-based interactions

“…There are many tools that use visualization to improve the process of understanding the behaviour of software applications. For instance, Extraviz [9] visualizes dynamic traces of Java applications to assist with program comprehension tasks. However, their approach does not concern itself with building a model of the web application, while ours does.…”