Program comprehension as fact finding

Abstract

Little is known about how developers think about design during code modification tasks or how experienced developers' design knowledge helps them work more effectively. We performed a lab study in which thirteen developers worked for 3 hours understanding the design of a 54 KLOC open source application. Participants had from 0 to 10.5 years of industry experience and were grouped into three "experts" and ten "novices." We observed that participants spent their time seeking, learning, critiquing, explaining, proposing, and implementing facts about the code such as "getFoldLevel has effects". These facts served numerous roles, such as suggesting changes, constraining changes, and predicting the amount of additional investigation necessary to make a change. Differences between experts and novices included that the experts explained the root cause of the design problem and made changes to address it, while novice changes addressed only the symptoms. Experts did not read more methods but also did not visit some methods novices wasted time understanding. Experts talked about code in terms of abstractions such as "caching" while novices more often described code statement by statement. Experts were able to implement a change faster than novices. Experts perceived problems novices did not and were able to explain facts novices could not. These findings have interesting implications for future tools.

The implications of method placement on API learnability

“…A more formal, empirically based model of programmers' behaviors would inform API, documentation and tool design. Recent research on programming understanding has framed programming comprehension in terms of fact finding -searching for discrete chunks of knowledge about how a large system works [13]. This might be a helpful way to frame programmers' exploration of APIs as well -for example in the tasks in this study, finding that a Transport class was required to send an email message was one fact that programmers needed to find in the documentation.…”

Searching the State Space: A Qualitative Study of API Protocol Usability

“…LaToza et al [20] brought programmers to the lab and asked them to contribute architecture-level design improvements to a 54KLOC open source tool. They noted several highlevel differences between experts and novice participants: novices focused more on symptoms of problems, experts on sources; novices spoke in terms of specifics, and experts in terms of abstractions; novices wasted more time understanding implementation details, while experts' focus was wider.…”

Finding causes of program output with the Java Whyline

“…Sillito extended this finding [18], showing that developers seek "focus points," or places in code related to the developer's goal. A similar study framed program understanding as "fact finding" [14], driven by efforts to discover properties of a program.…”