A cross-tool communication study on program analysis tool notifications

Security Notifications in Static Analysis Tools: Developers’ Attitudes, Comprehension, and Ability to Act on Them

“…Looking at SAT notification information specifically, notification content can be confusing and require a search engine to understand the words used [53]. Empirically, developers also require an unexpectedly large amount of time to fix low-complexity SAT-identified issues [46] suggesting that comprehending and applying the notification content was not straight-forward.…”

“…These results suggest that there is room for improvement in our understanding of how developers interpret SAT notification content and how that interpretation translates into their ability to correct identified issues. We extend the body of research on developers comprehension of and attitudes towards SAT security notification [53,54,89,90] by conducting a quantitative study with a larger sample and also looking at the effectiveness of SAT security notification content at assisting developers in fixing vulnerabilities.…”

“…SAT notification content can also be hard to understand [72,82]. Developers use web search engines to figure out what SAT notifications say [53]. At Microsoft, developers ranked "bad error messages" as their second most important issue with SATs [22], it was also the case that 75% of Google developer SAT bug reports were because of notifications misunderstandings [83].…”

Heard it through the Gitvine: an empirical study of tool diffusion across the npm ecosystem

“…Software engineering researchers and practitioners are often frustrated by the low adoption of practices and tools, such as static analysis, refactoring tools, program comprehension, or security testing, even though many of them are generally perceived to be beneficial. Researchers have found a wide range of technical and social pain points in tool adoption, such as false positives in static analysis tools [e.g., 39,64,71], missing trust in correctness [e.g., 74], crypting tool messages [e.g., 38,39], slow response times [e.g., 80], lack of workflow integration [e.g., 36,41,64,86], lack of collaboration support [39], lack of management buy-in [e.g., 18,80], overwhelming configuration effort [e.g., 25,36,39,71,80], and simply a lack of knowledge about tools [e.g., 60,74,87]. In response, most software engineering research has focused on technical solutions, such as improving functionality, accuracy, and performance [e.g., 8,64,72], improving usability [e.g., 38,46,51,72,79], and improving discoverability through recommendation mechanisms or process integration [e.g., 49,64,87].…”

One size does not fit all

“…Johnson et al [46], who investigated the reasons why developers rarely used static analyzers, found that large volumes of warnings, false positives, and poor warning presentation caused developers to stop using analyzers. Furthermore, Johnson et al [45] conducted a qualitative study, in which 26 participants interacted with warnings from FindBugs, the Eclipse Compiler, and EclEmma. The authors presented their resulting communication theory, which revealed understanding the notifications posed multiple challenges, which are caused by gaps and mismatches between developers' programming knowledge and communication methods used by the notifications.…”