A systematic literature review of actionable alert identification techniques for automated static code analysis

Abstract

a b s t r a c tContext: Automated static analysis (ASA) identifies potential source code anomalies early in the software development lifecycle that could lead to field failures. Excessive alert generation and a large proportion of unimportant or incorrect alerts (unactionable alerts) may cause developers to reject the use of ASA. Techniques that identify anomalies important enough for developers to fix (actionable alerts) may increase the usefulness of ASA in practice. Objective: The goal of this work is to synthesize available research results to inform evidence-based selection of actionable alert identification techniques (AAIT).Method: Relevant studies about AAITs were gathered via a systematic literature review. Results: We selected 21 peer-reviewed studies of AAITs. The techniques use alert type selection; contextual information; data fusion; graph theory; machine learning; mathematical and statistical models; or dynamic detection to classify and prioritize actionable alerts. All of the AAITs are evaluated via an example with a variety of evaluation metrics. Conclusion:The selected studies support (with varying strength), the premise that the effective use of ASA is improved by supplementing ASA with an AAIT. Seven of the 21 selected studies reported the precision of the proposed AAITs. The two studies with the highest precision built models using the subject program's history. Precision measures how well a technique identifies true actionable alerts out of all predicted actionable alerts. Precision does not measure the number of actionable alerts missed by an AAIT or how well an AAIT identifies unactionable alerts. Inconsistent use of evaluation metrics, subject programs, and ASAs in the selected studies preclude meta-analysis and prevent the current results from informing evidence-based selection of an AAIT. We propose building on an actionable alert identification benchmark for comparison and evaluation of AAIT from literature on a standard set of subjects and utilizing a common set of evaluation metrics.

Learning from Human Repairs Through the Exploitation of Software Repositories

“…In 2011, Sarah Heckmann et al [37] have made an inventory and evaluation of defects detections technique. To do this work, they first collected articles in scientific journals (ACM, IEEE, Springer); They sorted these articles manually according to their importance in relation to the problem.…”

“…As we have just seen, for the renaming of entities this will be an adequate behavior. The reasoning behind this decision was that research showed that the majority of programmers did not care about violations [37]. Therefore, a violation may go unnoticed for a long period of time before it simply disappears due to unexpected changes in the source code.…”

Finding patterns in static analysis alerts: improving actionable alert ranking

“…This fails for situations where a programmer writes code that performs as intended and does not wish to modify. We therefore use the term actionable alert (AA) to define a SA alert that the programmer would act on to resolve and unactionable alert (UA) to define a SA alert that the programmer would not act on to resolve [15].…”

“…Because of the prevalence of unactionable alerts in SA warnings, considerable work has attempted to predict whether alerts are actionable or not; Heckman and Williams survey this literature [15]. Of particular importance in this research are the set of characteristics used to predict what SA alerts are actionable or unactionable.…”

“…Of particular importance in this research are the set of characteristics used to predict what SA alerts are actionable or unactionable. Heckman and Williams refer to these characteristics as alert characteristics (AC) [15], a term which we adopt in this paper.…”