Towards Automated Anomaly Report Assignment in Large Complex Systems Using Stacked Generalization

Abstract

Maintenance costs can be substantial for organizations with very large and complex software systems. This paper describes research for reducing anomaly report turnaround time which, if successful, would contribute to reducing maintenance costs and at the same time maintaining a good customer perception. Specifically, we are addressing the problem of the manual, laborious, and inaccurate process of assigning anomaly reports to the correct design teams. In large organizations with complex systems this is particularly problematic because the receiver of the anomaly report from customer may not have detailed knowledge of the whole system. As a consequence, anomaly reports may be wrongly routed around in the organization causing delays and unnecessary work. We have developed and validated a machine learning approach, based on stacked generalization, to automatically route anomaly reports to the correct design teams in the organization. A research prototype has been implemented and evaluated on roughly one year of real anomaly reports on a large and complex system at Ericsson AB. The prediction accuracy of the automation is approaching that of humans, indicating that the anomaly report handling time could be significantly reduced by using our approach.

Automated bug assignment: Ensemble-based machine learning in large scale industrial contexts

“…Previous work report that bug tossing is frequent in large projects; 25% of bug reports are reassigned in the Eclipse Platform project (Anvik and Murphy, 2011) and over 90% of the xed bugs in both the Eclipse Platform project and in projects in the Mozilla foundation have been reassigned at least once (Bhattacharya et al, 2012). Moreover, we have previously highlighted the same phenomenon in large-scale maintenance at Ericsson (Jonsson et al, 2012).…”

“…In the eld of software engineering, applications of SG include predicting the numbers of remaining defects in black-box testing (Li et al, 2011), and malware detection in smartphones (Amamra et al, 2012). In a previous pilot study, we initially evaluated using SG for bug assignment with promising results (Jonsson et al, 2012). Building on our previous work, this paper constitutes a deeper study using bug reports from dierent proprietary contexts.…”

Automatic Localization of Bugs to Faulty Components in Large Scale Software Systems Using Bayesian Classification

“…Similar to what was pointed out in the study above, users and managers at our organization question why they should trust a "blackbox" ML system 1 . The problem concerning uncertainty in the results from the tool manifests itself when a user is employing a fault localization tool and is getting a recommendation, but the tool does not offer any measure of the confidence in the recommendation.…”

“…Efficiency in the bug handling process is essential, and previous research [1], [2], [3], [4], [5] has shown how the process can be partially automated using machine learning (ML). The field can be split into two areas: i) routing of bugs to human designers, and ii) localization of faults to specific parts of the software.…”

“…The field can be split into two areas: i) routing of bugs to human designers, and ii) localization of faults to specific parts of the software. The routing of bugs can be done either to an individual person [3], [4] or to a design team [1], [2], [5]. The fault localization can be done either at the component level [6], at the file level [7], [8], or at a finer grained level such as a method, a statement, or individual lines of code (see the survey by Wong et al [9] for an overview of the area).…”