Spectrum-Based Log Diagnosis

Abstract

Background: Continuous Engineering practices are increasingly adopted in modern software development. However, a frequently reported need is for more effective methods to analyze the massive amounts of data resulting from the numerous build and test runs. Aims: We present and evaluate Spectrum-Based Log Diagnosis (SBLD), a method to help developers quickly diagnose problems found in complex integration and deployment runs. Inspired by Spectrum-Based Fault Localization, SBLD leverages the differences in event occurrences between logs for failing and passing runs, to highlight events that are stronger associated with failing runs. Method: Using data provided by Cisco Norway, we empirically investigate the following questions: (i) How well does SBLD reduce the effort needed to identify all failure-relevant events in the log for a failing run? (ii) How is the performance of SBLD affected by available data? (iii) How does SBLD compare to searching for simple textual patterns that often occur in failure-relevant events? We answer (i) and (ii) using summary statistics and heatmap visualizations, and for (iii) we compare three configurations of SBLD (with resp. minimum, median and maximum data) against a textual search using Wilcoxon signed-rank tests and the Vargha-Delaney measure of stochastic superiority. Results: Our evaluation shows that (i) SBLD achieves a significant effort reduction for the dataset used, (ii) SBLD benefits from additional logs for passing runs in general, and it benefits from additional logs for failing runs when there is a proportional amount of logs for passing runs in the data. Finally, (iii) SBLD and textual search are roughly equally effective at effort-reduction, while textual search has slightly better recall. We investigate the cause, and discuss how it is due to characteristics of a specific part of our data. Conclusions: We conclude that SBLD shows promise as a method for diagnosing failing runs, that its performance is positively affected by additional data, but that it does not outperform textual search on the dataset considered. Future work includes investigating SBLD's generalizability on additional datasets. CCS CONCEPTS • Software and its engineering → Software development process management; Software defect analysis.

Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning

“…Rosenberg and Moonen [7] studied Spectrum-based method for aiding log diagnosis and root cause analysis using data from Cisco Norway. Their end goal is similar to ours but their method is quite different.…”

“…Spectrum based log analysis applies techniques from source code fault localization research where the idea is to compute a suspiciousness score for each source code line based on its participation in passing or failing test case [16]. In [7] authors extend the same idea to log events. Their approach requires a large sample of log sequences of failing executions and thus does not apply to our company case.…”

“…[1]- [5]. Yet, only a few works [6], [7] have focused on the rootcause analysis or failure localization in log sequences like we do in this paper. Recently there have been works on rootcause analysis or anomaly localization in Micro-service based systems, e.g.…”

Extending the Space of Software Test Monitoring: Practical Experience

“…Test diagnosability evaluation needs deeper analysis. In [38] a spectrum-based approach is proposed which optimizes test suit sets in relevance to fault localization capabilities. It is based on an activity matrix correlating software components (e.g., class, method, branch) with stressing them test cases.…”

“…This additionally confirms negligence of the pointed problems in correlation with testing, which we found as important practical issues needing investigation. [2, 5, 8, 12 -14, 34-36, 44] Testing project performance [16,17] Bug classification [17, 24-26, 28, 29] Test quality assessment [1 -4, 6, 10, 11, 15, 21, 34-37] Fault localization and program diagnosability [3,22,23,38,39] Test optimization and prioritization [18 -23, 42, 43] Logging schemes [9,28,30] Log analysis and parsing [9, 30 -32] Anomaly detection [9,33,34] Log statement optimization [9,34] Issue and version control tracking [7,25,30,37] Bug triaging [27] The created test scenarios are upgraded during the development and maintenance of the program. Moreover, in a large extent they are used during regression testing in relevance to function upgrades, extensions, or corrections.…”