Extending the Space of Software Test Monitoring: Practical Experience

Abstract

Software reliability depends on the performed tests. Bug detection and diagnosis are based on test outcome (oracle) analysis. Most practical test reports do not provide sufficient information for localizing and correcting bugs. We have found the need to extend the space of test result observation in data and time perspectives. This resulted in tracing supplementary test result features in event logs. They are explored with combined text mining and log parsing techniques. Another important point is correlating test life cycle with project development history journaled in issue tracking and software version control repositories. Dealing with the outlined problems, neglected in the literature, we have introduced original analysis schemes. They focus on assessing test coverage, reasons of low diagnosability, and test result profiles. Multidimensional investigation of test features and their management is supported with the developed test infrastructure. This assures a holistic insight into the test efficiency to identify test scheme deficiencies (e.g., functional inadequacy, aging, insufficient coverage) and possible improvements (test set updates). Our studies have been verified in relevance to a real commercial project (industrial case study) and confronted with the experience of testers engaged in other projects.INDEX TERMS bug diagnostics, software defect repository, software testing, testing framework, test monitoring.

Enhancing Software Project Monitoring with Multidimensional Data Repository Mining

“…In commercial project repositories, other specific fields appear, e.g., in projects considered in [3], over 100 fields were used. This included information related to testing (test source, category, group, and path), which, in another project [44], were stored in a special test repository. On the other hand, many fields were rarely used (filled sporadically or assuming a single value).…”