Software test results exploration and visualization with continuous integration and nightly testing

Abstract

Software testing is key for quality assurance of embedded systems. However, with increased development pace, the amount of test results data risks growing to a level where exploration and visualization of the results are unmanageable. This paper covers a tool, Tim, implemented at a company developing embedded systems, where software development occurs in parallel branches and nightly testing is partitioned over software branches, test systems and test cases. Tim aims to replace a previous solution with problems of scalability, requirements and technological flora. Tim was implemented with a reference group over several months. For validation, data were collected both from reference group meetings and logs from the usage of the tool. Data were analyzed quantitatively and qualitatively. The main contributions from the study include the implementation of eight views for test results exploration and visualization, the identification of four solutions patterns for these views (filtering, aggregation, previews and comparisons), as well as six challenges frequently discussed at reference group meetings (expectations, anomalies, navigation, integrations, hardware details and plots). Results are put in perspective with related work and future work is proposed, e.g., enhanced anomaly detection and integrations with more systems such as risk management, source code and requirements repositories.

Power Grid Intelligent Energy Dispatching Interactive System Based on Virtual Reality Technology

“…e visualization interface provides a platform for data display, function expression, and interactive calculation. Interactive calculation organically adds the human role to a closed-loop control system [19,20]. is kind of interactive computing integrates the dispatcher's perception and experience and can conduct real-time computing and make corresponding verication through the interactive computing window.…”

Application of Software Data Analysis Model Based on K-Means Clustering Algorithm

“…Combined with the detection of fuzziness, the abnormal classification and identification of portable multidimensional control software testing are realized. However, this method has a large computational load for the abnormal detection of portable multidimensional control software testing [7]. Dang et al proposed an anomaly detection method for portable multidimensional control software testing based on fusion scheduling of multisource information resources as a service (MIRaaS).…”

Making Sense of Failure Logs in an Industrial DevOps Environment

“…Data Collection: Westermo provided test results data from two months of nightly testing to design and evaluate the approach. The data was in the form of a test results database, log files from their nightly testing, as well as a tool for exploring and querying the test results [28]. To evaluate the approach in the studied industrial setting, we use log messages that are tagged as erroneous and critical from three time windows that engineers might typically use in daily work for log review, as follows.…”