An observable and controllable testing framework for modern systems

2014 IEEE Seventh International Conference on Software Testing, Verification and Validation

Cohen

et al. 2014

Self Cite

Embedded systems tend to be interrupt-driven, yet the presence of interrupts can affect system dependability because there can be delays in servicing interrupts. Such delays can occur when multiple interrupt service routines and interrupts of different priorities compete for resources on a given CPU. For this reason, researchers have sought approaches by which to estimate worst-case interrupt latencies (WCILs) for systems. Most existing approaches, however, are based on static analysis. In this paper, we present SIMLATTE, a testing-based approach for finding WCILs. SIMLATTE uses a genetic algorithm for test case generation that converges on a set of inputs and interrupt arrival points that are likely to expose WCILs. It also uses an opportunistic interrupt invocation approach to invoke interrupts at a variety of feasible locations. Our evaluation of SIMLATTE on several non-trivial embedded systems reveals that it is considerably more effective and efficient than random testing. We also determine that the combination of the genetic algorithm and opportunistic interrupt invocation allows SIMLATTE to perform better than it can when using either one in isolation.

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

SimLatte: A Framework to Support Testing for Worst-Case Interrupt Latencies in Embedded Software

2014 IEEE Seventh International Conference on Software Testing, Verification and Validation

Cohen

et al. 2014

Self Cite

“…There has been some work on dealing with interrupts using testing and dynamic analysis techniques . For example, Regehr proposes a random testing technique using a restricted interrupt discipline to improve naive random testing by eliminating aberrant interrupts.…”

Section: Related Workmentioning

confidence: 99%

A hybrid approach to testing for nonfunctional faults in embedded systems using genetic algorithms

Software Testing Verif & Rel

Cohen

et al. 2018

Self Cite

Embedded systems are challenging to program correctly, because they use an interrupt-driven programming paradigm and run in resource-constrained environments. This leads to various classes of nonfunctional faults that can be detected only by customized verification techniques.These nonfunctional faults are specifically related to usage of resources such as time and memory. For example, the presence of interrupts can induce delays in interrupt servicing and in system execution time. Such delays can occur when multiple interrupt service routines and interrupts of different priorities compete for resources on a given CPU. As another example, stack overflows are caused when the combination of active methods and interrupt invocations on the stack grows too large, and these can lead to data loss and other significant device failures. To detect these types of nonfunctional faults, developers need to estimate worst-case resource usage. Most existing approaches for calculating such estimates are based on static analysis; however, these have a tendency to overapproximate the resources needed. Dynamic techniques such as random testing, in contrast, often underapproximate resource usage. In this article, we present SIMESPRESSO, a framework that uses a combination of static analysis and a test case generation algorithm to estimate worst-case resource usage. There are three different worst-case resource usage scenarios that we consider: (1) worst-case execution times,(2) worst-case interrupt latencies, and (3) worst-case stack usage. SIMESPRESSO first uses static analysis to identify program paths and interrupt interleavings that potentially lead to worst-case scenarios. It then uses a genetic algorithm to generate test cases that guide program execution down these paths, using these particular interrupt interleavings. We performed an empirical study to evaluate the effectiveness of SIMESPRESSO; our results show that SIMESPRESSO is more effective than static analysis approaches and improves significantly over the state of the art dynamic technique, random test case generation. We also find that when we use only the genetic algorithm, omitting the static analysis, SIMESPRESSO performs almost as effectively, but takes significantly longer to complete its task.

“…We categorize OPs as reads or writes. For OPs related to memory accesses, SIMEXPLORER identifies reads and writes using the observability support implemented in our prior work [16,33]. For OPs related to hardware accesses, SIMEXPLORER uses the features provided by Simics to directly identify reads and writes.…”

Section: Shared Resource Accessesmentioning

confidence: 99%

“…There has been some work on testing for races caused by software signals and interrupts [33,[79][80][81]. For example, Tahara et al [80] issue signals when a PUT attempts to access a global variable, amplifying the chance of exposing faults.…”

Section: Related Workmentioning

confidence: 99%

An automated framework to support testing for process‐level race conditions

Software Testing Verif & Rel

Rothermel

2017

Self Cite

Summary Race conditions are difficult to detect because they usually occur only under specific execution interleavings. Numerous program analysis and testing techniques have been proposed to detect race conditions between threads on single applications. However, most of these techniques neglect races that occur at the process level due to complex system event interactions. This article presents a framework, SIMEXPLORER, that allows engineers to effectively test for process‐level race conditions. SIMEXPLORER first uses dynamic analysis techniques to observe system execution, identify program locations of interest, and report faults related to oracles. Next, it uses virtualization to achieve the fine‐grained controllability needed to exercise event interleavings that are likely to expose races. We evaluated the effectiveness of SIMEXPLORER on 24 real‐world applications containing both known and unknown process‐level race conditions. Our results show that SIMEXPLORER is effective at detecting these race conditions, while incurring an overhead that is acceptable given its effectiveness improvements.