Continuous Integration Applied to Software-Intensive Embedded Systems – Problems and Experiences

Mårtensson, Torvald; Ståhl, Daniel; Bosch, Jan

doi:10.1007/978-3-319-49094-6_30

Cited by 12 publications

(17 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Trying to keep up with fast IoT development cycles, agile processes such as Continuous Integration (CI), or applying checks on every change, are increasingly applied to embedded systems [40]. While systems evolve, they are likely to incorporate new bugs and testing needs to follow the agile development.…”

Section: Metricmentioning

confidence: 99%

PHiLIP on the HiL: Automated Multi-platform OS Testing with External Reference Devices

Weiss,

Rottleuthner,

Schmidt

et al. 2021

Preprint

View full text Add to dashboard Cite

Developing an operating system (OS) for low-end embedded devices requires continuous adaptation to new hardware architectures and components, while serviceability of features needs to be assured for each individual platform under tight resource constraints. It is challenging to design a versatile and accurate heterogeneous test environment that is agile enough to cover a continuous evolution of the code base and platforms. This mission is even more challenging when organized in an agile open-source community process with many contributors such as for the RIOT OS. Hardware in the Loop (HiL) testing and Continuous Integration (CI) are automatable approaches to verify functionality, prevent regressions, and improve the overall quality at development speed in large community projects.In this paper, we present PHiLIP (Primitive Hardware in the Loop Integration Product), an open-source external reference device together with tools that validate the system software while it controls hardware and interprets physical signals. Instead of focusing on a specific test setting, PHiLIP takes the approach of a tool-assisted agile HiL test process, designed for continuous evolution and deployment cycles. We explain its design, describe how it supports HiL tests, evaluate performance metrics, and report on practical experiences of employing PHiLIP in an automated CI test infrastructure. Our initial deployment comprises 22 unique platforms, each of which executes 98 peripheral tests every night. PHiLIP allows for easy extension of low-cost, adaptive testing infrastructures but serves testing techniques and tools to a much wider range of applications. CCS Concepts: • Hardware → Hardware test; • Software and its engineering → Software testing and debugging; Operating systems; • Computer systems organization → Embedded systems.

show abstract

Section: Metricmentioning

confidence: 99%

PHiLIP on the HiL: Automated Multi-platform OS Testing with External Reference Devices

Weiss,

Rottleuthner,

Schmidt

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Mårtensson et al [31] investigated problems and experiences when striving for continuous integration of embedded systems. They found that the test environment is a limited resource, leading to a tendency to construct many test systems with custom HW.…”

Section: Rq1 Revisited In Light Of Related Workmentioning

confidence: 99%

“…The software in embedded systems has to be tested under realistic conditions, using real hardware [8,47]. Even when using continuous practices, such as nightly testing, system testing is a resource constrained process when compared to unit level testing, and there is typically not enough time to execute all test cases every night on all hardware versions [31,41]. If test cases fail intermittently in nightly testing, troubleshooting is often very costly and reproduction of failures very difficult, leading staff to distrust test results [29].…”

Section: Introductionmentioning

confidence: 99%

Intermittently Failing Tests in the Embedded Systems Domain

Strandberg¹,

Ostrand²,

Weyuker³

et al. 2020

Preprint

View full text Add to dashboard Cite

Software testing is sometimes plagued with intermittently failing tests and finding the root causes of such failing tests is often difficult. This problem has been widely studied at the unit testing level for open source software, but there has been far less investigation at the system test level, particularly the testing of industrial embedded systems. This paper describes our investigation of the root causes of intermittently failing tests in the embedded systems domain, with the goal of better understanding, explaining and categorizing the underlying faults. The subject of our investigation is a currently-running industrial embedded system, along with the system level testing that was performed. We devised and used a novel metric for classifying test cases as intermittent. From more than a half million test verdicts, we identified intermittently and consistently failing tests, and identified their root causes using multiple sources. We found that about 1-3% of all test cases were intermittently failing. From analysis of the case study results and related work, we identified nine factors associated with test case intermittence. We found that a fix for a consistently failing test typically removed a larger number of failures detected by other tests than a fix for an intermittent test. We also found that more effort was usually needed to identify fixes for intermittent tests than for consistent tests. An overlap between root causes leading to intermittent and consistent tests was identified. Many root causes of intermittence are the same in industrial embedded systems and open source software. However, when comparing unit testing to system level testing, especially for embedded systems, we observed that the test environment itself is often the cause of intermittence.

show abstract

“…We can also see that line managers' and project managers' main interest are test activities late in the pipeline (measuring project progress). This could also possibly explain why developers sometimes struggle to get resources to test frequently early in the pipeline, which we have observed in our previous work …”

Section: Designing a Pipeline For All Stakeholdersmentioning

confidence: 99%

“…In this study, we focus on large‐scale implementations of continuous integration and continuous delivery, where the product is a large‐scale software‐intensive embedded system (software systems combined with electronic and mechanical systems). In previous work, we have found multiple problems related to both scale and proximity to hardware . As we wish to find solutions viable also in some of the more difficult cases, we focus on these industry segments in this study (see Section .3 for further discussion of generalizability).…”

Section: Introductionmentioning

confidence: 99%

Test activities in the continuous integration and delivery pipeline

Mårtensson

Ståhl

Bosch

2019

J Software Evolu Process

Self Cite

View full text Add to dashboard Cite

Based on 25 interviews with participants from four case study companies that develop large‐scale software embedded systems, this paper presents the Test Activity Stakeholders (TAS) model. The TAS model shows how the continuous integration and delivery pipeline can be designed to include test activities that support four stakeholder interests: “Check changes,” “Secure stability,” “Measure progress,” and “Verify compliance.” The model is developed to show how each of the stakeholder interests are best supported by unit/component tests or system tests, by automated testing or manual testing and by tests executed in simulated environments or on real hardware. The TAS model may serve as a starting point for companies when evaluating and designing their continuous integration and delivery pipeline. The validation of the TAS model included twelve individuals from three case study companies. The validation showed that the TAS model is actionable and useful in practice, enabling the identification of stakeholders and showing where improvement efforts should be focused in order to support all stakeholder interests in the continuous integration and delivery pipeline.

show abstract

Continuous Integration Applied to Software-Intensive Embedded Systems – Problems and Experiences

Cited by 12 publications

References 5 publications

PHiLIP on the HiL: Automated Multi-platform OS Testing with External Reference Devices

PHiLIP on the HiL: Automated Multi-platform OS Testing with External Reference Devices

Intermittently Failing Tests in the Embedded Systems Domain

Test activities in the continuous integration and delivery pipeline

Contact Info

Product

Resources

About