Test activities in the continuous integration and delivery pipeline

Mårtensson, Torvald; Ståhl, Daniel; Bosch, Jan

doi:10.1002/smr.2153

Cited by 10 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Differently from their work, we qualitatively study 1,270 open-source projects. Our work can be complementary to the work by Maartensson and colleagues [24] in the sense that we shed light on some bad CI practices that can be avoided when designing a CI pipeline.…”

Section: Related Workmentioning

confidence: 68%

“…The study by Maartensson and colleagues [24] investigated the following general question "How can the continuous integration and delivery pipeline be designed in order to support all existing stakeholder interests." To this end, the authors surveyed practitioners from 10 software development companies which develop largescale softwareintensive embedded system.They proposed a conceptual model that shows practitioners how better design a CI pipeline to include test activities that support all the different interests of the involved stakeholders.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Continuous Integration Theater

Felidré

Furtado

Costa

et al. 2019

2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

View full text Add to dashboard Cite

Background: Continuous Integration (CI) systems are now the bedrock of several software development practices. Several tools such as TravisCI, CircleCI, and Hudson, that implement CI practices, are commonly adopted by software engineers. However, the way that software engineers use these tools could lead to what we call "Continuous Integration Theater", a situation in which software engineers do not employ these tools effectively, leading to unhealthy CI practices. Aims: The goal of this paper is to make sense of how commonplace are these unhealthy continuous integration practices being employed in practice. Method: By inspecting 1,270 open-source projects that use TravisCI, the most used CI service, we quantitatively studied how common is to use CI (1) with infrequent commits, (2) in a software project with poor test coverage, (3) with builds that stay broken for long periods, and (4) with builds that take too long to run. Results: We observed that 748 (∼60%) projects face infrequent commits, which essentially makes the merging process harder. Moreover, we were able to find code coverage information for 51 projects. The average code coverage was 78%, although Ruby projects have a higher code coverage than Java projects (86% and 63%, respectively). However, some projects with very small coverage (∼4%) were found. Still, we observed that 85% of the studied projects have at least one broken build that take more than four days to be fixed. Interestingly, very small projects (up to 1,000 lines of code) are the ones that take the longest to fix broken builds. Finally, we noted that, for the majority of the studied projects, the build is executed under the 10 minutes rule of thumb. Conclusions: Our results are important to an increasing community of software engineers that employ CI practices on daily basis but may not be aware of bad practices that are eventually employed.

show abstract

Section: Related Workmentioning

confidence: 68%

Section: Related Workmentioning

confidence: 99%

Continuous Integration Theater

Felidré

Furtado

Costa

et al. 2019

2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

View full text Add to dashboard Cite

show abstract

“…In our studies of continuous integration impediments, ‘reliability of test environments’ was identified as one of twelve factors that could enable more frequent integration of software [1]. Furthermore, ‘simulated test environment or real hardware’ is one of the main aspects to consider when designing a continuous integration and delivery pipeline [2], and ‘test environments that support debugging and recording’ are one of the key factors that enable efficient and effective exploratory testing [3].…”

Section: Introductionmentioning

confidence: 99%

Test environments for large‐scale software systems—An industrial study of intrinsic and extrinsic success factors

Mårtensson

Ancher

Ståhl

2023

Software Testing Verif & Rel

Self Cite

View full text Add to dashboard Cite

The characteristics of the test environment are of vital importance to its ability to support the organizations testing objectives. This paper seeks to address the need for a structured and reliable approach, which can be used by companies and other organizations to optimize their test environments in each individual case. The reported study included a series of interviews with 30 individuals, a series of focus groups with in total 31 individuals and a cross‐company workshop with 30 participants from five large‐scale companies, operating in different industry segments. The study resulted in a list of success factors, including not only characteristics and capabilities existing within a test environment (intrinsic success factors) but also properties not inherent to the test environment, but still vital for a successfully implemented test environment (extrinsic success factors). This distinction is important, as the root causes differ and as addressing them requires distinct approaches—not only of technology but also of organization, communication and collaboration. We find that successful implementations of test environments for large‐scale software systems depend primarily on how they support the company's business strategy, test organization and product testability (extrinsic success factors). Based on this, test environments can then be optimized to improve test environment capabilities, usability and stability (intrinsic success factors). The list of intrinsic and extrinsic success factors was well received by all five companies included in the study, supporting that the intrinsic and extrinsic success factors for test environments can be applied to a large segment of the software industry.

show abstract

The Testing Hopscotch Model – Six Complementary Profiles Replacing the Perfect All-Round Tester

Mårtensson,

Sandahl

2023

Product-Focused Software Process Improvement

View full text Add to dashboard Cite

Test activities in the continuous integration and delivery pipeline

Cited by 10 publications

References 30 publications

Continuous Integration Theater

Continuous Integration Theater

Test environments for large‐scale software systems—An industrial study of intrinsic and extrinsic success factors

The Testing Hopscotch Model – Six Complementary Profiles Replacing the Perfect All-Round Tester

Contact Info

Product

Resources

About