Usage, costs, and benefits of continuous integration in open-source projects

Abstract

Continuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI rising as a big success story in automated software engineering, it has received almost no attention from the research community. For example, how widely is CI used in practice, and what are some costs and benefits associated with CI? Without answering such questions, developers, tool builders, and researchers make decisions based on folklore instead of data.In this paper, we use three complementary methods to study the usage of CI in open-source projects. To understand which CI systems developers use, we analyzed 34,544 opensource projects from GitHub. To understand how developers use CI, we analyzed 1,529,291 builds from the most commonly used CI system. To understand why projects use or do not use CI, we surveyed 442 developers. With this data, we answered several key questions related to the usage, costs, and benefits of CI. Among our results, we show evidence that supports the claim that CI helps projects release more often, that CI is widely adopted by the most popular projects, as well as finding that the overall percentage of projects using CI continues to grow, making it important and timely to focus more research on CI. CCS Concepts•Software and its engineering → Agile software development; Software testing and debugging;Keywords continuous integration; mining software repositories ASE'16, September 3-7, 2016, Singapore, Singapore c 2016 ACM. 978-1-4503-3845-5/16/09...$15.00 http://dx.

The impact of continuous integration on other software development practices: A large-scale empirical study

“…More recently, the ease of use of the TRAVIS CI [38] system led to its popularity on GITHUB, and triggered a series of research studies [5], [12], [22], [26], [39], [40]. This line of research is closer to the current work as it performs empirical analysis of TRAVIS CI data.…”

“…While no standard definitions exist (the term is often overloaded), here we refer to DEVOPS as a culture that emphasizes automation of the processes of building, testing, and deploying software. In practice, DEVOPS is supported by a multitude of tools for configuration management, cloud-based continuous integration, and automated deployment, which enjoy widespread open-source [5] and industrial adoption [6], [7].…”

“…In a decentralized, social coding context such as GITHUB, CI is particularly relevant. By automatically building and testing a project's code base, in isolation, with each incoming code change (i.e., push commit, pull request), CI has the potential to: (i) speed up development (code change throughput) [5], [10], [11]; (ii) help maintain * These authors contributed equally to this work code quality [12], [13]. Clearly, CI promises to be a disruptive technology in distributed software development.…”

“…In this paper we report on a study of a large sample of GITHUB open-source projects that adopted TRAVIS CI, by far the most popular CI infrastructure used on GITHUB [5]. In particular, we focus on the transition to using TRAVIS CI, and investigate how development practices changed following this shift.…”

TravisTorrent: Synthesizing Travis CI and GitHub for Full-Stack Research on Continuous Integration

“…• Do multiple integration environments lead to fewer defects? Only recently have researchers begun to discover TRAVIS CI as a data source [4][5][6][7]. They have, however, not yet taken advantage of the endless possibilities that the combination of a streamlined, popular and tightly coupled CI environment (TRAVIS CI), version control system (GIT) and collaboration platform (GITHUB) provide, as collecting and aggregating this data in a single data set is logistically and algorithmically complex.…”