Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices

Abstract

Context: Continuous practices, i.e., continuous integration, delivery, and deployment, are the software development industry practices that enable organizations to frequently and reliably release new features and products. With the increasing interest in and literature on continuous practices, it is important to systematically review and synthesize the approaches, tools, challenges, and practices reported for adopting and implementing continuous practices.Objective: This research aimed at systematically reviewing the state of the art of continuous practices to classify approaches and tools, identify challenges and practices in this regard, and identify the gaps for future research. Method:We used systematic literature review (SLR) method for reviewing the peer-reviewed papers on continuous practices published between 2004 and 1st June 2016. We applied thematic analysis method for analysing the data extracted from reviewing 69 papers selected using predefined criteria. Results:We have identified thirty approaches and associated tools, which facilitate the implementation of continuous practices in the following ways: (1) -reducing build and test time in continuous integration (CI)‖; (2) -increasing visibility and awareness on build and test results in CI‖; (3) -supporting (semi-) automated continuous testing‖; (4) -detecting violations, flaws and faults in CI‖; (5) -addressing security and scalability issues in deployment pipeline‖, and (6) -improving dependability and reliability of deployment process‖. We have also determined a list of critical factors such as -testing (effort and time)‖, -team awareness and transparency‖, -good design principles‖, -customer‖, -highly skilled and motivated team‖, -application domain‖, and -appropriate infrastructure‖ that should be carefully considered when introducing continuous practices in a given organization. The majority of the reviewed papers were validation (34.7%) and evaluation (36.2%) research types. This review also reveals that continuous practices have been successfully applied to both greenfield and maintenance projects. Conclusion:Continuous practices have become an important area of software engineering research and practice. Whilst the reported approaches, tools, and practices are addressing a wide range of challenges, there are several challenges and gaps which require future research work for: improving the capturing and reporting of contextual information in the studies reporting different aspects of continuous practices; gaining a deep understanding of how software-intensive systems should be (re-) architected to support continuous practices; addressing the lack of knowledge and tools for engineering processes of designing and running secure deployment pipelines.

Continuous software engineering and its support by usage and decision knowledge: An interview study with practitioners

“…Further literature studies list benefits and challenges regarding continuous integration, delivery, and deployment. () Kevic et al reveal the positive impact of experiments; this points toward a relationship between continuous deployment and a change in user behavior and supports our model (Figure ). Dybå and Dingsøyr highlight human and social factors in work settings that are similar to CSE .…”

An empirical study of architecting for continuous delivery and deployment

“…Nonetheless, whilst previous works indicate that an unsuitable architecture would be a major barrier to CD transition [18,19,25,39], there has been little empirical research on the role of SA as a contributing factor when adopting CD practices. Some initial efforts on this topic have been reported in [22-24, 32, 45].…”

“…Whilst the industrial community, through white papers and practitioners' blogs, has investigated the role of software architecture in CD adoption [2, 20,21], there is little empirical effort to study how software architecture is being impacted by or is impacting CD practices [22][23][24]. This is evident in the recently published systematic reviews on CD [18,19,25], in which a new line of research has been called to explore how an application should be (re-) architected for CD. Furthermore, to succeed in the DevOps/CD movement, which emphasizes treating operations teams and operational aspects as first-class entities in the software development process, modern architectures should deal with both design and runtime considerations (e.g., predictive monitoring) [1,[26][27][28][29].…”

Towards continuous value delivery with review meetings in agile methodologies: A structured review

“… Scrumban: As scrumban holds daily meetings and also embraces on-demand planning, the increments are adjusted according to selected planning method (Ladas, 2009).  Nexus: The nexus integration team is accountable for ensuring that a "Done" (combined work completed) is produced at least once every sprint (Bittner et al, 2017;Shahin et al, 2017).  EVO: Continuous Integration was introduced with which the developers get their work out onto the test servers every week (Johansen and Gilb, 2005).…”

“…The term "value-based software engineering" was coined in Boehm (2003) that questioned the absence of "value" in traditional practices and their concerns were costs, resource allocation, scheduling etc. Some of the practices that can make it happen are Continuous Integration (CI), Continuous Delivery (CDE), and Continuous Deployment (CD) (Shahin et al, 2017). CI presents the procedure of integrating work-in-progress multiple times in a day, whereas CDE and CD shares the process of automating quick and reliable release of software.…”