BACKGROUND-The software intensive industry is moving towards the adoption of a value-driven and adaptive real-time business paradigm. The traditional view of software as an item that evolves through releases every few months is being replaced by the continuous evolution of software functionality. OBJECTIVE-This study aims to classify and analyse the literature related to continuous deployment in the software domain in order to scope the phenomenon, provide an overview of the state-of-the-art, investigate the scientific evidence in the reported results and identify areas suitable for further research. METHOD-We conducted a systematic mapping study and classified the continuous deployment literature. The benefits and challenges related to continuous deployment were also analysed. RESULTS-The systematic mapping study includes 50 primary studies published between 2001 and 2014. An in-depth analysis of the primary studies revealed ten recurrent themes that characterize continuous deployment and provide researchers with directions for future work. In addition, a set of benefits and challenges of which practitioners may take advantage were identified. CONCLUSION-Overall, although the topic area is very promising, it is still in its infancy, thus offering a plethora of new opportunities for both researchers and software intensive companies.

No abstract

Two heuristics namely diversity-based (DBTP) and history-based test prioritization (HBTP) have been separately proposed in the literature. Yet, their combination has not been widely studied in continuous integration (CI) environments. The objective of this study is to catch regression faults earlier, allowing developers to integrate and verify their changes more frequently and continuously. To achieve this, we investigated six open-source projects, each of which included several builds over a large time period. Findings indicate that previous failure knowledge seems to have strong predictive power in CI environments and can be used to effectively prioritize tests. HBTP does not necessarily need to have large data, and its effectiveness improves to a certain degree with larger history interval. DBTP can be used effectively during the early stages, when no historical data is available, and also combined with HBTP to improve its effectiveness. Among the investigated techniques, we found that history-based diversity using NCD Multiset is superior in terms of effectiveness but comes with relatively higher overhead in terms of method execution time. Test prioritization in CI environments can be effectively performed with negligible investment using previous failure knowledge, and its effectiveness can be further improved by considering dissimilarities among the tests. arXiv:1809.00143v1 [cs.SE] 1 Sep 2018 functionality. RT is widely used in practice; it is common to have a dedicated regression test suite that is often run in its entirety [4].A test suite for enterprise-sized applications often includes thousands of test cases, the execution of which requires several hours or even days. For instance, the JOnAS Java EE middleware comprises 2,689 test cases [5]. Applying them all to its 16 configurations results in running 43,024 test cases. Furthermore, the cost of RT increases over the time with the increase in system size. Memon et. al [6] observed linear growth in both code submission rate and size of regression test suite, so incurring significant expenses to keep the RT running. The software engineering literature has proposed many techniques to improve RT processes. Test suite minimization (TSM) [7] aims to eliminate test cases from a test suite with a specific objective, i.e., removing obsolete or redundant test cases. Several experiments have been reported in the literature with differing conclusions regarding the impact of TSM on the fault detection capability of a test suite, e.g., [8,9,10]. However, the common understanding is that TSM may compromise such capability. Test case prioritization (TCP) [11], on the other hand, is concerned with the ideal ordering of test cases to maximize desirable properties (i.e., early fault detection). From the perspective of fault detection, TCP seems to be a safe approach because it does not eliminate test cases and simply permutes them within the test suite.The intersection of CI and RT poses great challenges for the software development industry. The testing budget is often...

No abstract

Context:The alignment of different software engineering activities for coordinated functioning and optimized product development is of great importance, particularly in industrialscale development. The link between intermediate activities has been researched extensively, but the link between requirements engineering (RE) and software testing (ST) is a relatively less explored area. Objective: The objective of this study is to aggregate, structure, and classify all existing research regarding alignment of RE and ST published by the end of 2015. Method: We conducted a systematic mapping study (SMS) and aggregated all studies relevant to our scope. The primary studies are analyzed in terms of publication trend, focus area, i.e., how alignment is supported, the application domain and benefits and challenges, methodological data, and scientific rigor and industrial relevance. Results: There is a growing interest towards the topic. Several different techniques have been identified to improve RE and ST alignment. Test generation from requirements specification has received most attention. Alignment of RE and ST is particularly important for large safety-critical domains. While many challenges have been reported, the supporting evidence for benefits is scarce. Frameworks/methods/techniques is the most frequent contribution type. Solution proposal and evaluation research were the most frequently applied research type. Case study research was the most frequently applied research method, however, almost half of the studies did not clearly report any research method. Conclusion: Despite the numerous approaches that are proposed, it is not clear what approach is suitable in what context and why. To support industry in RE and ST alignment, guidelines and tool support are needed. The supporting evidence for claimed benefits is very limited. Overall, the research area is in its early stages and an increase in both the number and rigor of empirical studies are required.

A classical heuristic in software testing is to reward diversity, which implies that a higher priority must be assigned to test cases that differ the most from those already prioritized. This approach is commonly known as similarity-based test prioritization (SBTP) and can be realized using a variety of techniques. The objective of our study is to investigate whether SBTP is more effective at finding defects than random permutation, as well as determine which SBTP implementations lead to better results. To achieve our objective, we implemented five different techniques from the literature and conducted an experiment using the defects4j dataset, which contains 395 real faults from six real-world open-source Java programs. Findings indicate that running the most dissimilar test cases early in the process is largely more effective than random permutation (Vargha-Delaney A [VDA]: 0.76-0.99 observed using normalized compression distance). No technique was found to be superior with respect to the effectiveness. Locality-sensitive hashing was, to a small extent, less effective than other SBTP techniques (VDA: 0.38 observed in comparison to normalized compression distance), but its speed largely outperformed the other techniques (i.e., it was approximately 5-111 times faster). Our results bring to mind the well-known adage, "don't put all your eggs in one basket". To effectively consume a limited testing budget, one should spread it evenly across different parts of the system by running the most dissimilar test cases early in the testing process.

A classical heuristic in software testing is to reward diversity, which implies that a higher priority must be assigned to test cases that differ the most from those already prioritized. This approach is commonly known as similarity-based test prioritization (SBTP) and can be realized using a variety of techniques. The objective of our study is to investigate whether SBTP is more effective at finding defects than random permutation, as well as determine which SBTP implementations lead to better results. To achieve our objective, we implemented five different techniques from the literature and conducted an experiment using the defects4j dataset, which contains 395 real faults from six real-world open-source Java programs. Findings indicate that running the most dissimilar test cases early in the process is largely more effective than random permutation (Vargha-Delaney A [VDA]: 0.76-0.99 observed using normalized compression distance). No technique was found to be superior with respect to the effectiveness. Locality-sensitive hashing was, to a small extent, less effective than other SBTP techniques (VDA: 0.38 observed in comparison to normalized compression distance), but its speed largely outperformed the other techniques (i.e., it was approximately 5-111 times faster). Our results bring to mind the well-known adage, "don't put all your eggs in one basket". To effectively consume a limited testing budget, one should spread it evenly across different parts of the system by running the most dissimilar test cases early in the testing process.