Towards developer support for merging forked test cases

To Share, or Not to Share: Exploring Test-Case Reusability in Fork Ecosystems

“…Unfortunately, comparative benchmarks and empirical studies in practice indicate that test generators and automatic program repair are not perfect for generating meaningful test cases or propagating bug fixes [4], [15], [65]. To improve this situation, we have suggested in recent works to involve developers more directly into the processes of deriving test cases [52], [53], [64]. Unfortunately, existing techniques are limited to specific testing techniques (e.g., performance testing), and only recommend modifications to developers [3].…”

“…Code-Test Co-Evolution. Co-evolution in fork ecosystems is a well-known problem, and, identically, source code and the test cases covering it may also co-evolve [34], [64], [73], [82]. To resolve co-evolution problems, several techniques and refactorings have been proposed that aim to identify whether a source-code change also requires updates to test cases-and to ideally update the test cases automatically [39], [56], [76].…”

“…Of course, sharing quality improvements is only relevant for actively maintained forks rather than for those that are abandoned after a pull request to the mainline project is accepted. To improve the quality of the projects within a fork ecosystem, we advocate the propagation of quality improvements, such as bug fixes and newly created or evolved test cases, among forks [34], [52], [64]. While change-propagation and program repair techniques have been proposed [19], there is no specific support to reuse test cases within fork ecosystems, not even studies of the potential of test-case reuse-our focus in this paper.…”

“…Reusing test cases has been recognized as a viable avenue to improve the quality of software. Various studies and techniques have been presented that focus on specific kinds of applications and application parts, such as on UI testing in Android apps and in web applications [26], [42], [57], [84], or on highly configurable systems [16], [18], [34], [64]. However, none of these works on test-case reuse considers ecosystems of forked projects, where the positive effect may be highest.…”

“…If the tested functionality also exists in other forks, it may be highly important for the developers to propagate the test case to assess whether their fork is threatened (i.e., vulnerable to SQL injections). Beyond cases of simple equivalence, it is challenging to assess whether a test case is applicable [64]. Among others, the tested functionality may have been modified, or the test may be concerned with a feature interaction that does not exist in the other fork.…”