Identifying overly strong conditions in refactoring implementations

Journal of Systems and Software

Gheyi

Murphy-Hill

et al. 2013

Self Cite

Some approaches have been used to investigate evidence on how developers refactor their code, whether refactorings activities may decrease the number of bugs, or improve developers' productivity. However, there are some contradicting evidence in previous studies. For instance, some investigations found evidence that if the number of refactoring changes increases in the preceding time period the number of defects decreases, different from other studies. They have used different approaches to evaluate refactoring activities. Some of them identify committed behavior-preserving transformations in software repositories by using manual analysis, commit messages, or dynamic analysis. Others focus on identifying which refactorings are applied between two programs by using manual inspection or static analysis. In this work, we compare three different approaches based on manual analysis, commit message (Ratzinger's approach) and dynamic analysis (SafeRefactor's approach) to detect whether a pair of versions determines a refactoring, in terms of behavioral preservation. Additionally, we compare two approaches (manual analysis and Ref-Finder) to identify which refactorings are performed in each pair of versions. We perform both comparisons by evaluating their accuracy, precision, and recall in a randomly selected sample of 40 pairs of versions of JHotDraw, and 20 pairs of versions of Apache Common Collections. While the manual analysis presents the * Corresponding author Email addresses: gsoares@dsc.ufcg.edu.br (Gustavo Soares), rohit@dsc.ufcg.edu.br (Rohit Gheyi), emerson@csc.ncsu.edu (Emerson Murphy-Hill), bijohnso@csc.ncsu.edu (Brittany Johnson) Preprint submitted to Journal of Systems and Software November 6, 2012 best results in both comparisons, it is not as scalable as the automated approaches. Ratzinger's approach is simple and fast, but presents a low recall; differently, SafeRefactor is able to detect most applied refactorings, although limitations in its test generation backend results for some kinds of subjects in low precision values. Ref-Finder presented a low precision and recall in our evaluation.

Section: Refactoring Implementationmentioning

confidence: 99%

Comparing approaches to analyze refactoring activity on software repositories

Journal of Systems and Software

Gheyi

Murphy-Hill

et al. 2013

Self Cite

“…Our technique also identified 17 overly strong preconditions in Eclipse JDT [8]. We have not received feedback about them from Eclipse developers prior to this submission.…”

Section: Research Statusmentioning

confidence: 82%

“…We propose an oracle to detect overly strong preconditions based on differential testing [8]. When the refactoring implementation under test rejects a transformation, we apply the same transformation by using one or more other refactoring implementations.…”

Section: Approachmentioning

confidence: 99%

Automated behavioral testing of refactoring engines

Proceedings of the 3rd Annual Conference on Systems, Programming, and Applications: Software for Humanity

2012

Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions. However, defining and implementing preconditions is a complex task. As a result, even mainstream refactoring engines contain critical bugs. We propose an automated approach for testing of Java refactoring engines based on program generation.

“…Our approach detects bugs related to transformation failures or weak preconditions. Although it is also relevant to find overly strong preconditions in refactoring engines-which may produce potentially correct but undesirably restrictive refactorings-these bugs are not the focus of our technique (nevertheless, JDolly and SAFEREFACTOR have already been evaluated as a detection strategy for overly strong preconditions [26], as detailed in Section 6.2). From our catalog, most bugs were accepted (86).…”

Section: Resultsmentioning

confidence: 99%

Automated Behavioral Testing of Refactoring Engines

IIEEE Trans. Software Eng.

Gheyi

Massoni

2013

Self Cite

Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions determined for each refactoring transformation. However, to formally define these preconditions and transfer them to program checks is a rather complex task. In practice, refactoring engine developers commonly implement refactorings in an ad hoc manner since no guidelines are available for evaluating the correctness of refactoring implementations. As a result, even mainstream refactoring engines contain critical bugs. We present a technique to test Java refactoring engines. It automates test input generation by using a Java program generator that exhaustively generates programs for a given scope of Java declarations. The refactoring under test is applied to each generated program. The technique uses SAFEREFACTOR, a tool for detecting behavioral changes, as an oracle to evaluate the correctness of these transformations. Finally, the technique classifies the failing transformations by the kind of behavioral change or compilation error introduced by them. We have evaluated this technique by testing 29 refactorings in Eclipse JDT, NetBeans, and the JastAdd Refactoring Tools. We analyzed 153,444 transformations, and identified 57 bugs related to compilation errors, and 63 bugs related to behavioral changes.