Efficient token based clone detection with flexible tokenization

Abstract

Code clones are similar code fragments that occur at multiple locations in a software system. Detection of code clones provides useful information for maintenance, reengineering, program understanding and reuse. Several techniques have been proposed to detect code clones. These techniques differ in the code representation used for analysis of clones, ranging from plain text to parse trees and program dependence graphs. Clone detection based on lexical tokens involves minimal code transformation and gives good results, but is computationally expensive because of the large number of tokens that need to be compared. We explored string algorithms to find suitable data structures and algorithms for efficient token based clone detection and implemented them in our tool Repeated Tokens Finder (RTF). Instead of using suffix tree for string matching, we use more memory efficient suffix array. RTF incorporates a suffix array based linear time algorithm to detect string matches. It also provides a simple and customizable tokenization mechanism. Initial analysis and experiments show that our clone detection is simple, scalable, and performs better than the previous wellknown tools.

The last line effect explained

“…This vagueness is reflected in the definitions " [c]lones are segments of code that are similar according to some definition of similarity" by Baxter et al (1998) and "code clones [...] are code fragments of considerable length and significant similarity" by Basit and Jarzabek (2007). The latter definition identifies clones as long enough pieces of code that share sufficiently many traits, while the first has no such requirements.…”

Literature Survey of Clone Detection Techniques

“…Even though Baker [2,4]also used the token scheme to detect the clone but it did not use any transformation technique resulting in detection of false positives. For more flexible tokenization RTF [7] used suffix array rather than suffix tree so that unnecessary tokens can be removed so as to reduce the false detection but this technique is more complex to implement.…”

Computing regularities in strings

“…Software Engineering: Identifying approximate clones of methods or classes in large software systems is important to software maintenance [4].…”

“…If M x,u is neither LE nor RE, we say that it is nonextendible (NE). In x = abaababa, the NE repeats are M x,a = (1; 1, 3,4,6,8) and M x,aba = (3; 1, 4, 6).…”