Internet-scale Real-time Code Clone Search Via Multi-level Indexing

Keivanloo, Iman; Rilling, Juergen; Charland, Philippe

doi:10.1109/wcre.2011.13

Cited by 32 publications

(31 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Koshke [20] used suffix trees to scale license violation detection between a subject system and a large inter-project repository. Keivanloo et al [19] use an index approach to scale clone search to large inter-project repositories. Chen et al [6] implement a technique for detecting cloned Android applications across large application markets.…”

Section: Related Workmentioning

confidence: 99%

“…clone detection can be used for mining library candidates [15], detecting similar mobile applications [6], license violation detection [20,9], reverse engineering product lines [12,9], finding the provenance of a component [8], and code search [19,18]. Large-scale clone detection allows researchers to study cloning in large software ecosystems (e.g., Debian), or study cloning in open-source development communities (e.g., GitHub).…”

Section: Introductionmentioning

confidence: 99%

“…A number of tools have been proposed to achieve a few specific applications of large-scale clone detection [20,19,6]. These tools make some assumptions regarding the requirements of their target domain that help with scalability.…”

Section: Introductionmentioning

confidence: 99%

“…The novel scalable algorithms [14,15] do not support Type-3 near-miss clones, where minor to significant editing activities might have taken place in the copy/pasted fragments, and therefore miss a large portion of the clones, since there are more Type-3 in the repositories than other types [24,26,30]. Type-3 clones can be the most needed in large-scale clone detection applications [24,19,6]. While input partitioning can scale existing non-scalable Type-3 detectors, this significantly increases the cumulative runtime, and requires distribution over a large cluster of machines to achieve scalability in absolute runtime [21,31].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

SourcererCC

Sajnani

Saini

Svajlenko

et al. 2016

Proceedings of the 38th International Conference on Software Engineering

337

View full text Add to dashboard Cite

Despite a decade of active research, there is a marked lack in clone detectors that scale to very large repositories of source code, in particular for detecting near-miss clones where significant editing activities may take place in the cloned code. We present SourcererCC, a token-based clone detector that targets three clone types, and exploits an index to achieve scalability to large inter-project repositories using a standard workstation. SourcererCC uses an optimized invertedindex to quickly query the potential clones of a given code block. Filtering heuristics based on token ordering are used to significantly reduce the size of the index, the number of code-block comparisons needed to detect the clones, as well as the number of required token-comparisons needed to judge a potential clone.We evaluate the scalability, execution time, recall and precision of SourcererCC, and compare it to four publicly available and state-of-the-art tools. To measure recall, we use two recent benchmarks, (1) a large benchmark of real clones, BigCloneBench, and (2) a Mutation/Injection-based framework of thousands of fine-grained artificial clones. We find SourcererCC has both high recall and precision, and is able to scale to a large inter-project repository (250MLOC) using a standard workstation.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

SourcererCC

Sajnani

Saini

Svajlenko

et al. 2016

Proceedings of the 38th International Conference on Software Engineering

337

View full text Add to dashboard Cite

show abstract

“…There are various kinds of clone detection techniques, based on similarities among strings, tokens, syntax trees, dependency graphs, and even program memory states and functionalities [9], [10], [29], [32], [33], [37], [40], [49], [56], [58]. With the evolving software, there are also studies aiming to detect clones incrementally and improve the clone detection efficiency when programs change [18], [20], [43].…”

Section: Studies On Code Clonesmentioning

confidence: 99%

Understanding Widespread Changes: A Taxonomic Study

Jiang

2013

2013 17th European Conference on Software Maintenance and Reengineering

View full text Add to dashboard Cite

Abstract-Many active research studies in software engineering, such as detection of recurring bug fixes, detection of copyand-paste bugs, and automated program transformation tools, are motivated by the assumption that many code changes (e.g., changing an identifier name) in software systems are widespread to many locations and are similar to one another. However, there is no study so far that actually analyzes widespread changes in software systems. Understanding the nature of widespread changes could empirically support the assumption, which provides insight to improve the research studies and related tools.Our study in this paper addresses such a need. We propose a semi-automated approach that recovers code changes involving widespread changes in software systems. We further manually analyze more than nine hundred widespread changes recovered from eight software systems and categorize them into 11 families. These widespread changes and their associated families help us understand better why these widespread changes are made.

show abstract

Large‐scale inter‐system clone detection using suffix trees and hashing

Koschke

2013

J Software Evolu Process

View full text Add to dashboard Cite

Detecting a similar code between two systems has various applications such as comparing two software variants or versions or finding potential license violations. Techniques detecting suspiciously similar code must scale in terms of resources needed to very large code corpora and need to have high precision because a human needs to inspect the results. This paper demonstrates how suffix trees can be used to obtain a scalable comparison. The evaluation is carried out for very large code corpora. Our evaluation shows that our approach is faster than index-based techniques when the analysis is run only once. If the analysis is to be conducted multiple times, creating an index pays off. We report how much code can be filtered out from the analysis using an index-based filter. In addition to that, this paper proposes a method to improve precision through user feedback. A user validates a sample of the found clone candidates. An automated data mining technique learns a decision tree on the basis of the user decisions and different code metrics. We investigate the relevance of several metrics and whether criteria learned from one application domain can be generalized to other domains.All of the aforementioned variants of clone detection are facing challenges with respect to detection quality and scalability. Detection quality requires high recall and high precision in finding the relevant code. Relevance depends on the use case. In particular, inter-system and intra-system clone detections need to deal with re-occurring similar code that is similar from a lexical or syntactical point of view, but that is not interesting for the given task. Frequent examples of such irrelevant similar code are import statement lists, array initializers, setter/getter sequences, or sequences of pure declarations or simple assignments.Another challenge is scalability. Whereas intra-system clone detection searches only within one system, inter-system clone search may face a much larger code base, often larger by orders of magnitude. Also, fragment search may face this problem, when the code is searched in very large software repositories [3,4].Several researchers have recently proposed to use an index-based code search to address scalability for the search in very large code bases [13,3,4,17,18].The index-based techniques first create an index against which code of a subject system is compared later. The purpose of the index is to identify the code that has a chance of being similar. The code filtered out by the index is not compared. The index is a first seed of a similar code fragment. This seed is then extended by merging with neighboring similar code fragments [13,3,4].Creating the index can be expensive. The idea is to invest upfront in an index that is created only once but whose cost is amortized in multiple subsequent searches.Contributions. Our conference paper introduced a way to extend traditional suffix-tree-based clone detection for inter-system clone search that scales for very large programs [19]. This approach avoids the nee...

show abstract

Internet-scale Real-time Code Clone Search Via Multi-level Indexing

Cited by 32 publications

References 8 publications

SourcererCC

SourcererCC

Understanding Widespread Changes: A Taxonomic Study

Large‐scale inter‐system clone detection using suffix trees and hashing

Contact Info

Product

Resources

About