Local text reuse detection

Abstract

Text reuse occurs in many different types of documents and for many different reasons. One form of reuse, duplicate or near-duplicate documents, has been a focus of researchers because of its importance in Web search. Local text reuse occurs when sentences, facts or passages, rather than whole documents, are reused and modified. Detecting this type of reuse can be the basis of new tools for text analysis. In this paper, we introduce a new approach to detecting local text reuse and compare it to other approaches. This comparison involves a study of the amount and type of reuse that occurs in real documents, including TREC newswire and blog collections.

Detecting and modeling local text reuse

“…Many duplicate detection systems for the web use p = 50 or above, which drastically reduces index sizes [6]. This downsampling technique is less effective for local as opposed to global text reuse [10] and can also hurt recall for noisily OCR'd documents, as we see below.…”

“…• We are looking for reuse of substantial amounts of text, on the order of 100 words or more, in contrast to work on detecting shorter quotations [9,10,11,2].…”

VChunkJoin: An Efficient Algorithm for Edit Similarity Joins

“…Researchers tackle this issue by omitting some of the grams when building the index. Depending on the techniques used, discarding some grams may [13], [14], [15] or may not [16], [17] miss some similar pairs.…”

Detecting Partial and Near Duplication in the Blogosphere

“…There are many different types of clipping behaviors, such as clipping the entire text, clipping a few phrases, or clipping and then correcting some errors in the text [1]. However, web surfers typically do not want to see redundant documents in search results, and a whole lot of duplicate documents make a system less efficient by consuming considerable resources [2]. For some popular applications such as spam site detection and duplicate web page removal in search engines [3], some duplicate document detection approaches have been proposed as follows.…”

“…For efficient duplicate document detection, the document fingerprint is generated based on significant words without common words [4], [5], named entities and multi-word terms [6], or shingles indicating contiguous subsequences [7]. Still, these approaches cannot detect the partial duplicates that agglomerate segments of many originals [2], [3] as presented in (b) of Fig. 1 comparable document, and mark the target document as a duplicate when some segment fingerprints in the target document are retrieved from the hash table.…”