A new technique for detecting similar documents based on term co-occurrence and conceptual property of the text

Abstract

The importance of detecting similar documents grows rapidly as the amount of information increases exponentially. This paper presents a new technique for identifying similar documents. It combines statistical properties of documents with Persian linguistic features. The proposed technique is mostly suited for detecting similar documents in specific fields. The proposed method is built on lexical chain of important words and based on term co-occurrence property of the text. It prevents the irrelevant documents to be identified similar due to polysemy property of the words. It also considers the order of words in identifying the similar documents. If a document consists of more than one subject, it could also be founded and similar documents according to different topics of the text could be detected. Our results shows improved performance compared to existing wordbased methods like LSI and VSM.

Optimizing Document Similarity Detection in Persian Information Retrieval

“…In this case, the document is represented as a vector of words and the frequency of each word in each document is calculated. If words are chosen as terms, then every word in the vocabulary becomes an independent dimension in a very high dimensional vector space [4,12,13]. The similarity between two documents is determined by a similarity measure such as cosine similarity [4,13] between their corresponding vectors (since cosine has the nice property that it is 1.0 for identical vectors and 0.0 for orthogonal vectors).…”

“…It is computationally inefficient due to the sparse sentence vector. The other drawback is that texts with similar meaning do not necessarily share many words [5,12].…”

“…Another important word-based method is Latent Semantic Indexing (LSI) [14] that focuses on the cooccurrence of the term, but it is inefficient both in time and accuracy, because of the lack of considering other relations between terms [12,14]. This results in creating relations between terms which are not real.…”

“…This results in creating relations between terms which are not real. It also does not take into account the order of the words [12,14].…”

Recuperación de información en Persa: revisión crítica y propuestas de mejora

“…Otro enfoque básico para el cálculo de la similitud de documentos es el método de co-ocurrencia que ya está aplicado para la lengua persa por el trabajo de Zamanifar (Zamanifar, et al, 2008). Este método tiene tres pasos principales: la identificación de tema, la interpretación de tema y la medida de similitud.…”