Pairwise document similarity measure based on present term set

Abstract

IntroductionIn text mining, a similarity (or distance) measure is the quintessential way to calculate the similarity between two text documents, and is widely used in various Machine Learning (ML) methods, including clustering and classification. ML methods help learn from enormous collections, known as big data [1,2]. In big data, which includes masses of unstructured data, Information Retrieval (IR) is the dominant form of information access [3]. Among ML methods, classification and clustering help discover patterns and correlations and extract information from large-scale collections [1]. These two techniques also offer benefits to different IR applications. For example, document clustering can be applied to the document collection to improve search speed, precision, and recall or to the search results to provide more effective information presentation to user [3]. Document classification is also used in vertical search engines [4] and sentiment detection [5].In large-scale collections, one of the challenging issues is to identify documents with high similarity values, known as near-duplicate documents (or near-duplicates) [6][7][8].Integration of heterogeneous collections, storing multiple copies of the same document, and plagiarism are the main causes for the existence of near-duplicates. These documents increase processing overheads and storage. Detecting and filtering near-duplicates AbstractMeasuring pairwise document similarity is an essential operation in various text mining tasks. Most of the similarity measures judge the similarity between two documents based on the term weights and the information content that two documents share in common. However, they are insufficient when there exist several documents with an identical degree of similarity to a particular document. This paper introduces a novel text document similarity measure based on the term weights and the number of terms appeared in at least one of the two documents. The effectiveness of our measure is evaluated on two real-world document collections for a variety of text mining tasks, such as text document classification, clustering, and near-duplicates detection. The performance of our measure is compared with that of some popular measures. The experimental results showed that our proposed similarity measure yields more accurate results.

Boolean logic algebra driven similarity measure for text based applications

“…In our work, Euclidean was seen more efficient when run on web-kb and Reuters, though, That is due to the Manhattan is being contingent upon the rotation of the coordinate system, leading to its being disadvantageous for both document classification and clustering tasks ( Kumar, Chhabra & Kumar, 2014 ). Meanwhile, Jaccard, Ex-Jaccard, Kullback–Leibler divergences (KLD), and Bhattacharya coefficient were all used for the several tasks of ML and IR including text clustering and classification ( Amer & Abdalla, 2020 ; Tanimoto, 1957 ; Tata & Patel, 2007 ; Oghbaie & Mohammadi Zanjireh, 2018 , François, Wertz & Verieysen, 2007 ; D’hondt et al, 2010 ; Li et al, 2017 ; Kullback & Leibler, 1951 ).…”

“…To tackle such challenges, in IR literature, a dozen of works have introduced several effective similarity measures for text clustering and classification ( Amer & Abdalla, 2020 ; Oghbaie & Mohammadi Zanjireh, 2018 ; Sohangir & Wang, 2017 ; Lin, Jiang & Lee, 2014 ; Shahmirzadi, Lugowski & Younge, 2019 ; Ke, 2017 ; White & Jose, 2004 ; Lakshmi & Baskar, 2021 ; Kotte, Rajavelu & Rajsingh, 2020 ; Thompson, Panchev & Oakes, 2015 ). However, except for Amer & Abdalla (2020) , these studies proposed similarity measures without providing sufficient insights into run-time efficiency.…”

“…In other words, these studies might introduce effective measures yet time-inefficient. Moreover, these measures, which are shown effective ( Amer & Abdalla, 2020 ; Oghbaie & Mohammadi Zanjireh, 2018 ; Sohangir & Wang, 2017 ; Lin, Jiang & Lee, 2014 ; Shahmirzadi, Lugowski & Younge, 2019 ; Lakshmi & Baskar, 2021 ; Robertson, 2004 ) suffer from design complexity. Motivated by this, this work comes with the ultimate aim of finding an influential solution to the efficiency as well as the design complexity of those similarity measures.…”

An effective assessment of cluster tendency through sampling based multi-viewpoints visual method

“…The VAT (Bezdek and Hathaway 2002 ) and cVAT (Prasad et al 2019b , a ) use the Euclidean and cosine measures while evaluating the clustering tendency of tweets data. Cosine accurately measures the similarity features for the text documents (Oghbaie and Zanjireh 2018 ) than Euclidean. Thus, for the assessment of clusters, cVAT performs better than VAT.…”

Assessment of Twitter Data Clusters with Cosine-Based Validation Metrics Using Hybrid Topic Models

“…The most commonly used measure is Euclidean distance, which shows poor results in high dimensionality document clustering. In this paper, a novel cosine based internal and external validity metrics proposed for internally evaluating the results of a document clustering by considering into account the peculiarity of textual data [18], the closeness between documents [19], considering the lexical similarity [20], and also considered cluster classification metrics in the classification of elements in the cluster are well classified or not. Experimentally evaluated the effectiveness of proposed cluster validity metrics with benchmark and Twitter-based datasets.…”