A SOM-Based Document Clustering Using Frequent Max Substrings for Non-Segmented Texts

Chumwatana, Todsanai; Wong, Kok Wai; Xie, Hong

doi:10.4236/jilsa.2010.23015

Cited by 9 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PCA (Principal Component Analysis) and LSI (Latent Semantic Indexing) are two widely used dimension reduction techniques used in literature [13,12,17,21] to reduce the corpus size without impacting the results significantly. Another way to look at tokenization of documents is using Frequent max substring technique [5,3,19] to improve the efficiency of Information Retrieval. This is especially effective in certain Asian languages such as Chinese, Japanese, Korean and Thai.…”

Section: Literature Reviewmentioning

confidence: 99%

Document Clustering using Self-Organizing Maps

Rafi¹,

Waqar²,

Ajaz³

et al. 2017

mendel

View full text Add to dashboard Cite

Cluster analysis of textual documents is a common technique for better ltering, navigation, under-standing and comprehension of the large document collection. Document clustering is an autonomous methodthat separate out large heterogeneous document collection into smaller more homogeneous sub-collections calledclusters. Self-organizing maps (SOM) is a type of arti cial neural network (ANN) that can be used to performautonomous self-organization of high dimension feature space into low-dimensional projections called maps. Itis considered a good method to perform clustering as both requires unsupervised processing. In this paper, weproposed a SOM using multi-layer, multi-feature to cluster documents. The paper implements a SOM usingfour layers containing lexical terms, phrases and sequences in bottom layers respectively and combining all atthe top layers. The documents are processed to extract these features to feed the SOM. The internal weightsand interconnections between these layers features(neurons) automatically settle through iterations with a smalllearning rate to discover the actual clusters. We have performed extensive set of experiments on standard textmining datasets like: NEWS20, Reuters and WebKB with evaluation measures F-Measure and Purity. Theevaluation gives encouraging results and outperforms some of the existing approaches. We conclude that SOMwith multi-features (lexical terms, phrases and sequences) and multi-layers can be very e ective in producinghigh quality clusters on large document collections.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Document Clustering using Self-Organizing Maps

Rafi¹,

Waqar²,

Ajaz³

et al. 2017

mendel

View full text Add to dashboard Cite

show abstract

“…Similarity indices have been used in various domains for a long time: e.g. in clustering ecological species (Jaccard, 1908), in plant genetics (Meyeri et al, 2004) or in documents clustering (Chumwatana et al, 2010). Several similarity indices can be used to measure the agreement between two partitions of the same data -P K and P Q with K and Q groups, respectively.…”

Section: Indices Of Paired Agreement Between Partitionsmentioning

confidence: 99%

Paired Indices for Clustering Evaluation - Correction for Agreement by Chance

Amorim

Cardoso

2014

Proceedings of the 16th International Conference on Enterprise Information Systems

View full text Add to dashboard Cite

In the present paper we focus on the performance of clustering algorithms using indices of paired agreement to measure the accordance between clusters and an a priori known structure. We specifically propose a method to correct all indices considered for agreement by chance-the adjusted indices are meant to provide a realistic measure of clustering performance. The proposed method enables the correction of virtually any index-overcoming previous limitations known in the literature-and provides very precise results. We use simulated datasets under diverse scenarios and discuss the pertinence of our proposal which is particularly relevant when poorly separated clusters are considered. Finally we compare the performance of EM and K-Means algorithms, within each of the simulated scenarios and generally conclude that EM generally yields best results.

show abstract

“…As far as we know, this paper is the first one that gives a quantitative comparison of bag of maximal substrings representation with bag of words representation in document clustering. While Chumwatana et al conduct a similar experiment with respect to Thai documents (Chumwatana et al, 2010), they fail to give reliable evaluation, because their datasets consist of only tens of documents. Further, they do not compare bag of maximal substrings representation with bag of words representation.…”

Section: Introductionmentioning

confidence: 99%

DOCUMENTS AS A BAG OF MAXIMAL SUBSTRINGS - An Unsupervised Feature Extraction for Document Clustering

Masada

Shibata

Oguri

2011

Proceedings of the 13th International Conference on Enterprise Information Systems

View full text Add to dashboard Cite

This paper provides experimental results showing how we can use maximal substrings as elementary features in document clustering. We extract maximal substrings, i.e., the substrings each giving a smaller number of occurrences even after adding only one character at its head or tail, from the given document set and represent each document as a bag of maximal substrings after reducing the variety of maximal substrings by a simple frequency-based selection. This extraction can be done in an unsupervised manner. Our experiment aims to compare bag of maximal substrings representation with bag of words representation in document clustering. For clustering documents, we utilize Dirichlet compound multinomials, a Bayesian version of multinomial mixtures, and measure the results by F-score. Our experiment showed that maximal substrings were as effective as words extracted by a dictionary-based morphological analysis for Korean documents. For Chinese documents, maximal substrings were not so effective as words extracted by a supervised segmentation based on conditional random fields. However, one fourth of the clustering results given by bag of maximal substrings representation achieved F-scores better than the mean F-score given by bag of words representation. It can be said that the use of maximal substrings achieved an acceptable performance in document clustering.

show abstract

A SOM-Based Document Clustering Using Frequent Max Substrings for Non-Segmented Texts

Cited by 9 publications

References 20 publications

Document Clustering using Self-Organizing Maps

Document Clustering using Self-Organizing Maps

Paired Indices for Clustering Evaluation - Correction for Agreement by Chance

DOCUMENTS AS A BAG OF MAXIMAL SUBSTRINGS - An Unsupervised Feature Extraction for Document Clustering

Contact Info

Product

Resources

About