TF-ICF: A New Term Weighting Scheme for Clustering Dynamic Data Streams

Reed, Joel W.; Jiao, Yu; Potok, Thomas E.; Klump, Brian; Elmore, Mark; Hurson, Ali R.

doi:10.1109/icmla.2006.50

Cited by 106 publications

(61 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We then stemmed the document content using a Porter Stemming algorithm [16]. Finally, we generated a term frequency list using TF-ICF [7] and normalized these frequencies for direct document comparison.…”

Section: Documentsmentioning

confidence: 99%

Flocking-based Document Clustering on the Graphics Processing Unit

Charles

Potok

Patton

et al. 2008

Nature Inspired Cooperative Strategies for Optimization (NICSO 2007)

View full text Add to dashboard Cite

Analyzing and grouping documents by content is a complex problem. One explored method of solving this problem borrows from nature, imitating the fl ocking behavior of birds. Each bird represents a single document and fl ies toward other documents that are similar to it. One limitation of this method of document clustering is its complexity O(n 2 ). As the number of documents grows, it becomes increasingly diffi cult to receive results in a reasonable amount of time. However, fl ocking behavior, along with most naturally inspired algorithms such as ant colony optimization and particle swarm optimization, are highly parallel and have experienced improved performance on expensive cluster computers. In the last few years, the graphics processing unit (GPU) has received attention for its ability to solve highly-parallel and semi-parallel problems much faster than the traditional sequential processor. Some applications see a huge increase in performance on this new platform. The cost of these high-performance devices is also marginal when compared with the price of cluster machines. In this paper, we have conducted research to exploit this architecture and apply its strengths to the document fl ocking problem. Our results highlight the potential benefi t the GPU brings to all naturally inspired algorithms. Using the CUDA platform from NVIDIA ® , we developed a document fl ocking implementation to be run on the NVIDIA ® GEFORCE 8800. Additionally, we developed a similar but sequential implementation of the same algorithm to be run on a desktop CPU. We tested the performance of each on groups of news articles ranging in size from 200 to 3,000 documents. The results of these tests were very signifi cant. Performance gains ranged from three to nearly fi ve times improvement of the GPU over the CPU implementation. This dramatic improvement in runtime makes the GPU a potentially revolutionary platform for document clustering algorithms.

show abstract

Section: Documentsmentioning

confidence: 99%

Flocking-based Document Clustering on the Graphics Processing Unit

Charles

Potok

Patton

et al. 2008

Nature Inspired Cooperative Strategies for Optimization (NICSO 2007)

View full text Add to dashboard Cite

show abstract

“…TFIDF ise en çok kullanılan geleneksel yöntemdir. Ayrıca Okapi BM25 [7] ile birlikte TFIDF varyasyonu olan çeşitli yöntemler de bulunmaktadır [8], [9]. Geleneksel yöntemleri yerel ve global kapsamlı olarak kategorize etmek de mümkündür.…”

Section: İlgili çAlışmalarunclassified

“…RelDF (Relative Document Frequency) doküman filtreleme için önerilen ve belirli bir konudaki terimlerin ilgili dokümanlarda daha fazla gözlenmesi gerektiğini varsayan bir yöntemdir [39]. TFICF (Term Frequency-Inverse Corpus Frequency) yöntemi terimin global frekansından bağımsızdır ve dinamik doküman kümeleme için önerilmiştir [9]. (1 + )…”

Section: Yeni Yaklaşımlarunclassified

The impact of term weighting method on Twitter sentiment analysis

Çoban¹,

Özyer²

2018

Pamukkale J Eng Sci

View full text Add to dashboard Cite

“…From the literature on non-learning statistical TWS, we found that most of the TWS proposed by researchers are a variation of the TF-IDF weighting scheme (Reed et al, 2006;Salton and Buckley, 1988;Sparck Jones, 1988).…”

Section: Traditional Twsmentioning

confidence: 99%

“…The above and other TWS in the literature (Reed et al, 2006;Greengrass, 2000;McGill, 1979) use some of the document collection characteristics, such as the total numbers of documents in the collection and the document term frequency (number of documents in the document collection that contain this term). In real-world IR systems, these characteristics should be considered as changing over time because nowadays document collections are mostly dynamic instead of static.…”

Section: ) Pivoted Document Length Normalization-idf (Ltu)mentioning

confidence: 99%

Term frequency with average term occurrences for textual information retrieval

Ibrahim

Landa-Silva

2015

Soft Comput

View full text Add to dashboard Cite

Ibrahim, O. and Landa-Silva, Dario (2016) Term frequency with average term occurrences for textual information retrieval. Soft Computing, 20 (8 A note on versions:The version presented here may differ from the published version or from the version of record. If you wish to cite this item you are advised to consult the publisher's version. Please see the repository url above for details on accessing the published version and note that access may require a subscription.For more information, please contact eprints@nottingham.ac.uk Abstract In the context of Information Retrieval (IR) from text documents, the term-weighting scheme (TWS) is a key component of the matching mechanism when using the vector space model (VSM). In this paper we propose a new TWS that is based on computing the average term occurrences of terms in documents and it also uses a discriminative approach based on the document centroid vector to remove less significant weights from the documents. We call our approach Term Frequency With Average Term Occurrence (TF-ATO). An analysis of commonly used document collections shows that test collections are not fully judged as achieving that is expensive and may be infeasible for large collections. A document collection being fully judged means that every document in the collection acts as a relevant document to a specific query or a group of queries. The discriminative approach used in our proposed approach is a heuristic method for improving the IR effectiveness and performance, and it has the advantage of not requiring previous knowledge about relevance judgements. We compare the performance of the proposed TF-ATO to the well-known TF-IDF approach and show that using TF-ATO results in better effectiveness in both static and dynamic document collections. In addition, this paper investigates the impact that stop-words removal and our discriminative approach have on TF-IDF and TF-ATO. The results show that both, stopwords removal and the discriminative approach, have a positive effect on both term-weighting schemes. More importantly, it is shown that using the proposed discriminative approach is beneficial for improving IR effectiveness and performance with no information in the relevance judgement for the collection.

show abstract

TF-ICF: A New Term Weighting Scheme for Clustering Dynamic Data Streams

Cited by 106 publications

References 14 publications

Flocking-based Document Clustering on the Graphics Processing Unit

Flocking-based Document Clustering on the Graphics Processing Unit

The impact of term weighting method on Twitter sentiment analysis

Term frequency with average term occurrences for textual information retrieval

Contact Info

Product

Resources

About