The optimum clustering framework: implementing the cluster hypothesis

Abstract

Document clustering offers the potential of supporting users in interactive retrieval, especially when users have problems in specifying their information need precisely. In this paper, we present a theoretic foundation for optimum document clustering. Key idea is to base cluster analysis and evalutation on a set of queries, by defining documents as being similar if they are relevant to the same queries. Three components are essential within our optimum clustering framework, OCF: (1) a set of queries, (2) a probabilistic retrieval method, and (3) a document similarity metric. After introducing an appropriate validity measure, we define optimum clustering with respect to the estimates of the relevance probability for the query-document pairs under consideration. Moreover, we show that well-known clustering methods are implicitly based on the three components, but that they use heuristic design decisions for some of them. We argue that with our framework more targeted research for developing better document clustering methods becomes possible. Experimental results demonstrate the potential of our considerations.

Ranking document clusters using markov random fields

“…Furthermore, positioning the constituent documents of these clusters at the top of the result list yields highly effective retrieval performance; specifically, much better than that of state-of-the art retrieval methods that rank documents directly [8,32,25,14,10].…”

Fine-Grained Algorithm for Improving KNN Computational Performance on Clinical Trials Text Classification

“…Table 1 and Figure 3 below are the results of research conducted by Bustos and Pertusa, who carried out the precision, recall, F1 score, and Cohen's K process [16]. Fine-grained algorithm (FGA): FGA presumes the turned form of the cluster hypothesis [23], that is, the relevant documents returned in response to a query will be inclined to be similar to one another. FGA uses a combination of loci and relevant cluster concepts to efficiently form clusters.…”

“…Since this dataset is quite large, we corrected for it using random balanced under sampling, which resulted in a reduced dataset size with 4 million labeled samples. A snippet of clinical statements and classes can be seen in Figure 2 Fine-grained algorithm (FGA): FGA presumes the turned form of the cluster hypothesis [23], that is, the relevant documents returned in response to a query will be inclined to be similar to one another. FGA uses a combination of loci and relevant cluster concepts to efficiently form clusters.…”

Identifying Covid-19 misinformation tweets and learning their spatio-temporal topic dynamics using Nonnegative Coupled Matrix Tensor Factorization

“…The relevance score of j th document in with is calculated using term frequency * inverse document frequency (tf*idf) (Fuhr et al. 2012 ) as, where is the frequency of term w in document , is the frequency of term w in the document collection , is a normalization controlling the importance of terms based on term frequency (Gormley and Tong 2015 ).…”