Using interdocument similarity information in document retrieval systems

Abstract

The first part of this paper reports a comparative study of the document classifications produced by the use of the single linkage, complete linkage, group average, and Ward clustering methods. Studies of cluster membership and of the effectiveness of cluster searches support previous findings that suggest that the single linkage classifications are rather different from those produced by the other three methods. These latter methods all produce large numbers of small clusters containing just pairs of documents. This finding motivates the work reported in the second part of the paper, which considers the use of clusters consisting of a document together with that document with which it is most similar. A comparison of the use of such clusters with conventional best match searches using seven document test collections suggests that the two types of search are of comparable effectiveness, but they retrieve noticeably different sets of relevant documents.

Subject and citation indexing. Part II: The optimal, cluster-based retrieval performance of composite representations

“…The minimum value of EX and the highest performance level for a random structure is EX = .79; a performance level that compares favorably to current cluster-based retrieval results from a variety of test collections using a variety of clustering criteria (Griffiths et al, 1986).…”

“…When E = 0, P = R = 1.0, and when E = 1, P = R = 0.0. The E measure has been selected because it provides the opportunity to compare results of the current investigation with existing clusterbased retrieval results (Croft, 1980;Van Rijsbergen, 1974a;Van Rijsbergen & Croft, 197.5;Willett, 1984), especially the comprehensive results provided by Griffiths, Luckhurst, & Willett (1986).…”

Better than the real thing?

“…Indeed, there is a history of successful applications of the general nearest-neighbor approach (e.g., [9]). Within each iteration, Cluster-Audition scoring consists of two phases.…”

Respect my authority!

“…Since this set is query-dependent, at least some of the clustering process must occur at retrieval time, mandating the use of extremely efficient algorithms [6,37]. The approach we adopt is to use overlapping nearest-neighbor clusters, which have formed the basis of effective retrieval algorithms in other work [12,17,19,33]: for each document d ∈ Dinit, we have the cluster {d} ∪ N bhd(d | k − 1, Dinit − {d}), where k is the cluster-size parameter.…”