Refinement of TF-IDF schemes for web pages using their hyperlinked neighboring pages

Abstract

In IR (information retrieval) systems based on the vector space model, the TF-IDF scheme is widely used to characterize documents. However, in the case of documents with hyperlink structures such as Web pages, it is necessary to develop a technique for representing the contents of Web pages more accurately by exploiting the contents of their hyperlinked neighboring pages. In this paper, we first propose several approaches to refining the TF-IDF scheme for a target Web page by using the contents of its hyperlinked neighboring pages, and then compare the retrieval accuracy of our proposed approaches. Experimental results show that, generally, more accurate feature vectors of a target Web page can be generated in the case of utilizing the contents of its hyperlinked neighboring pages at levels up to second in the backward direction from the target page.

Retracted: Extracting User Interests on Facebook

“…Among researches based on term frequency weighting, Sugiyama et al have proposed a method to accurately express the contents of web documents by segmenting Term Frequency-Inverse Document Frequency (TF-IDF) for pages connected with hyperlink. They have concluded, by forming feature vector, the curved shape of recall and precision [11]. Zhang et al experimented and searched the effective method for text classification through TF-IDF and Latent Semantic Indexing (LSI) (http://en .wikipedia.org/wiki/Latent semantic indexing).…”

Scholarly paper recommendation via user's recent research interests

“…A key aspect of our approach is that we include contextual evidence about each paper in the form of its neighboring papers: the papers that cite the target paper (we term these citation papers) and papers referenced by the target paper (reference papers). While the use of contextual information from neighbors is not new -it has also been successfully applied to the problems of Web page representation [33], Web page classification [25], spam detection [6] -to the best of our knowledge, it has not been utilized in scholarly paper recommendation. An additional desirable property of our approach is that it is domain-independent.…”

Correlation of Term Count and Document Frequency for Google N-Grams

“…Sugiyama et al [18] used the TREC-9 Web Track dataset [7] to estimate IDF values for web pages. The novel part of their work was to also include the content of hyperlinked neighboring pages in the TF-IDF calculation of a centroid page.…”

“…It is often used for term weighting in the vector space model as described by Salten et al [15]. It further can be used to generate lexical signatures (LSs) of web pages as shown in [14,13,6,10,18].…”

“…These corpora are generally considered a representative sample for the Internet [16] but have also been found to be somewhat dated [1]. The TREC Web Track is probably the most common corpus and has, for example, been used in [18] for IDF estimation. The British National Corpus (BNC) [11], as another example, has been used in [17].…”