A comparison of techniques for estimating IDF values to generate lexical signatures for the web

Abstract

For bounded datasets such as the TREC Web Track the computation of term frequency (TF) and inverse document frequency (IDF) is not difficult. However, since IDF cannot be directly calculated for the entire web, it must be estimated. We see a need to estimate accurate IDF values to generate TF-IDF based lexical signatures (LSs) of web pages. Future applications for generating such LSs require a real time IDF computation. Therefore we conducted a comparison study of different methods to estimate IDF values of web pages. Our objective is to investigate how accurate these estimation methods are compared to the a baseline. We use the Google N-grams as our baseline and compare it against two IDF estimation techniques which are based on: 1) a "local universe" consisting of textual content and the according document frequencies from copies of URLs from the Internet Archive and 2) "screen scraping", a technique to query the Google web interface for document frequencies. We found a term overlap of 70 to 80% between the results of the two methods and the baseline. We further discovered a great agreement in rank correlation of TF-IDF ranked terms between our methods. Kendall τ is approximately 0.8 and the M-Score (penalizing discordances in higher ranks) is even higher, it peaks at well above 0.9. These preliminary results lead us to the conclusion that both methods are appropriate for creating accurate IDF values for web pages.

Is this a good title?

“…We used the "screen scraping" approach and queried the Yahoo BOSS API to determine document frequency values for all terms and used numbers published by the website www.worldwidewebsize.com in October 2009 to estimate the size of the Yahoo index. We have shown in [23] that this approach is feasible and performs very well compared to other methods.…”

Evaluating methods to rediscover missing web pages from the web infrastructure

“…As a common approach researchers use search engines to estimate the document frequency of a term ( [13,19,31,39]). Even though the obtained values are only estimates ( [1]) our earlier work [20] has shown that this approach actually works well compared to using a modern text corpus.…”

Revisiting Lexical Signatures to (Re-)Discover Web Pages

“…Computing IDF values requires knowledge about: 1) the size of the entire corpus (the Internet) in terms of number of documents and 2) the number of documents the term appears in. A related study (6) investigates different techniques for creating IDF values for web pages.…”