Link-based ranking methods have been described in the literature and applied in commercial Web search engines. However, according to recent TREC experiments, they are no better than traditional content-based methods. We conduct a different type of experiment, in which the task is to find the main entry point of a specific Web site. In our experiments, ranking based on link anchor text is twice as effective as ranking based on document content, even though both methods used the same BM25 formula. We obtained these results using two sets of 100 queries on a 18.5 million document set and another set of 100 on a 0.4 million document set. This site finding effectiveness begins to explain why many search engines have adopted link methods. It also opens a rich new area for effectiveness improvement, where traditional methods fail.

The presentation of query biased document snippets as part of results pages presented by search engines has become an expectation of search engine users. In this paper we explore the algorithms and data structures required as part of a search engine to allow efficient generation of query biased snippets. We begin by proposing and analysing a document compression method that reduces snippet generation time by 58% over a baseline using the zlib compression library. These experiments reveal that finding documents on secondary storage dominates the total cost of generating snippets, and so caching documents in RAM is essential for a fast snippet generation process. Using simulation, we examine snippet generation performance for different size RAM caches. Finally we propose and analyse document reordering and compaction, revealing a scheme that increases the number of document cache hits with only a marginal affect on snippet quality. This scheme effectively doubles the number of documents that can fit in a fixed size cache.

In recent years, TREC has broadened its scope to include many more facets of the Web searching process. In TREC-8 (1999), the Web special interest track evaluated link-based retrieval methods investigated differences between Web and traditional TREC ad hoc documents, and studied efficiency and effectiveness tradeoffs on large data sets. In addition, although neither used Web data, both the Cross-Lingual track and the Question & Answer track studied issues of considerable importance to everyday Web search.In TREC-9, the main Web track task will use a larger set of Web documents than last year and will use search topics derived from search engine logs. This task will be the closest approximation in TREC-9 to the traditional TREC Ad Hoc retrieval task.

No abstract

No abstract