A smart itsy bitsy spider for the Web

Abstract

As part of the ongoing Illinois Digital Library Initiative project, this research proposes an intelligent agent approach to Web searching. In this experiment, we developed two Web personal spiders based on best first search and genetic algorithm techniques, respectively. These personal spiders can dynamically take a user's selected starting homepages and search for the most closely related homepages in the Web, based on the links and keyword indexing. A graphical, dynamic, Java‐based interface was developed and is available for Web access. A system architecture for implementing such an agent‐based spider is presented, followed by detailed discussions of benchmark testing and user evaluation results. In benchmark testing, although the genetic algorithm spider did not outperform the best first search spider, we found both results to be comparable and complementary. In user evaluation, the genetic algorithm spider obtained significantly higher recall value than that of the best first search spider. However, their precision values were not statistically different. The mutation process introduced in genetic algorithm allows users to find other potential relevant homepages that cannot be explored via a conventional local search process. In addition, we found the Java‐based interface to be a necessary component for design of a truly interactive and dynamic Web agent. © 1998 John Wiley & Sons, Inc.

Focused crawling: a new approach to topic-specific Web resource discovery

“…To our knowledge the earliest example of using a query to direct a limited Web crawl is the Fish Search system [14]. Similar results are reported for the WebCrawler [11, chapter 4], Shark Search [17], and by Chen et al [10]. The focused crawler is different in using a topic taxonomy, learning from example, and using graph distillation to track topical hubs.…”

A test of genetic algorithms in relevance feedback

“…Chen and co-workers (Chen, 1995;Chen & Iyer, 1998;Chen, Chung, & Ramsey, 1998) use a GA to learn the query terms which best represent a set of documents supplied by the user (this process they call inductive query by examples), and to end up with the construction of an intelligent agent which uses this genetics to implement the feedback module. Their fitness function finds the Jaccard index for each chromosome (possible query) with the rest of the chromosomes of the population to then obtain the mean value of these similarities.…”

Genetic algorithms in relevance feedback: a second test and new contributions

“…In recent years, there have appeared numerous applications of ''genetic algorithms'' (GAs) to information retrieval (Belew, 1989;Chen, 1995;Chen, Chung, & Ramsey, 1998;Chen & Iyer, 1998;Cordo on, Moya, & Zarco, 2000Gordon, 1988aGordon, ,b, 1991Horng & Yeh, 2000;Kraft, Petry, Buckles, & Sadasivan, 1994Lo opez-Pujalte, 2000;Lo opez-Pujalte, Guerrero Bote, & Moya Anego on, 2002a,b;Martı ın-Bautista, 2000;Martı ın-Bautista, Vila, & Larsen, 1999;Raghavan & Agarwal, 1987;Raghavan & Birchard, 1979;Robertson & Willett, 1994, 1996Sanchez, 1994;Sanchez, Miyano, & Brachet, 1995;Sanchez & Pierre, 1994;Smith & Smith, 1997;Vrajitoru, 1997Vrajitoru, , 1998Yang & Korfhage, 1992, 1993, 1994. Most of these applications refer to ''relevance feedback'', as is indicated in a review of the topic (Cordo on, Moya, & Zarco, 1999).…”