Personalized spiders for web search and analysis

Abstract

Searching for useful information on the World Wide Web has become increasingly difficult. While Internet search engines have been helping people to search on the web, low recall rate and outdated indexes have become more and more problematic as the web grows. In addition, search tools usually present to the user only a list of search results, failing to provide further personalized analysis which could help users identify useful information and comprehend these results. To alleviate these problems, we propose a client-based architecture that incorporates noun phrasing and self-organizing map techniques. Two systems, namely CI Spider and Meta Spider, have been built based on this architecture. User evaluation studies have been conducted and the findings suggest that the proposed architecture can effectively facilitate web search and analysis.

Mining communities and their relationships in blogs: A study of online hate groups

“…Spiders, also known as crawlers, wanderers, or Webbots, are defined as ''software programs that traverse the World Wide Web information space by following hypertext links and retrieving Web documents by standard HTTP protocol'' (Cheong, 1996). Since the early days of the Web, spiders have been widely used to build the underlying databases of search engines (e.g., Pinkerton, 1994), to perform personal search (e.g., Chau et al, (2001)), to archive particular Web sites or even the whole Web (e.g., Kahle, 1997), or to collect Web statistics (e.g., Broder et al, 2000).…”

Searching and mining the web for personalized and specialized information

“…Intelligence Spider, or CI Spider, collects Web pages on a real-time basis from Web sites specified by the user and performs indexing and categorization analysis on them, to provide the user with a comprehensive view of the Web sites of interest (Chau et al, 2001b;Chen et al, 2002). The second tool, Meta Spider, has similar functionalities as the CI Spider, but instead of performing breadth-first search on a particular Web site, connects to different search engines on the Internet and integrates the results (Chau et al, 2001b;).…”

“…The second tool, Meta Spider, has similar functionalities as the CI Spider, but instead of performing breadth-first search on a particular Web site, connects to different search engines on the Internet and integrates the results (Chau et al, 2001b;). The architecture of CI Spider and Meta Spider is shown in …”

“…Chapters 3 to 5 are devoted to personalized search agents. Chapter 3 discusses the design, implementation, and evaluation of two personalized search agents called CI Spider and Meta Spider (Chau et al, 2001b;Chen et al, 2002). These two agents combine Web searching, natural language processing, and text clustering techniques to provide advanced searching for Web users.…”

Design and evaluation of a multi-agent collaborative Web mining system

“…Also, more computation, time, and memory are often required. These limitations may be severe, especially for Web-based search engines that have to handle thousands to millions of search queries per day [6].…”