Effective collection metasearch in a hierarchical environment

Abstract

We compare standard global IR searching with user-centric localized techniques to address the database selection problem. We conduct a series of experiments to compare the retrieval effectiveness of three separate search modes applied to a hierarchically structured data environment of textual database representations. The data environment is represented as a tree-like directory containing over 15,000 unique databases and over 100,000 total leaf nodes. Our search modes consist of varying degrees of browse and search, from a global search at the root node to a refined search at a subnode using dynamically-calculated inverse document frequencies (idf s) to score candidate databases for probable relevance. Our findings indicate that a browse and search approach that relies upon localized searching from sub-nodes is capable of producing the most effective results. INTRODUCTIONThe continued growth of online databases has made the work of finding the most relevant collections increasingly difficult. Until recently, the ability to execute a 'search' in a database directory as well as 'drill down' into its hierarchical structure have largely been regarded as separate activities. If either approach does not provide desired results, large numbers of users exit online systems with unmet information needs. Yahoo! and the Open Directory Project are exceptions that permit integrated browse and search. Research has begun to explore categorization and retrieval in such environments [4]. We hypothesized that if users could first browse to a potentially relevant sub-node in a large directory, results from a search in the sub-directory would be more precise than results from a search in the entire directory. To test the effectiveness of browse plus search functionality, we designed and conducted a series of experiments on three search modes. Using the same set of real user queries, these search modes included: (1) a global search of the directory from the root node, (2) a localized search of the relevant sub-directories using global idfs, and (3) a localized search of the relevant sub-directories using the appropriate dynamically-calculated local idfs.

An Overview of Aggregating Vertical Results into Web Search Results

“…The results returned by the selected resources are integrated and merged into a single list and presented to the user. Some of the examples of federated search systems are Westlaw, providing federated search for legal professionals covering thousands of databases [6,44,45,46] and Fedstats, an online portal for statistical information. In Federated search cooperative environment a central section or the broker has the extensive information about the contents of each collection, collection sizes and lexicon statistics whereas in uncooperative environment the broker has to use query-based sampling [7] to know the lexicon statistics for each collection [6].…”