PurposeThere have been many challenges in crawling deep web by search engines due to their proprietary nature or dynamic content. Distributed Information Retrieval (DIR) tries to solve these problems by providing a unified searchable interface to these databases. Since a DIR must search across many databases, selecting a specific database to search against the user query is challenging. The challenge can be solved if the past queries of the users are considered in selecting collections to search in combination with word embedding techniques. Combining these would aid the best performing collection selection method to speed up retrieval performance of DIR solutions.Design/methodology/approachThe authors propose a collection selection model based on word embedding using Word2Vec approach that learns the similarity between the current and past queries. They used the cosine and transformed cosine similarity models in computing the similarities among queries. The experiment is conducted using three standard TREC testbeds created for federated search.FindingsThe results show significant improvements over the baseline models.Originality/valueAlthough the lexical matching models for collection selection using similarity based on past queries exist, to the best our knowledge, the proposed work is the first of its kind that uses word embedding for collection selection by learning from past queries.
Federated search routes the user's search query to multiple component collections and presents a merged results list in ranked order by comparing the relevance score of each returned result. However, the heterogeneity of the component collections makes it challenging for the central broker to compare these relevance scores while fusing the results in ranked order. To address this issue, most existing approaches merged the returned results by changing the document ranks to their ranking scores or downloading the documents and computing their relevance score at query time. However, these approaches are less efficient as the former suffer from limited efficiency of results merging due to negligible overlapping documents among the component collections, and the latter is resource intensive. This research addresses this problem by proposing a new method that extracts features of both the documents and component collections from the available information provided by the collections at querying time. Next, each document and its collection features are exploited together to establish the document relevance score. The ant colony optimization is then used for information foraging to create a merged results list. The empirical results on a real-world dataset demonstrate significant improvements by the proposed approach over baseline approaches.
In federated search, the central broker simultaneously forwards the search query to multiple resources. The returned results from those resources are then merged into a single ranked list. An autonomous resource in a federated search system usually does not provide scores for the retrieved documents; even if some of them do, scores from different resources are incomparable due to the heterogeneity in many aspects of those resource involved such as retrieval models and corpus statistics. Many results merging approaches have been proposed in the literature to deal with this problem. However, to the best of our knowledge, none of them has utilised snippets. This article proposes a snippet-based weighting scheme for the query terms involved. It quantifies the importance of each query term from two angles: the frequency of the term and the part in which the term occurs inside a snippet. Three parts – which are URL, title, and description – are given different weights. Experiments are conducted with the TREC 2013 FedWeb data set. The results show that the proposed methods consistently outperform several baseline models. We also find in many instances, a further small slight performance improvement is achievable by an extra measure of weighting each of the resources involved, which can be done in the phase of resource selection.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.