Agent-Based Document Expansion for Information Retrieval Based on Topic Modeling of Local Information

Abstract

With the advent of data ecosystems finding information in distributed and federated catalogs and marketplaces becomes more and more important. One of the problems in data search and search in general is the mismatch between the terminology of users and of the searched items, be it dataset metadata or web pages. The paper proposes an agent-based approach to document expansion (ADE). The idea is to represent documents with agents that exploit local information collected from user searches and relevant signals to improve the representation of the document in a search index and subsequently to improve the search performance of the system. The agents collect terms from relevant queries and perform topic modeling on these terms and publish different variants expanded with the topic terms to the search index. We find that the approach achieves good improvement in search performance and is a valuable tool because is places no burden on the information retrieval pipeline and is complementary to other document expansion and information retrieval approaches.