A distributed full-text top-k document dissemination system in distributed hash tables

Abstract

Recent years witnessed the explosive growth of 'live' web content in the World Wide Web like Weblogs, RSS feeds, and real-time news, etc. The popular usage of RSS feeds/readers enables end users to subscribe for favorite contents via input RSS URLs. However, the RSS feeds/readers architecture suffers from (i) the high bandwidth consumption issue, and (ii) limited filtering semantics. In this paper, we proposed a stateful full text dissemination scheme over structured P2Ps to address both issues. Specifically, for the semantic side, end users are allowed to subscribe for favorite contents via input keywords; for the network bandwidth side, the cooperative content polling, filtering and disseminating via DHT-based P2P overlay networks save the network bandwidth consumption. Our contributions include the novel techniques to (i) reduce the unit-publishing cost by pruning irreverent documents during the forwarding path towards destinations, and (ii) reduce the publication amount by selecting a very small number of meaningful terms. Based on real data sets, our experimental results show that the proposed scheme can significantly reduce the publishing cost with low maintenance overhead and a high document quality.

Distributed top-k full-text content dissemination

“…Thought the previous work [28] and this paper share the similar top-k data model problem, there exist significant differences as follows. (i) Assumption: Rao and Chen [28] target on distribute hash table-based P2P networks, where all nodes in P2P networks are equally responsible for content publishing, subscription, and forwarding.…”

“…(i) Assumption: Rao and Chen [28] target on distribute hash table-based P2P networks, where all nodes in P2P networks are equally responsible for content publishing, subscription, and forwarding. Instead, in our problem settings, the ISes are dedicate servers which are typically provided by vendors of content distribution networks (e.g., Akami and Google); in particular, given the assumption I < S and I < U, the ISes bridge a large number of content sources and subscription agents, such that the content polling and forwarding are shared and the overall network traffic is reduced.…”

“…Instead, in our problem settings, the ISes are dedicate servers which are typically provided by vendors of content distribution networks (e.g., Akami and Google); in particular, given the assumption I < S and I < U, the ISes bridge a large number of content sources and subscription agents, such that the content polling and forwarding are shared and the overall network traffic is reduced. (ii) Data Location: in [28], the home node of t i acts as a rendezvous node to forward d to those filters containing t i . Rao and Chen [28] focuses on how to reduce the number of selected terms and then reduce the number of forwarding messages to the home nodes of selected terms.…”

“…(ii) Data Location: in [28], the home node of t i acts as a rendezvous node to forward d to those filters containing t i . Rao and Chen [28] focuses on how to reduce the number of selected terms and then reduce the number of forwarding messages to the home nodes of selected terms. Instead, in this paper, the documents (and filters) are arbitrarily located at sources (and agents), and for a term t i , the filters containing t i appear in an arbitrary number of subscription agents.…”

“…Instead, in this paper, the documents (and filters) are arbitrarily located at sources (and agents), and for a term t i , the filters containing t i appear in an arbitrary number of subscription agents. Thus, the rendezvous node in [28] does not help reduce the overall cost of this paper. (iii) Solution: Rao and Chen [28] reduce the forwarding cost by reducing the number of selected terms over a very large number of peer-topeer nodes (e..g, millions).…”