Constructing scalable overlays for pub-sub with many topics

Abstract

We investigate the problem of designing a scalable overlay network to support decentralized topic-based pub/sub communication. We introduce a new optimization problem, called Minimum Topic-Connected Overlay (Min-TCO), that captures the tradeoff between the scalability of the overlay (in terms of the nodes' fanout) and the message forwarding overhead incurred by the communicating parties. Roughly, the Min-TCO problem is as follows: Given a collection of nodes and their subscriptions, connect the nodes using the minimum possible number of edges so that for each topic t, a message published on t could reach all the nodes interested in t by being forwarded by only the nodes interested in t.We show that the decision version of Min-TCO is NPcomplete, and present a polynomial algorithm that approximates the optimal solution within a logarithmic factor with respect to the number of edges in the constructed overlay. We further prove that this approximation ratio is almost tight by showing that no polynomial algorithm can approximate Min-TCO within a constant factor (unless P=NP). We show experimentally that on typical inputs, the fanout of the overlay constructed by our approximation algorithm is significantly lower than that of the overlays built by the existing algorithms, and that its running time is just a small fraction of the analytical worst case bound. As Min-TCO can be shown to capture several important aspects of most known overlay-based pub/sub implementations, our study sheds light on the inherent limitations of the existing systems and provides an insight into the best possible feasible solution.Finally, we introduce a flexible framework that generalizes Min-TCO and formalizes most similar overlay design problems that occur in scalable pub/sub systems. We also briefly discuss several examples of such problems, and show some results with respect to their complexity.

Designing Overlay Networks for Handling Exhaust Data in a Distributed Topic-based Pub/Sub Architecture

“…The property of relay-free [10], also called Topic-connected Overlay, is primarily discussed in studies based on unstructured overlay networks [11]. The definition is as follows:…”

“…For overcoming such inefficiencies, we first clarify the requirements of overlay networks for improving the adaptivity and define a desirable property called "strong relay-free" by expanding relay-free, which is mainly known in studies based on unstructured overlay networks [10], [11]. In the strong relay-free property, publishers and subscribers compose connected subgraphs respectively, and these subgraphs are also connected to each other.…”

PolderCast: Fast, Robust, and Scalable Architecture for P2P Topic-Based Pub/Sub

“…Furthermore, guaranteed absence of relays, a.k.a. topic-connectivity [6], simplifies message routing mechanisms. On the other hand, fanout is a common minimization parameter in overlay design, which strongly affects system scalability.…”

“…Unfortunately, the desirable characteristics of having a low node degree and relay-free routing exhibit a fundamental trade-off [6]. At one extreme is having a fixed node degree independent of the number of topics a node is interested in.…”

“…In particular, the list includes: (1) Correct delivery of all publications, i.e., absence of false negatives or deterministic 100% hit-ratio guarantee in a failure-free run, (2) High hit-ratio under realistic node churn, (3) Fast recovery at the end of a churn period and mending of the overlay so as to achieve 100% hit-ratio, (4) Low degree of overlay nodes, (5) Relay-free routing (also called topic-connectivity), which means that only subscribers interested in a topic are involved in routing events for that topic, (6) Scalability with the number of nodes, topics, number of nodes interested in a topic, and number of topics a node is interested in, (7) Effective dissemination: fast, with as little duplicate delivery as possible, and fair distribution of load due to routing and processing, and (8) Low overhead of overlay maintenance. The design challenge is amplified due to a number of trade-offs: low node degree and relay-free routing, robustness under churn and lack of duplicate delivery, scalability and precise delivery with few false negatives and false positives are fundamentally at odds with each other.…”