Technology-Driven, Highly-Scalable Dragonfly Topology

Abstract

Evolving technology and increasing pin-bandwidth motivate the use of high-radix routers to reduce the diameter, latency, and cost of interconnection networks. High-radix networks, however, require longer cables than their low-radix counterparts. Because cables dominate network cost, the number of cables, and particularly the number of long, global cables should be minimized to realize an efficient network. In this paper, we introduce the dragonfly topology which uses a group of high-radix routers as a virtual router to increase the effective radix of the network. With this organization, each minimally routed packet traverses at most one global channel. By reducing global channels, a dragonfly reduces cost by 20% compared to a flattened butterfly and by 52% compared to a folded Clos network in configurations with ≥ 16K nodes.We also introduce two new variants of global adaptive routing that enable load-balanced routing in the dragonfly. Each router in a dragonfly must make an adaptive routing decision based on the state of a global channel connected to a different router. Because of the indirect nature of this routing decision, conventional adaptive routing algorithms give degraded performance. We introduce the use of selective virtual-channel discrimination and the use of credit round-trip latency to both sense and signal channel congestion. The combination of these two methods gives throughput and latency that approaches that of an ideal adaptive routing algorithm.

New Systems, New Behaviors, New Patterns: Monitoring Insights from System Standup

“…While Cielo utilized the Cray Gemini interconnect configured in a 3D torus, Trinity's interconnect uses the Cray Aries interconnect configured in a Dragonfly [1], [2] configuration. While some of the features of these two interconnect technologies are similar, there are large differences in message logging, performance counters, error handling, adaptive routing, and hot swapability.…”

Extreme-Scale Viability of Collective Communication for Resilient Task Scheduling and Work Stealing

“…A binary fat-tree is pedagogically most common but would require too many switches to achieve 100K network endpoints and is not considered. A dragonfly topology [30] is another strong candidate for extreme-scale. However, maximizing throughput requires more complicated routing schemes, e.g.…”

Combining HoL-blocking avoidance and differentiated services in high-speed interconnects

“…Recently, hybrid and hierarchical topologies [10], [11] have been proposed to get the benefits from both direct and indirect topologies. Indeed, hybrid topologies aim to provide a similar performance to (or even higher than) that of indirect topologies, but at a similar cost to that of direct topologies.…”