Cray Cascade: A scalable HPC system based on a Dragonfly network

Faanes, Greg; Bataineh, Abdulla; Roweth, Duncan; Court, Tom Van; Froese, Edwin; Alverson, Bob; Johnson, Tim; Kopnick, Joe; Higgins, Mike; Reinhard, James

doi:10.1109/sc.2012.39

Cited by 176 publications

(101 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multi-level direct networks have been proposed recently by several researchers as a scalable topology for connecting a large number of nodes together [1], [2], [3], [4]. The basic idea behind these networks is to have a topology that resembles an all-to-all at each level of the hierarchy which gives the impression of a highly connected network.…”

Section: The Dragonfly Interconnectmentioning

confidence: 99%

“…Two prominent implementations of multi-level direct networks are the PERCS interconnect by IBM [3] and the Cascade system by Cray [4]. We focus on the Cascade system which is based on the dragonfly topology designed by Kim et al [1].…”

Section: The Dragonfly Interconnectmentioning

confidence: 99%

“…This has pushed the HPC community towards designing low-diameter fast networks with large bisection bandwidth. The dragonfly topology [1] and its variants [2], [3], [4] are actively being explored as the interconnects that satisfy all these requirements.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Maximizing Throughput on a Dragonfly Network

Jain

Bhatelé

et al. 2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Abstract-Interconnection networks are a critical resource for large supercomputers. The dragonfly topology, which provides a low network diameter and large bisection bandwidth, is being explored as a promising option for building multi-Petaflop/s and Exaflop/s systems. Unlike the extensively studied torus networks, the best choices of message routing and job placement strategies for the dragonfly topology are not well understood. This paper aims at analyzing the behavior of an interconnect based on the dragonfly topology for various routing strategies, job placement policies, and application communication patterns. Our study is based on a novel model that predicts traffic on individual links for direct, indirect, and adaptive routing strategies. We analyze results for individual communication patterns and some common parallel job workloads. The predictions presented in this paper are for a 100+ Petaflop/s prototype machine with 92,160 highradix routers and 8.8 million cores.

show abstract

Section: The Dragonfly Interconnectmentioning

confidence: 99%

Section: The Dragonfly Interconnectmentioning

confidence: 99%

See 1 more Smart Citation

Maximizing Throughput on a Dragonfly Network

Jain

Bhatelé

et al. 2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…[10]) as well as in what follows, the cost of traditional tori and fat-trees becomes discouraging at very large scale and under arbitrary traffic. More scalable alternatives, in particular, Flattened Butterfly [3], Dragonfly [11], Jelly-fish [12] and Slim-fly [13] have thus been proposed.…”

Section: Introductionmentioning

confidence: 99%

Design Methodology for Optimizing Optical Interconnection Networks in High Performance Systems

Rumley

Glick

Hammond

et al. 2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Modern high performance computers connect hundreds of thousands of endpoints and employ thousands of switches. This allows for a great deal of freedom in the design of the network topology. At the same time, due to the sheer numbers and complexity involved, it becomes more challenging to easily distinguish between promising and improper designs. With ever increasing line rates and advances in optical interconnects, there is a need for renewed design methodologies that comprehensively capture the requirements and expose trade-offs expeditiously in this complex design space. We introduce a systematic approach, based on Generalized Moore Graphs, allowing one to quickly gauge the ideal level of connectivity required for a given number of end-points and traffic hypothesis, and to collect insight on the role of the switch radix in the topology cost. Based on this approach, we present a methodology for the identification of Pareto-optimal topologies. We apply our method to a practical case with 25,000 nodes and present the results.

show abstract

“…The topologies of the local and global interconnects are typically low-diameter direct topologies that exploit highradix routers. For example, the IBM PERCS Interconnect [2] employs an all-to-all topology (1D-Flattened Butterfly) in both the local and global interconnects, while the Cray XC30 (codenamed "Cascade", [3]) employs an 1D-Flattened Butterfly (FB) for the global interconnect and a 2D-FB within groups.…”

mentioning

confidence: 99%

OFAR-CM: Efficient Dragonfly Networks with Simple Congestion Management

García

Vallejo

Beivide

et al. 2013

2013 IEEE 21st Annual Symposium on High-Performance Interconnects

View full text Add to dashboard Cite

Abstract-Dragonfly networks are appealing topologies for large-scale Datacenter and HPC networks, that provide high throughput with low diameter and moderate cost. However, they are prone to congestion under certain frequent traffic patterns that saturate specific network links. Adaptive nonminimal routing can be used to avoid such congestion. That kind of routing employs longer paths to circumvent local or global congested links. However, if a distance-based deadlock avoidance mechanism is employed, more Virtual Channels (VCs) are required, what increases design complexity and cost.OFAR (On-the-Fly Adaptive Routing) is a previously proposed routing that decouples VCs from deadlock avoidance, making local and global misrouting affordable. However, the severity of congestion with OFAR is higher, as it relies on an escape subnetwork with low bisection bandwidth. Additionally, OFAR allows for unlimited misroutings on the escape subnetwork, leading to unbounded paths in the network and long latencies.In this paper we propose and evaluate OFAR-CM, a variant of OFAR combined with a simple congestion management (CM) mechanism which only relies on local information, specifically the credit count of the output ports in the local router. With simple escape subnetworks such as a Hamiltonian ring or a tree, OFAR outperforms former proposals with distance-based deadlock avoidance. Additionally, although long paths are allowed in theory, in practice packets arrive at their destination in a small number of hops. Altogether, OFAR-CM constitutes the first practicable mechanism to the date that supports both local and global misrouting in Dragonfly networks.

show abstract

Cray Cascade: A scalable HPC system based on a Dragonfly network

Cited by 176 publications

References 13 publications

Maximizing Throughput on a Dragonfly Network

Maximizing Throughput on a Dragonfly Network

Design Methodology for Optimizing Optical Interconnection Networks in High Performance Systems

OFAR-CM: Efficient Dragonfly Networks with Simple Congestion Management

Contact Info

Product

Resources

About