Performance trade-offs in reconfigurable networks for HPC

Abstract

Designing efficient interconnects to support high-bandwidth and low-latency communication is critical toward realizing high performance computing (HPC) and data center (DC) systems in the exascale era. At extreme computing scales, providing the requisite bandwidth through overprovisioning becomes impractical. These challenges have motivated studies exploring reconfigurable network architectures that can adapt to traffic patterns at runtime using optical circuit switching. Despite the plethora of proposed architectures, surprisingly little is known about the relative performances and trade-offs among different reconfigurable network designs. We aim to bridge this gap by tackling two key issues in reconfigurable network design. First, we study how cost, power consumption, network performance, and scalability vary based on optical circuit switch (OCS) placement in the physical topology. Specifically, we consider two classes of reconfigurable architectures: one that places OCSs between top-of-rack (ToR) switches—ToR-reconfigurable networks (TRNs)—and one that places OCSs between pods of racks—pod-reconfigurable networks (PRNs). Second, we tackle the effects of reconfiguration frequency on network performance. Our results, based on network simulations driven by real HPC and DC workloads, show that while TRNs are optimized for low fan-out communication patterns, they are less suited for carrying high fan-out workloads. PRNs exhibit better overall trade-off, capable of performing comparably to a fully non-blocking fat tree for low fan-out workloads, and significantly outperform TRNs for high fan-out communication patterns.

Overview of router architecture in high performance computing

“…Of late, the development of interconnects to facilitate low-latency and high-bandwidth is crucial in the direction of HPC and data center systems. To resolve this problem, Teh et al [19] investigated reconfigurable network architectures that can adapt to traffic patterns at runtime using optical circuit switching. The study of how network performance, cost, scalability, and power consumption differ based on optical circuit switch (OCS) placement in the physical topology is presented.…”

NegotiaToR: Towards A Simple Yet Effective On-demand Reconfigurable Datacenter Network

“…Optical switching technology has developed rapidly, and there has been a great interest in using them in DCNs in recent years [4,8,10,15,21,23,28,29,32,33,38,39,42,47,50,51,54]. To fully utilize the high capacity and power efficiency of optical switching, one trend is to connect racks with optical switches directly [4,10,21,32,33,51].…”

“…Meanwhile, the detouring also damages mice flow FCT, particularly when elephant flows are spread across the network and block the mice ones at intermediate nodes. The performance downgrade worsens under heavier loads, which is a critical concern for HPC tasks like large-scale ML training where large amounts of flows are synchronously released to the network [25,27,47]. NegotiaToR is designed to meet these needs, offering a practical solution that can accommodate the high-performance requirements of modern DCNs where fast optical switching technology is ready.…”

“…ACM ISBN 979-8-4007-0614-1/24/08 https://doi.org/10.1145/3651890.3672222 [35,36]. To address this, the development of optical switching technology [11,13,14,19,24,37,42,43] especially fast optical switching [13,14,19,43] has led researchers to turn to reconfigurable optical DCNs [4,5,8,10,15,21,23,28,29,32,33,38,39,47,50,51,54], which provide higher capacity as well as lower cost compared with packet switching.…”

Enabling Quasi-Static Reconfigurable Networks With Robust Topology Engineering

“…• Scalability: Using pods with hundreds of uplinks to the OCSs, our architecture could support up to about 100 pods. Since each pod could support Θ(1000) servers, our architecture can easily scale up to over 100k servers [43].…”