Overcoming far-end congestion in large-scale networks

Won, Jongmin; Kim, Gwangsun; Kim, John; Jiang, T.Z.; Parker, Michael; Scott, Steve

doi:10.1109/hpca.2015.7056051

Cited by 47 publications

(22 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other works analyze the limitations of adaptive routing and propose solutions to improve it. Won et al [46] shown how the "far-end" congestion should be considered as phantom congestion, because it may be not properly represented with local information, such as the credit count. They propose a solution to overcome this problem and to avoid transient congestion, validating it by means of simulations.…”

Section: Related Workmentioning

confidence: 99%

“…We validate our solution on microbenchmarks and real-world applications on two systems relying on a Dragonfly interconnection network, showing noise reduction and performance improvement.is feared by datacenter operators in so-called incast or hot-spot (many-to-one) patterns. In addition to this, adaptive routing may be affected by the so-called phantom congestion problem [46]. Namely, as congestion information is propagated with some delay, a node may react too late to congestion events.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Mitigating network noise on Dragonfly networks through application-aware routing

Sensi

Girolamo

Hoefler

2019

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

System noise can negatively impact the performance of HPC systems, and the interconnection network is one of the main factors contributing to this problem. To mitigate this effect, adaptive routing sends packets on non-minimal paths if they are less congested. However, while this may mitigate interference caused by congestion, it also generates more traffic since packets traverse additional hops, causing in turn congestion on other applications and on the application itself. In this paper, we first describe how to estimate network noise. By following these guidelines, we show how noise can be reduced by using routing algorithms which select minimal paths with a higher probability. We exploit this knowledge to design an algorithm which changes the probability of selecting minimal paths according to the application characteristics. We validate our solution on microbenchmarks and real-world applications on two systems relying on a Dragonfly interconnection network, showing noise reduction and performance improvement.is feared by datacenter operators in so-called incast or hot-spot (many-to-one) patterns. In addition to this, adaptive routing may be affected by the so-called phantom congestion problem [46]. Namely, as congestion information is propagated with some delay, a node may react too late to congestion events. In this paper, we will be first to demonstrate and quantify the influence of different routing schemes on network noise in practice.Analyzing network noise in detail is delicate because in practice, when observing application delays, it is hard to distinguish between network noise, operating system noise, and application imbalance. Other works have used network counters but may run into the fallacy that correlation is not causation by ignoring the aspects of unrelated traffic. We will clarify several potential problems in investigations of network noise and develop a set of general guidelines for our analysis. Using these guidelines, we study the relationship between different adaptive routing schemes, application performance, and network noise. Our findings on two large-scale Cray Aries systems, Piz Daint and NERSC's Cori, are remarkable: not only will changing the adaptive routing mode reduce communication times and speed up applications up to twice, but it will also significantly reduce performance variation.We find that the best routing mode that minimizes network noise and maximizes performance depends not only on the characteristics of the allocation but also on the communication load. For example, large-scale alltoall communications are best routed with the default mode whereas many other communication patterns benefit from mostly minimal routing. Because this all depends on the location of the communication peers, there is no simple static rule to select the best routing mode. To address this, we develop a simple but effective dynamic routing library that observes the network state through local network counters and adjusts the routing mode for each message based on application characteristics s...

show abstract

Section: Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

Mitigating network noise on Dragonfly networks through application-aware routing

Sensi

Girolamo

Hoefler

2019

Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…In the intermediate or destination groups, local misrouting can be used if the links from the minimal path are considered saturated. This avoids pathological performance issues identified in [8], [10]. Deadlock avoidance implements Opportunistic Local Misrouting OLM, [3] to reduce the cost of the implementation.…”

Section: Routing Mechanismsmentioning

confidence: 99%

Throughput Unfairness in Dragonfly Networks under Realistic Traffic Patterns

Fuentes

Vallejo

Camarero

et al. 2015

2015 IEEE International Conference on Cluster Computing

View full text Add to dashboard Cite

Abstract-Dragonfly networks have a two-level hierarchical arrangement of the network routers, and allow for a competitive cost-performance solution in large systems. Nonminimal adaptive routing is employed to fully exploit the path diversity and increase the performance under adversarial traffic patterns. Throughput unfairness prevents a balanced use of the resources across the network nodes and degrades severely the performance of any application running on an affected node. Previous works have demonstrated the presence of throughput unfairness in Dragonflies under certain adversarial traffic patterns, and proposed different alternatives to effectively combat such effect.In this paper we introduce a new traffic pattern denoted adversarial consecutive (ADVc), which portrays a real use case, and evaluate its impact on network performance and throughput fairness. This traffic pattern is the most adversarial in terms of network fairness. Our evaluations, both with or without transit-over-injection priority, show that global misrouting policies do not properly alleviate this problem. Therefore, explicit fairness mechanisms are required for these networks.

show abstract

“…At intermediate or destination groups, local misrouting (sending traffic to a non-minimal router in the same group) is used if the links from the minimal path are considered saturated. This avoids pathological performance issues identified in [14,25].…”

Section: In-transit Adaptive Routingmentioning

confidence: 99%

Network unfairness in dragonfly topologies

et al. 2016

View full text Add to dashboard Cite

Dragonfly networks arrange network routers in a two-level hierarchy, providing a competitive cost-performance solution for large systems. Nonminimal adaptive routing (adaptive misrouting) is employed to fully exploit the path diversity and increase the performance under adversarial traffic patterns. Network fairness issues arise in the dragonfly for several combinations of traffic pattern, global misrouting and traffic prioritization policy. Such unfairness prevents a balanced use of the resources across the network nodes and degrades severely the performance of any application running on an affected node. This paper reviews the main causes behind network unfairness in dragonflies, including a new adversarial traffic pattern which can easily occur in actual systems and congests all the global output links of a single router. A solution for the observed unfairness is evaluated using age-based arbitration. Results show that age-based arbitration mitigates fairness issues especially when using in-transit adaptive routing. However, when using source adaptive routing, the saturation of the new traffic pattern interferes with the mechanisms employed to detect remote congestion, and the problem grows with the network size. This makes source adaptive routing in dragonflies based on remote notifications prone to reduced performance, even when using age-based arbitration.

show abstract

Overcoming far-end congestion in large-scale networks

Cited by 47 publications

References 27 publications

Mitigating network noise on Dragonfly networks through application-aware routing

Mitigating network noise on Dragonfly networks through application-aware routing

Throughput Unfairness in Dragonfly Networks under Realistic Traffic Patterns

Network unfairness in dragonfly topologies

Contact Info

Product

Resources

About