Fail-in-Place Network Design: Interaction Between Topology, Routing Algorithm and Failures

Abstract

Abstract-The growing system size of high performance computers results in a steady decrease of the mean time between failures. Exchanging network components often requires whole system downtime which increases the cost of failures. In this work, we study a fail-in-place strategy where broken network elements remain untouched. We show, that a fail-in-place strategy is feasible for todays networks and the degradation is manageable, and provide guidelines for the design. Our network failure simulation toolchain allows system designers to extrapolate the performance degradation based on expected failure rates, and it can be used to evaluate the current state of a system. In a case study of real-world HPC systems, we will analyze the performance degradation throughout the systems lifetime under the assumption that faulty network components are not repaired, which results in a recommendation to change the used routing algorithm to improve the network performance as well as the fail-in-place characteristic.

Combining OpenFabrics Software and Simulation Tools for Modeling InfiniBand-Based Interconnection Networks

“…Fig.2 shows the architecture of the proposed combination of OFS and OMNeT++. In general, we have followed similar ideas to those proposed in [10], [14], but we introduce several improvements. The main contributions of this methodology are the following:…”

“…Next, we use the IB tools included in OFS to obtain the information needed to build the network in the simulation tool. For the time being, we have put efforts in the development of a software layer which integrates OFS and the network simulators proposed in [8], [9] and [10]. The rest of this paper is organized as follows: Section II shows an overview of the InfiniBand architecture.…”

“…After writing NED files, the programmer has to populate with C++ code the modules defined in the NED files, in order to implement their functionality. Examples of OMNeT++-based simulation modeles for HPC interconnection networks have been used in [8], [9] and [10].…”

Dare

“…Then, it Component AFR MTTF Reliability Network [12,17] 1.00% 876, 000 4-nines NIC [12,17] 1.00% 876, 000 4-nines DRAM [18] 39.5% 22, 177 2-nines CPU [18] 41.9% 20, 906 2-nines Server [17,39] 47.9% 18, 304 2-nines Table 2: Worst case scenario reliability data. The reliability is estimated over a period of 24 hours and expressed in the "nines" notation; the MTTF is expressed in hours.…”

“…Various sources provide failure data of systems and system components [12,18,31,36]. Yet, systems range from very reliable ones with AFRs per component below 0.2% [31] to relatively unreliable ones with component failure log events at an annual rate of more than 40% [18] (here we assume that a logged error impacted the function of the device).…”