A routing methodology for achieving fault tolerance in direct networks

Abstract

Abstract-Massively parallel computing systems are being built with thousands of nodes. The interconnection network plays a key role for the performance of such systems. However, the high number of components significantly increases the probability of failure. Additionally, failures in the interconnection network may isolate a large fraction of the machine. It is therefore critical to provide an efficient fault-tolerant mechanism to keep the system running, even in the presence of faults. This paper presents a new fault-tolerant routing methodology that does not degrade performance in the absence of faults and tolerates a reasonably large number of faults without disabling any healthy node. In order to avoid faults, for some source-destination pairs, packets are first sent to an intermediate node and then from this node to the destination node. Fully adaptive routing is used along both subpaths. The methodology assumes a static fault model and the use of a checkpoint/restart mechanism. However, there are scenarios where the faults cannot be avoided solely by using an intermediate node. Thus, we also provide some extensions to the methodology. Specifically, we propose disabling adaptive routing and/or using misrouting on a per-packet basis. We also propose the use of more than one intermediate node for some paths. The proposed fault-tolerant routing methodology is extensively evaluated in terms of fault tolerance, complexity, and performance.

On the Probability of Facing Fault Patterns: A Performance and Comparison Measure of Network Fault-Tolerance

“…Some of these definitions are reiterated from previous works [6,7,[10][11][12], for the sake of completeness.…”

“…The torus has been popular interconnection network topology in contemporary systems [6] due to their desirable properties, such as ease of implementation and ability to exploit communication locality to reduce message latency [12]. In addition, torus is a regular (i.e., all nodes have the same degree) and edge-symmetric network, which improves load balancing across the channels [13].…”

“…However, rectangular fault regions sacrifice many nonfaulty nodes and hence its resources are wasted. In order to reduce non-faulty nodes in rectangular fault regions, many studies have addressed the concept of fault patterns with different shapes [7][8][9][10]12] may form convex or concave regions. A convex region is defined as a region ϕ in which a line segment connecting any two points in ϕ lies entirely within ϕ .…”

“…The detailed mathematical expressions for characterization of the most common concave and convex fault regions in torus and mesh networks have been reported in [14]. For a comprehensive survey of the important issues of the faulttolerant systems and networks, the reader is referred to the articles in [12][13][14][15].…”

“…To be able to adapt with faults without serious degradation of the service, networks and routing protocols have to be set up so that they are fault-tolerant. Several recent studies address faulttolerance in a diverse range of systems and applications [1][2][3][4][5][6][7][8][9][10][11][12]. Almost all of the performance evaluation studies for functionality of these systems, however, have made use solely of simulation experiments.…”