Region‐based fault‐tolerant distributed file storage system design in networks

Abstract

Distributed storage of data files in different nodes of a network enhances its fault tolerance capability by offering protection against node and link failures. Reliability is often achieved through redundancy in one of the following two ways: (i) storage of multiple copies of the entire file at different locations (nodes) or (ii) storage of file segments (not entire files) at different node locations. In the (N , K) file distribution scheme, N file segments from a file F are created in such a way that it is possible to reconstruct the entire file, just by accessing any K ≤ N segments. For the reconstruction scheme to work, it is essential that the K segments of the file are stored in nodes that are connected in the network. However, in the event of node/link failures, the network might become disconnected (i.e., split into several connected components). We focus on node failures that are spatially correlated or region based. Such failures are often encountered in disaster situations or natural calamities where only the nodes in the disaster zone are affected. The first goal of this research is to design a least cost file storage scheme to ensure that no matter which region is destroyed; resulting in fragmentation of the network, a largest connected component of the residual network will have enough file segments with which to reconstruct the entire file. In case the least cost to ensure this objective is within the allocated budget, the storage design will be all region fault-tolerant (ARFT). In case the least cost exceeds the allocated budget, design of an ARFT file storage system design is impossible. The second goal of this research is to design file storage schemes that will be maximum region fault-tolerant within the allocated budget. The third goal of this research is to investigate the impact of the coding parameters N and K on storage requirements for ensuring all region or maximum region fault-tolerant design. We provide approximation algorithms for the problems and evaluate their performance through simulation using two real networks and compare their results to the optimal solutions obtained using Integer Linear Program. The simulation results demonstrate that the approximation algorithms almost always produce near optimal results in a fraction of the time needed to find the optimal solution.

In search of network resilience: An optimization‐based view

“…Silva et al [199], Canale et al [47], Sen et al [195], Canale et al [48], Martins et al [141], Ghedini et al [92], Barbosa et al [25], Castillo et al [50], Lin and Zhou [135], and Vass et al [212] analyze problems in survivability, reliability, or fault‐tolerance for different required structures and/or operations. Kucharzak et al [129], Fouquet et al [86], Yang et al [225], Cheng et al [54], Myslitski et al [150], Busing et al [45], and Natalino et al [151] examine novel post‐perturbation routing or path selection protocols.…”

Motion standard level system based on Internet of things

“…Considering the total kT DataNode , we can estimate the average write delay of each file as T DataNode =k. Assuming that n small files are currently merged into m large files, HDFS contains T NameNode Writing operations of HMFS are predicted by formula (2), 15 where T cache refers to the time cost of caching a file in the memory buffer. In formula T NameNode and formula (1), the meaning is the same.T Block DataNode refers to the time cost of writing a data block to HDFS.…”

A Network Planning and Management Tool for mitigating the impact of spatially correlated failures in infrastructure networks

“…From the context of data survivability in the presence of spatially correlated faults in networks, the NPMT supports a "Region-based Distributed File Storage Module" that implements an algorithm proposed in [11] that ensures that: (i) even when the network is fractured into disconnected components due to a region-based fault, at least one of the largest components will have access to at least K distinct file segments with which to reconstruct the entire file, and (ii) the total storage requirement is minimized. As of writing this paper, this module is currently under development and will be part of the NPMT upon its completion.…”

“…Several studies in the network research community have focused on different aspects of spatially correlated or regionbased faults in networks [5][6][7][8][9][10][11], however, to the best of our knowledge there does not exist an executable platform that consolidates the findings and techniques of these studies into a readily usable tool. The NPMT is intended to fill that gap and be such a platform that can incorporate the outcomes developed in [5][6][7][8][9][10][11] into executable modules to be integrated into the NPMT. This will allow network designers, planners and operators to use the results of these studies in their real world operational networks.…”