MemEFS: A network-aware elastic in-memory runtime distributed file system

Uta, Alexandru; Danner, Ove; Weegen, Cas van der; Oprescu, Ana-Maria; Sandu, Andreea; Costache, Stefania; Kielmann, Thilo

doi:10.1016/j.future.2017.03.017

Cited by 5 publications

(3 citation statements)

References 41 publications

(70 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Aiming at multilayer data center, a multilayer topology structure is constructed from the perspective of network planning, and a network bandwidth model is established to localize network traffic and reduce the communication overhead in the upper layer network switch, thus ultimately reducing the overall traffic across the data center and reducing the network consumption of the cloud data center [16,17]. (2) In other application environments, such as distributed storage system [18][19][20][21], grid computing [22], online social network [23], peer-to-peer (P2P) network [24,25], software-defined network (SDN) [26][27][28], content distribution network (CDN) [29,30], and big data network storage [31,32], the research on data layout management is of great value.…”

Section: Research Progress Of Data Layout In Storage Systemmentioning

confidence: 99%

“…In the distributed storage system, the network aware repair framework based on the dependency between data, storage demand, and available bandwidth [18][19][20] can find the data repair scheme with the minimum bandwidth cost in the dynamic network and realize the load balance of storage and network traffic. Hedera [21] is a scalable dynamic flow scheduling system that schedules a multistage switching structure adaptively to utilize the resources of the aggregate network effectively.…”

Section: Research Progress Of Data Layout In Storage Systemmentioning

confidence: 99%

See 1 more Smart Citation

Network-Aware Data Placement Strategy in Storage Cluster System

Shao

Song

Bian

et al. 2020

Mathematical Problems in Engineering

View full text Add to dashboard Cite

The dramatic increase of storage devices in distributed storage cluster system and the inherent characteristics of distributed deployment mode make network resources become one of the bottlenecks of data storage process. By analyzing the functional characteristics of data flow transmission of the components of the storage system, the network topology structure of the storage system is constructed, and the evaluation index of node flow load is put forward based on the degree centrality and the betweenness centrality theory of the network to explore the network topology and real-time flow characteristics. According to the evaluation index of node flow load, a network-aware data layout scheme is proposed. By balancing the flow load of bottleneck link, congestion and transmission delay can be reduced to further shorten the total task execution time and improve the efficiency of data writing.

show abstract

Section: Research Progress Of Data Layout In Storage Systemmentioning

confidence: 99%

Section: Research Progress Of Data Layout In Storage Systemmentioning

confidence: 99%

Network-Aware Data Placement Strategy in Storage Cluster System

Shao

Song

Bian

et al. 2020

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…Provided that the data migration mechanism is further optimized, and reduced by at least 50%, the resource utilization for both BFS and PR can be improved via elastic scaling policies (MF5). This could be achieved via non-blocking [44], or hot-migration [45] data movement mechanisms. Moreover, as static processing scalability is not linear (e.g., Static-20, and Static-10 use a similar amount of resources), as a user, or system administrator it is not trivial to determine an optimal configuration.…”

Section: The Cost and Benefits Of Elasticitymentioning

confidence: 99%

Elasticity in Graph Analytics? A Benchmarking Framework for Elastic Graph Processing

Uta

Ilyushkin

et al. 2018

2018 IEEE International Conference on Cluster Computing (CLUSTER)

Self Cite

View full text Add to dashboard Cite

Graphs are a natural fit for modeling concepts used in solving diverse problems in science, commerce, engineering, and governance. Responding to the diversity of graph data and algorithms, many parallel and distributed graph-processing systems exist. However, until now these platforms use a static model of deployment: they only run on a pre-defined set of machines. This raises many conceptual and pragmatic issues, including misfit with the highly dynamic nature of graph processing, and could lead to resource waste and high operational costs. In contrast, in this work we explore the benefits and drawbacks of the dynamic model of deployment. Building a threelayer benchmarking framework for assessing elasticity in graph analytics, we conduct an in-depth elasticity study of distributed graph processing. Our framework is composed of state-ofthe-art workloads, autoscalers, and metrics, derived from the LDBC Graphalytics benchmark and SPEC RG Cloud Group's elasticity metrics. We uncover the benefits and cost of elasticity in graph processing: while elasticity allows for fine-grained resource management, and does not degrade application performance, we find that graph workloads are sensitive to data migration while leasing or releasing resources. Moreover, we identify non-trivial interactions between scaling policies and graph workloads, which add an extra level of complexity to resource management and scheduling for graph processing.

show abstract