IOrchestrator: Improving the Performance of Multi-node I/O Systems via Inter-Server Coordination

Zhang, Xuechen; Davis, Kei; Jiang, Song

doi:10.1109/sc.2010.30

Cited by 56 publications

(37 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table VIII reports the average of absolute difference over the course of each run (in its entirety, and not restricted to the stationary regime). We also compare the performance of Omnisc'IO with the immediate reaccess estimation used by some I/O schedulers (e.g., [27]), which consists of assuming that the next I/O operation is likely to immediately follow the current one (i.e., interarrival time are always estimated to 0) and use a time window during which a potential new operation is expected). In all situations, Omnisc'IO appears to be very good at predicting the interarrival time of I/O accesses.…”

Section: E Temporal Predictionmentioning

confidence: 99%

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

Dorier

Ibrahim²,

Antoniu³

et al. 2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Abstract-The increasing gap between the computation performance of post-petascale machines and the performance of their I/O subsystem has motivated many I/O optimizations including prefetching, caching, and scheduling techniques. In order to further improve these techniques, modeling and predicting spatial and temporal I/O patterns of HPC applications as they run has became crucial.In this paper we present Omnisc'IO, an approach that builds a grammar-based model of the I/O behavior of HPC applications and uses it to predict when future I/O operations will occur, and where and how much data will be accessed. Omnisc'IO is transparently integrated into the POSIX and MPI I/O stacks and does not require any modification in applications or higherlevel I/O libraries. It works without any prior knowledge of the application and converges to accurate predictions within a couple of iterations only. Its implementation is efficient in both computation time and memory footprint.

show abstract

Section: E Temporal Predictionmentioning

confidence: 99%

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

Dorier

Ibrahim²,

Antoniu³

et al. 2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…Zhang, Davis, and Jiang [116] propose an approach named IOrchestrator to the PVFS parallel file system. Their idea is to synchronize all data servers to serve only one application during a given period.…”

Section: I/o Schedulingmentioning

confidence: 99%

“…Dong et al [22] use a time series model to estimate file system servers' load. The approach by Zhang, Davis, and Jiang [116] applies a "reuse distance", defined as the time difference between consecutive requests from the same application at the servers.…”

Section: Runtime Detectionmentioning

confidence: 99%

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

We present a comprehensive survey on parallel I/O in the high performance computing (HPC) context. This is an important field for HPC because of the historic gap between processing power and storage latencies, which causes applications performance to be impaired when accessing or generating large amounts of data. As the available processing power and amount of data increase, I/O remains a central issue for the scientific community. In this survey, we focus on a traditional I/O stack, with a POSIX parallel file system. We present background concepts everyone could benefit from. Moreover, through the comprehensive study of publications from the most important conferences and journals in a five-year time window, we discuss the state of the art of I/O optimization approaches, access pattern extraction techniques, and performance modeling, in addition to general aspects of parallel I/O research. Through this approach, we aim at identifying the general characteristics of the field and the main current and future research topics.

show abstract

“…This choice can be justified by the LiU's ability of providing data access of much reduced latency. The ability allows the client to see predictable and consistent service times across sub-requests, which the client breaks a request into and are issued to different data nodes, without communication between clients and data nodes and coordination between data nodes [23]. Specifically in PVFS2, on each data node there is a server daemon, pvfs2-server, responsible for creating I/O jobs (i.e., I/O requests).…”

Section: A Positioning Liu In the Parallel File Systemmentioning

confidence: 99%

LiU: Hiding Disk Access Latency for HPC Applications with a New SSD-Enabled Data Layout

Huang¹,

Zhang

Shi

et al. 2013

2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems

Self Cite

View full text Add to dashboard Cite

Abstract-Unlike in the consumer electronics and personal computing areas, in the HPC environment hard disks can hardly be replaced by SSDs. The reasons include hard disk's large capacity, very low price, and decent peak throughput. However, when latency dominates the I/O performance (e.g., when accessing random data), the hard disk's performance can be compromised. If the issue of high latency could be effectively solved, the HPC community would enjoy a large, affordable and fast storage without having to replace disks completely with expensive SSDs.In this paper, we propose an almost latency-free hard-diskdominated storage system called LiU for HPC. The key technique is leveraging limited amount of SSD storage for its low-latency access, and changing data layout in a hybrid storage hierarchy with low-latency SSD at the top and high-latency hard disk at the bottom. If a segment of data would be randomly accessed, we lift its top part (the head) up in the hierarchy to the SSD and leave the remaining part (the body) untouched on the disk. As a result, the latency of accessing this whole segment can be removed because access latency of the body can be hidden by the access time of the head on the SSD. Combined with the effect of prefetching a large segment, LiU (Lift it Up) can effectively remove disk access latency so disk's high peak throughput can now be fully exploited for data-intensive HPC applications.We have implemented a prototype of LiU in the PVFS parallel file system and evaluated it with representative MPI-IO microbenchmarks, including mpi-io-test, mpi-tile-io, and ior-mpi-io, and one macro-benchmark BTIO. Our experimental results show that LiU can effectively improve the I/O performance for HPC applications, with the throughput improvement ratio up to 5.8. Furthermore, LiU can bring much more benefits to sequential-I/O MPI applications when the applications are interfered by other workloads. For example, LiU improves the I/O throughput of mpi-io-test, which is under interference, by 1.1-3.4 times, while improving the same workload without interference by 15%.

show abstract

IOrchestrator: Improving the Performance of Multi-node I/O Systems via Inter-Server Coordination

Cited by 56 publications

References 16 publications

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

A Checkpoint of Research on Parallel I/O for High-Performance Computing

LiU: Hiding Disk Access Latency for HPC Applications with a New SSD-Enabled Data Layout

Contact Info

Product

Resources

About