Interfaces for coordinated access in the file system

Lang, Samuel; Latham, Robert; Ross, Rob; Kimpe, Dries

doi:10.1109/clustr.2009.5289152

Cited by 8 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Numerous optimizations also exist to improve the file system performance, such as data staging services [2,3], coordinated access interface [18], performance bridging [14], a log-structured interposition layer and latent asynchrony I/O [43]. While parallel file systems perform well for large and well-formed data streams, they often perform inadequately in dealing with many small and noncontiguous data requests.…”

Section: Related Workmentioning

confidence: 99%

Performance model-directed data sieving for high-performance I/O

et al. 2014

View full text Add to dashboard Cite

Many scientific computing applications and engineering simulations exhibit noncontiguous I/O access patterns. Data sieving is an important technique to improve the performance of noncontiguous I/O accesses by combining small and noncontiguous requests into a large and contiguous request. It has been proven effective even though more data are potentially accessed than demanded. In this study, we propose a new data sieving approach namely performance model-directed data sieving, or PMD data sieving in short. It improves the existing data sieving approach from two aspects:(1) dynamically determines when it is beneficial to perform data sieving; and (2) dynamically determines how to perform data sieving if beneficial. It improves the performance of the existing data sieving approach considerably and reduces the memory consumption as verified by both theoretical analysis and experimental results. Given the importance of supporting noncontiguous accesses effectively and reducing the memory pressure in a large-scale system, the proposed PMD data sieving approach in this research holds a great promise and will have an impact on high-performance I/O systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Performance model-directed data sieving for high-performance I/O

et al. 2014

View full text Add to dashboard Cite

show abstract

“…Parallel file systems, such as Lustre (Lustre 2.0 Operations Manual), GPFS (Schmuck and Haskin, 2002), PanFS (Welch et al, 2008), and PVFS/PVFS2 (Carns et al), enable concurrent I/O accesses from multiple clients to files. Numerous optimizations also exist that improve the file system performance, such as data staging services (Abbasi et al, 2011), coordinated access interface (Lang et al, 2009), performance bridging (Gu et al, 2008), a log-structured interposition layer, and latent asynchrony I/O (Widener et al, 2011). While parallel file systems perform well for large and well-formed data streams, however, they often perform inadequately when dealing with many small and noncontiguous data requests.…”

Section: Related Workmentioning

confidence: 99%

Collective input/output under memory constraints

Yin

Chen

Zhuang

et al. 2014

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input/output (I/O) performance. In this study, we introduce a memory-conscious collective I/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intranode and internode layers, and determines I/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I/O can have a significant positive impact on scientific discovery productivity.

show abstract

“…Optimistic concurrency control and conditional operations have been used on file data in order to enable richer file system primitives for application coordination [42]- [44]. BatchFS is different from these techniques in that it focuses on metadata concurrency and targets batch applications that do not use the file system as a synchronization mechanism.…”

Section: Related Workmentioning

confidence: 99%

BatchFS: Scaling the File System Control Plane with Client-Funded Metadata Servers

Zheng

Ren

Gibson

2014

2014 9th Parallel Data Storage Workshop

View full text Add to dashboard Cite

Abstract-Parallel file systems are often characterized by a layered architecture that decouples metadata management from I/O operations, allowing file systems to facilitate fast concurrent access to file contents. However, metadata intensive workloads are still likely to bottleneck at the file system control plane due to namespace synchronization, which taxes application performance through lock contention on directories, transaction serialization, and RPC overheads. In this paper, we propose a client-driven file system metadata architecture, BatchFS, that is optimized for noninteractive, or batch, workloads. To avoid metadata bottlenecks, BatchFS features a relaxed consistency model marked by lazy namespace synchronization and optimistic metadata verification. Capable of executing namespace operations on client-provisioned resources without contacting any metadata server, BatchFS clients are able to delay namespace synchronization until synchronization is really needed. Our goal in this vision paper is to handle these delayed operations securely and efficiently with metadata verification and bulk insertion. Preliminary experiments demonstrate that our client-funded metadata architecture outperforms a traditional synchronous file system by orders of magnitude.

show abstract

Interfaces for coordinated access in the file system

Cited by 8 publications

References 15 publications

Performance model-directed data sieving for high-performance I/O

Performance model-directed data sieving for high-performance I/O

Collective input/output under memory constraints

BatchFS: Scaling the File System Control Plane with Client-Funded Metadata Servers

Contact Info

Product

Resources

About