A New Data Sieving Approach for High Performance I/O

Abstract

Abstract. Many scientific computing applications and engineering simulations exhibit noncontiguous I/O access patterns. Data sieving is an important technique to improve the performance of noncontiguous I/O accesses by combining small and noncontiguous requests into a large and contiguous request. It has been proven effective even though more data is potentially accessed than demanded. In this study, we propose a new data sieving approach namely Performance Model Directed Data Sieving, or PMD data sieving in short. It improves the existing data sieving approach from two aspects: 1) dynamically determines when it is beneficial to perform data sieving; and 2) dynamically determines how to perform data sieving if beneficial. It improves the performance of the existing data sieving approach and reduces the memory consumption as verified by experimental results. Given the importance of supporting noncontiguous accesses effectively and reducing the memory pressure in a large-scale system, the proposed PMD data sieving approach in this research holds a promise and will have an impact on high performance I/O systems.

Remote sensing big data computing: Challenges and opportunities

“…The noncontiguous I/O requests could be rearranged into a contiguous one for exploring better data locality but also brings in extra network communication. Data sieving [76] normally loads a large chunk of data and sieves out non-requested ones in one single I/O. This approach could apparently reduce I/O calls but at the penalty of reading extra data.…”

Orthrus: A Framework for Implementing Efficient Collective I/O in Multi-core Clusters

“…To provision a framework with adaptivity to accommodate workload and system dynamics to maximize performance an effort has been made using data sieving [8]. Instead of conducting actual test runs to select the best performer among multiple candidates they a model is used to predict the performance of candidates in various settings.…”

Performance model-directed data sieving for high-performance I/O

“…An initial finding was presented in [22] and this paper presents a complete study of the PMD data sieving technique based on access patterns to improve the parallel I/O performance and reduce the memory consumption. The remainder of this paper is organized as follows.…”