Improving the Average Response Time in Collective I/O

Chen, Jin; Sehrish, Saba; Liao, Wei-keng; Choudhary, Alok; Schuchardt, Karen

doi:10.1007/978-3-642-24449-0_10

Cited by 7 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The request size was set as 15 MB, and the aggregator's request size is equal. The number of storage nodes was varied as 2, 4, 6, 8, and 16.…”

Section: Results and Analysesmentioning

confidence: 99%

“…5 shows the aggregator's sending time, which is the time this aggregator takes to send the data to the collective buffer. The lower dashed portion in other bars (2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16) shows the waiting time of non-aggregators and the higher portion refers to their receiving time. We can see that the data exchange time between aggregator and non-aggregators is various, due to which, we can argue that the shuffle cost is determined by the maximum data exchange time (in this case, i.e., 2.3 ms).…”

Section: Analysis and Prediction Of Shuffle Costmentioning

confidence: 99%

“…In [16], the authors proposed three scheduling algorithms, with considering the number of processes per file stripe and the number of accesses per process, to minimize the average response time in collective I/O. In servicing one aggregator, instead of scheduling one stripe at a time in the increasing file offset order, they propose to prioritize the file stripes based on their access degree, the number of accessing processes.…”

Section: I/o Schedulingmentioning

confidence: 99%

See 2 more Smart Citations

Hierarchical Collective I/O Scheduling for High-Performance Computing

2015

View full text Add to dashboard Cite

The non-contiguous access pattern of many scientific applications results in a large number of I/O requests, which can seriously limit the data-access performance. Collective I/O has been widely used to address this issue. However, the performance of collective I/O could be dramatically degraded in today's high-performance computing systems due to the increasing shuffle cost caused by highly concurrent data accesses. This situation tends to be even worse as many applications become more and more data intensive. Previous research has primarily focused on optimizing I/O access cost in collective I/O but largely ignored the shuffle cost involved. Previous works assume that the lowest average response time leads to the best QoS and performance, while that is not always true for collective requests when considering the additional shuffle cost. In this study, we propose a new hierarchical I/O scheduling (HIO) algorithm to address the increasing shuffle cost in collective I/O. The fundamental idea is to schedule applications' I/O requests based on a shuffle cost analysis to achieve the optimal overall performance, instead of achieving optimal I/O accesses only. The algorithm is currently evaluated with the MPICH3 and PVFS2. Both theoretical analysis and experimental tests show that the proposed hierarchical I/O scheduling has a potential in addressing the degraded performance issue of collective I/O with highly concurrent accesses.

show abstract

“…The request size was set as 15 MB, and the aggregator's request size is equal. The number of storage nodes was varied as 2, 4, 6, 8, and 16.…”

Section: Results and Analysesmentioning

confidence: 99%

Section: Analysis and Prediction Of Shuffle Costmentioning

confidence: 99%

Section: I/o Schedulingmentioning

confidence: 99%

See 1 more Smart Citation

Hierarchical Collective I/O Scheduling for High-Performance Computing

2015

View full text Add to dashboard Cite

show abstract

“…In [11], the authors proposed three scheduling algorithms, with considering the number of processes per file stripe and the number of accesses per process, to minimize the average response time in collective I/O. In servicing one aggregator, instead of scheduling one stripe at a time in the increasing file offset order, they propose to prioritize the file stripes based on their access degree, the number of accessing processes.…”

Section: Figure 6: Average Execution Time With Different Storage Nodesmentioning

confidence: 99%

Hierarchical I/O Scheduling for Collective I/O

Liu

Chen

Zhuang

2013

2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing

View full text Add to dashboard Cite

The non-contiguous access pattern of many scientific applications results in a large number of I/O requests, which can seriously limit the data-access performance. Collective I/O has been widely used to address this issue. However, the performance of collective I/O could be dramatically degraded in today's high-performance computing system due to the increasing shuffle cost caused by highly concurrent data accesses. This situation tends to be even worse as many applications become more and more data intensive. Previous research has primarily focused on optimizing I/O access cost in collective I/O but largely ignored the shuffle cost involved. In this study, we propose a new hierarchical I/O scheduling (HIO) algorithm to address the increasing shuffle cost in collective I/O. The fundamental idea is to schedule applications' I/O requests based on a shuffle cost analysis to achieve the optimal overall performance, instead of achieving optimal I/O accesses only. The algorithm is currently evaluated with the MPICH2 and PVFS2. Both theoretical analysis and experimental tests show that the proposed hierarchical I/O scheduling has a potential in addressing the degraded performance issue of collective I/O with highly concurrent accesses.

show abstract

A formal method for rule analysis and validation in distributed data aggregation service

Serbanescu¹,

Pop²,

Cristea³

et al. 2015

World Wide Web

View full text Add to dashboard Cite

The usage of Cloud Serviced has increased rapidly in the last years. Data management systems, behind any Cloud Service, are a major concern when it comes to scalability, flexibility and reliability due to being implemented in a distributed way. A Distributed Data Aggregation Service relying on a storage system meets these demands and serves as a repository back-end for complex analysis and automatic mining of any type of data. In this paper we continue our previous work on data management in Cloud storage. We present a formal approach to express retrieval and aggregation rules with a compact, yet powerful tool called Rule Markup Language. Our extended solution proposes a standard form to schemes and uses the tool to match the rules to the XML form of the structured data in order to obtain the unstructured entries from BlobSeer data storage system. This allows the Distributed Data Aggregation Service (DDAS) to bypass several steps when processing a retrieval request. Our new architecture is more loosely-coupled with a separate module, the new tool, used for transforming the XML entries to standard XML files which represent the final result. We model the dynamic behavior of the system using this new standard to ensure a simpler and efficient representation of the operations performed by the client while maintaining the constraints imposed by a distributed system running in the Cloud. Furthermore we prove that this method correctly performs the translation between the storage model's unstructured view of data and the client's structured objects.

show abstract

Improving the Average Response Time in Collective I/O

Cited by 7 publications

References 13 publications

Hierarchical Collective I/O Scheduling for High-Performance Computing

Hierarchical Collective I/O Scheduling for High-Performance Computing

Hierarchical I/O Scheduling for Collective I/O

A formal method for rule analysis and validation in distributed data aggregation service

Contact Info

Product

Resources

About