Server-Side Log Data Analytics for I/O Workload Characterization and Coordination on Large Shared Storage Systems

Liu, Yang; Gunasekaran, Raghul; Ma, Xiaosong; Sudharshan, S Vazhkudai

doi:10.1109/sc.2016.69

Cited by 35 publications

(13 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The examples illustrates that HPC applications tend to present a consistent I/O behavior, with a few access patterns being repeated multiple times over an extensive period [15]. Therefore, one way of adapting the stateless I/O system would be to observe the current access pattern over some time and combine it with a tuning strategy.…”

Section: Motivation From Real-world Hpcmentioning

confidence: 99%

Adaptive request scheduling for the I/O forwarding layer using reinforcement learning

Bez

Boito

Nou

et al. 2020

Future Generation Computer Systems

View full text Add to dashboard Cite

I/O optimization techniques such as request scheduling can improve performance mainly for the access patterns they target, or they depend on the precise tune of parameters. In this paper, we propose an approach to adapt the I/O forwarding layer of HPC systems to the application access patterns by tuning a request scheduler. Our case study is the TWINS scheduling algorithm, where performance improvements depend on the time window parameter, which depends on the current workload. Our approach uses a reinforcement learning technique-contextual bandits-to make the system capable of learning the best parameter value to each access pattern during its execution, without a previous training phase. We evaluate our proposal and demonstrate it can achieve a precision of 88% on the parameter selection in the first hundreds of observations of an access pattern. After having observed an access pattern for a few minutes (not necessarily contiguously), we demonstrate that the system will be able to optimize its performance for the rest of the life of the system (years).

show abstract

Section: Motivation From Real-world Hpcmentioning

confidence: 99%

Adaptive request scheduling for the I/O forwarding layer using reinforcement learning

Bez

Boito

Nou

et al. 2020

Future Generation Computer Systems

View full text Add to dashboard Cite

show abstract

“…Another similar approach is ASCAR [20], that uses traffic controllers on storage clients to detect I/O congestion and introduces traffic rules to reduce it. AIS [22] is an I/O-aware scheduler that performs an offline analysis of the I/O traffic for identifying I/O characteristics of the applications. Based on that, the applications are scheduled avoiding I/O conflicts.…”

Section: Related Workmentioning

confidence: 99%

Mapping and scheduling HPC applications for optimizing I/O

Carretero

Jeannot

Pallez

et al. 2020

Proceedings of the 34th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

In HPC platforms, concurrent applications are sharing the same file system. This can lead to conflicts, especially as applications are more and more data intensive. I/O contention can represent a performance bottleneck. The access to bandwidth can be split in two complementary yet distinct problems. The mapping problem and the scheduling problem. The mapping problem consists in selecting the set of applications that are in competition for the I/O resource. The scheduling problem consists then, given I/O requests on the same resource, in determining the order to these accesses to minimize the I/O time. In this work we propose to couple a novel bandwidth-aware mapping algorithm to I/O list-scheduling policies to develop a cross-layer optimization solution.We study this solution experimentally using an I/O middleware: CLARISSE. We show that naive policies such as FIFO perform relatively well in order to schedule I/O movements, and that the important part to reduce congestion lies mostly on the mapping part. We evaluate the algorithm that we propose using a simulator that we validated experimentally. This evaluation shows important gains for the simple, bandwidth-aware mapping solution that we provide compared to its non bandwidth-aware counterpart. The gains are both in terms of machine efficiency (makespan) and application efficiency (stretch). This stresses even more the importance of designing efficient, bandwidth-aware mapping strategies to alleviate the cost of I/O congestion.

show abstract

“…Inúmeros trabalhos caracterizam E/S como os de [Zoll et al 2010,Liu et al 2016, porém, diferentemente destes, neste trabalho propomos uma abordagem de caracterização, não somente uma análise dos dados. Outros trabalhos como [Di et al 2014, Calzarossa et al 2016] aplicam abordagens de aprendizado não supervisionado na caracterização de jobs similares, porém não abordam a E/S considerando os padrões de acessos.…”

Section: Trabalhos Relacionados E Metodologiaunclassified

Caracterização do Comportamento de E/S Utilizando Aprendizado não Supervisionado

Pavan¹,

Bez²,

Serpa³

et al. 2020

Anais Da Escola Regional De Alto Desempenho Da Região Sul (ERAD RS 2020)

View full text Add to dashboard Cite

Nas aplicações de HPC operações de E/S são gargalos devido a diferença entre velocidade de processamento e de acesso a um dado. Assim, caracterizar as operações pode auxiliar na busca de desempenho. Deste modo, este trabalho propõe uma abordagem com aprendizado não supervisionado para a caracterização de E/S. Utilizando dados do supercomputador Intrepid, fomos capazes de identificar a principal característica de suas aplicações.

show abstract

Server-Side Log Data Analytics for I/O Workload Characterization and Coordination on Large Shared Storage Systems

Cited by 35 publications

References 48 publications

Adaptive request scheduling for the I/O forwarding layer using reinforcement learning

Adaptive request scheduling for the I/O forwarding layer using reinforcement learning

Mapping and scheduling HPC applications for optimizing I/O

Caracterização do Comportamento de E/S Utilizando Aprendizado não Supervisionado

Contact Info

Product

Resources

About