On Performance Resilient Scheduling for Scientific Workflows in HPC Systems with Constrained Storage Resources

Abstract

Although the storage capacity is rapidly increasing, the size of datasets is also ever-growing, especially for those workflows in HPC that perform the parameter sweep studies. Consequently, the deadlock caused by the storage competition between concurrent workflow instances is still a major pragmatic concern and storage management remains important for high performance and throughput computing. In practice, there are various ways to this issue, ranging from admission control to deadlock resolution. Despite being a simple solution, the admission control is conservative and not space efficient to storage utilization. Therefore, in this paper, we study the performance of the deadlock resolution approach by proposing a resource allocation algorithm which is performance resilient to the workflows characterized by different features. The algorithm is designed based on our previous result, called DDS, which takes advantages of the dataflow information of the workflow to resolve deadlock based on detection&recovery principle. We improve DDS to allow it to not only resolve the deadlock but also overcome the performance anomaly, a not yet investigated problem in our previous studies. We thus called the improved algorithm performance-resilience algorithm, denoted as DDS + . The studies in this paper can be viewed as a follow-up research on DDS and show the performance behavior of the improved algorithm in various conditions. Therefore, the results in this paper are more useful to adapt DDS + to the workflows with different characteristics in reality while keeping the performance stable.