Characterizing and Synthesizing Task Dependencies of Data-Parallel Jobs in Alibaba Cloud

Tian, Huangshi; Zheng, Yunchuan; Wang, Wei

doi:10.1145/3357223.3362710

Cited by 48 publications

(12 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…7 More precisely, 50% of functions run for less than 3 seconds, or p50=3 seconds for short, and p90=60 seconds [43]. Lifetime for containers are: p50=50 seconds and p90=1000 seconds [47]. VMs lifetimes are the longest: p50=900 seconds (15 mins) and p90=86,400 seconds (24 hours) [4].…”

Section: Discussionmentioning

confidence: 99%

Gridiron: A Technique for Augmenting Cloud Workloads with Network Bandwidth Requirements

Kodirov¹,

Bergsma²,

Iqbal³

et al. 2022

Preprint

View full text Add to dashboard Cite

Cloud applications use more than just server resources, they also require networking resources. We propose a new technique to model network bandwidth demand of networked cloud applications. Our technique, Gridiron, augments VM workload traces from Azure cloud with network bandwidth requirements. The key to the Gridiron technique is to derive inter-VM network bandwidth requirements using Amdahl's second law. As a case study, we use Gridiron to generate realistic traces with network bandwidth demands for a distributed machine learning training application. Workloads generated with Gridiron allow datacenter operators to estimate the network bandwidth demands of cloud applications and enable more realistic cloud resource scheduler evaluation.

show abstract

Section: Discussionmentioning

confidence: 99%

Gridiron: A Technique for Augmenting Cloud Workloads with Network Bandwidth Requirements

Kodirov¹,

Bergsma²,

Iqbal³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It is also simple and easy to implement in real time. This work uses the cluster data from Alibaba cloud to conduct experiments, which has been applied in some other relevant studies [53]- [55]. The experiments were conducted on the number of demanded containers counted from this dataset, and the results demonstrate the effectiveness and time cost problem of our algorithm when evaluating and comparing its performance with that of other algorithms.…”

Section: Discussionmentioning

confidence: 99%

An Adaptive Short-Term Prediction Algorithm for Resource Demands in Cloud Computing

Chen

Wang

2020

IEEE Access

View full text Add to dashboard Cite

Cloud computing has been widely applied in various fields with the development of big data and artificial intelligence. The associated resource demands exhibit characteristics such as diversity, large scale, burst and uncertainty. This paper analyzes these characteristics of cloud resource demands based on Alibaba cluster data, and proposes an adaptive short-term prediction algorithm for those demands. The proposed algorithm uses a principal component analysis method to extract the primary types of container demands from a time series of resource demands, and executes outlier detection and replacement to obtain a more stationary sequence. An adaptive short-term prediction strategy is proposed to adaptively select a higher-accuracy short-term prediction method to implement the prediction. Further, an error adjustment factor is proposed to reduce the prediction error. Thus, the short-term prediction accuracy of cloud resource demands is improved via outlier detection and replacement, an adaptive selection strategy and an error adjustment. We evaluated the effectiveness of these improvements, and compared our algorithm with existing algorithms in terms of effectiveness and time cost. The experimental results demonstrate that the proposed algorithm improves short-term prediction accuracy effectively.

show abstract

“…After individual cloud providers started to include information about the DAG of workloads in their traces, researchers also began to investigate the cluster traces with regards to the graph-structured task dependencies of workloads [16], [31]. Our work is most similar to theirs.…”

Section: B Cluster Traces From Cloud Providersmentioning

confidence: 93%

“…In [16], the authors discuss a methodology for identifying recurrent jobs. It is applied on the Alibaba trace dataset for gaining potential insights.…”

Section: Baselinementioning

confidence: 99%

On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds

Scheinert,

Alamgiralem,

Bader

et al. 2021

Preprint

View full text Add to dashboard Cite

With the growing amount of data, data processing workloads and the management of their resource usage becomes increasingly important. Since managing a dedicated infrastructure is in many situations infeasible or uneconomical, users progressively execute their respective workloads in the cloud. As the configuration of workloads and resources is often challenging, various methods have been proposed that either quickly profile towards a good configuration or determine one based on data from previous runs. Still, performance data to train such methods is often lacking and must be costly collected.In this paper, we propose a collaborative approach for sharing anonymized workload execution traces among users, mining them for general patterns, and exploiting clusters of historical workloads for future optimizations. We evaluate our prototype implementation for mining workload execution graphs on a publicly available trace dataset and demonstrate the predictive value of workload clusters determined through traces only.

show abstract

Characterizing and Synthesizing Task Dependencies of Data-Parallel Jobs in Alibaba Cloud

Cited by 48 publications

References 27 publications

Gridiron: A Technique for Augmenting Cloud Workloads with Network Bandwidth Requirements

Gridiron: A Technique for Augmenting Cloud Workloads with Network Bandwidth Requirements

An Adaptive Short-Term Prediction Algorithm for Resource Demands in Cloud Computing

On the Potential of Execution Traces for Batch Processing Workload Optimization in Public Clouds

Contact Info

Product

Resources

About