Cooperative Job Scheduling and Data Allocation in Data-Intensive Parallel Computing Clusters

Wang, Haoyu; Liu, Guoxin; Shen, Haiying

doi:10.1109/tcc.2022.3206206

Cited by 3 publications

(1 citation statement)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…AI applications, ranging from complex machine learning algorithms to sophisticated deep learning models, have become integral components of numerous industries. [3] The ability of AI to extract meaningful insights from vast datasets has led to an exponential increase in computational requirements, prompting a paradigm shift in the architecture and management of cloud infrastructures.…”

mentioning

confidence: 99%

Efficient Workload Allocation and Scheduling Strategies for AI-Intensive Tasks in Cloud Infrastructures.

Ahmad Fawad

2023

pst

View full text Add to dashboard Cite

- The rapid proliferation of Artificial Intelligence (AI) applications has underscored the need for advanced cloud infrastructures capable of efficiently managing AI-intensive workloads. This paper delves into the intricacies of workload allocation and scheduling in the context of cloud environments, specifically focusing on the challenges posed by AI-intensive tasks. Our research endeavors to scrutinize existing strategies, discern their limitations, and proffer innovative approaches tailored to optimize the allocation and scheduling of AI workloads within cloud infrastructures. In elucidating the challenges, we pinpoint resource heterogeneity, dynamic workload characteristics, and scalability as the crux of the issues confronting AI-intensive workload management. The diverse computational demands of AI workloads make it challenging to allocate resources optimally, while the dynamic nature of these tasks necessitates adaptive strategies to accommodate varying computational requirements over time. [1] Additionally, as AI models and datasets burgeon in complexity and size, ensuring scalability becomes paramount for sustaining performance in cloud environments. Our literature review encompasses an examination of both traditional and state-of-the-art workload allocation strategies, shedding light on their respective strengths and shortcomings. We also delve into scheduling techniques employed for managing AI-intensive tasks, providing a comprehensive overview of the existing landscape. To address these challenges, we propose a novel framework centered around dynamic resource provisioning, machine learning-based scheduling, and efficient task migration strategies. The framework aims to adaptively allocate resources based on the evolving nature of AI workloads, leveraging machine learning algorithms to predict workload characteristics and employing efficient task migration to handle workload fluctuations. The paper concludes with an experimental evaluation of the proposed strategies, conducted in a simulated environment using diverse datasets. Key performance metrics, such as throughput, latency, and resource utilization, are employed to assess the effectiveness of our strategies compared to existing approaches. By offering insights into the efficient management of AI-intensive workloads in cloud infrastructures, this research contributes to the ongoing efforts to enhance the scalability and performance of cloud environments in the face of burgeoning AI applications.

show abstract

mentioning

confidence: 99%