End-to-end Machine Learning using Kubeflow

George, Johnu; Saha, Amit

doi:10.1145/3493700.3493768

Cited by 8 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is designed to easily deploy ML workflows on a Kubernetes cluster. Kubeflow functionality echoes that of Kubernetes, as it aims to deploy, scale, and manage ML workloads [65]. Kubeflow also allows running automated ML tasks and supports hyperparameter tuning [66], thus supporting end-to-end ML workflows [67].…”

Section: Workflow Management Systemsmentioning

confidence: 99%

Autonomy and Intelligence in the Computing Continuum: Challenges, Enablers, and Future Directions for Orchestration

Kokkonen¹,

Lovén²,

Motlagh³

et al. 2022

Preprint

View full text Add to dashboard Cite

Future AI applications require performance, reliability, and privacy that the existing, cloud-dependant system architectures cannot provide. In this article, we study orchestration in the device-edge-cloud continuum, and focus on AI for edge, that is, the AI methods used in resource orchestration. We claim that to support the constantly growing requirements of intelligent applications in the device-edge-cloud computing continuum, resource orchestration needs to embrace edge AI and emphasize local autonomy and intelligence. To justify the claim, we provide a general definition for continuum orchestration, and look at how current and emerging orchestration paradigms are suitable for the computing continuum. We describe certain major emerging research themes that may affect future orchestration, and provide an early vision of an orchestration paradigm that embraces those research themes. Finally, we survey current key edge AI methods and look at how they may contribute into fulfilling the vision of future continuum orchestration.

show abstract

Section: Workflow Management Systemsmentioning

confidence: 99%

Autonomy and Intelligence in the Computing Continuum: Challenges, Enablers, and Future Directions for Orchestration

Kokkonen¹,

Lovén²,

Motlagh³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Given this landscape, any entity involved in the business of scaled computing will fall behind if these technological needs are not prioritized. 4 In cloud computing communities, machine learning workloads are also becoming increasingly important, [7][8][9][10] and the cloud container orchestration technology Kubernetes is becoming the de facto standard for orchestration of these workflows following its success orchestrating microservices. As of June of 2023, the Kubernetes project had over 74,000 contributors, making it the second largest open source project ever after Linux, and the "most widely used container orchestration platform in existence" (the CNCF project report).…”

Section: Introductionmentioning

confidence: 99%

The Flux Operator

Sochat,

Culquicondor,

Ojea

et al. 2024

F1000Res

View full text Add to dashboard Cite

Converged computing is an emerging area of computing that brings together the best of both worlds for high performance computing (HPC) and cloud-native communities. The economic influence of cloud computing and the need for workflow portability, flexibility, and manageability are driving this emergence. Navigating the uncharted territory and building an effective space for both HPC and cloud require collaborative technological development and research. In this work, we focus on developing components for the converged workload manager, the central component of batch workflows running in any environment. From the cloud we base our work on Kubernetes, the de facto standard batch workload orchestrator. From HPC the orchestrator counterpart is Flux Framework, a fully hierarchical resource management and graph-based scheduler with a modular architecture that supports sophisticated scheduling and job management. Bringing these managers together consists of implementing Flux inside of Kubernetes, enabling hierarchical resource management and scheduling that scales without burdening the Kubernetes scheduler. This paper introduces the Flux Operator – an on-demand HPC workload manager deployed in Kubernetes. Our work describes design decisions, mapping components between environments, and experimental features. We perform experiments that compare application performance when deployed by the Flux Operator and the MPI Operator and present the results. Finally, we review remaining challenges and describe our vision of the future for improved technological innovation and collaboration through converged computing.

show abstract