Migration of CMSWEB cluster at CERN to Kubernetes

Imran, Muhammad; Kuznetsov, Valentin; Marcella, Lina; Dziedziniewicz-Wojcik, Katarzyna Maria; Pfeiffer, A.; Paparrigopoulos, Panos

doi:10.22323/1.390.0911

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, we have performed the migration of the CMSWEB cluster to the Kubernetes [5]. In this paper, we give an overview of the CMSWEB services and VM cluster in Sect.…”

Section: Extended Author Information Available On the Last Page Of The Articlementioning

confidence: 99%

“…Availability zones are an end-user visible logical abstraction for partitioning a cloud without knowing the physical infrastructure. 5 This issue was related to infrastructure use and was subsequently fixed by experts in CERN IT. -Ceph mount issues A problem with CephFS mounts showed up after migrating clusters to the new Kubernetes version.…”

Section: Issues Faced During Migrationmentioning

confidence: 99%

See 1 more Smart Citation

Migration of CMSWEB cluster at CERN to Kubernetes: a comprehensive study

Imran

Kuznetsov

Dziedziniewicz-Wojcik

et al. 2021

Cluster Comput

Self Cite

View full text Add to dashboard Cite

The Compact Muon Solenoid (CMS) experiment heavily relies on the CMSWEB cluster to host critical services for its operational needs. The cluster is deployed on virtual machines (VMs) from the CERN OpenStack cloud and is manually maintained by operators and developers. The release cycle is composed of several steps, from building RPMs to their deployment, validation, and integration tests. To enhance the sustainability of the CMSWEB cluster, CMS decided to migrate its cluster to a containerized solution based on Docker and orchestrated with Kubernetes (K8s). This allows us to significantly speed up the release upgrade cycle, follow the end-to-end deployment procedure, and reduce operational cost. In this paper, we give an overview of the CMSWEB VM cluster and the issues we discovered during this migration. We discuss the architecture and the implementation strategy in the CMSWEB Kubernetes cluster. Even though Kubernetes provides horizontal pod autoscaling based on CPUs and memory, in this paper, we provide details of horizontal pod autoscaling based on the custom metrics of CMSWEB services. We also discuss automated deployment procedure based on the best practices of continuous integration/continuous deployment (CI/CD) workflows. We present performance analysis between Kubernetes and VM based CMSWEB deployments. Finally, we describe various issues found during the implementation in Kubernetes and report on lessons learned during the migration process.

show abstract

“…Recently, we have performed the migration of the CMSWEB cluster to the Kubernetes [5]. In this paper, we give an overview of the CMSWEB services and VM cluster in Sect.…”

Section: Extended Author Information Available On the Last Page Of The Articlementioning

confidence: 99%

Section: Issues Faced During Migrationmentioning

confidence: 99%

Migration of CMSWEB cluster at CERN to Kubernetes: a comprehensive study

Imran

Kuznetsov

Dziedziniewicz-Wojcik

et al. 2021

Cluster Comput

Self Cite

View full text Add to dashboard Cite

show abstract

“…The cluster was based on virtual machines (VMs) on the CERN OpenStack cloud infrastructure. Recently, we migrated CMSWEB VM cluster to Kubernetes (k8s) infrastructure to enhance the sustainability of CMSWEB services [2,3].…”

Section: Introductionmentioning

confidence: 99%

Evaluation and Implementation of Various Persistent Storage Options for CMSWEB Services in Kubernetes Infrastructure at CERN

Imran

Кузнецов

Paparrigopoulos

et al. 2023

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

This paper summarizes the various storage options that we implemented for the CMSWEB cluster in Kubernetes infrastructure. All CMSWEB services require storage for logs, while some services also require storage for data. We also provide a feasibility analysis of various storage options and describe the pros/cons of each technique from the perspective of the CMSWEB cluster and its users. In the end, we also propose recommendations according to the service needs. The first option is the CephFS which can be mounted multiple times across various clusters and VMs and works very well with k8s. We use it both for data and the logs. The second option is the Cinder volume. It is the block storage that runs the filesystem on top of it. It can only be attached to one instance at a time. We use this option only for the data. The third option is S3 storage. It is object storage that offers a scalable storage service that can be used by applications compatible with the Amazon S3 protocol. It is used for the logs. For S3, we explored two mechanisms. For the first scenario, we consider fluentd that runs as a sidecar container in the service pods and sends logs to S3 bucket. For the second scenario, we considered filebeat that runs as a sidecar container in the service pod and scaps those logs to fluentd which runs as a daemonset in each node and sends those logs to S3 in the end. The fourth option is EOS. We configured EOS inside the pods of the CMSWEB services. The fifth option that we explored is to use dedicated VMs that have Ceph volume attached to them. In EOS and VM, the logs from the service pods are sent to EOS/VM using the rsync approach. The last option is to send service logs to Elasticsearch. It has been implemented using fluentd that runs as a daemonset in each node. In parallel to the sending logs to S3 fluentd also sends those logs to the Elasticsearch infrastructure at CERN.

show abstract