Cloud-agnostic architectures for machine learning based on Apache Spark

Abstract

Reference architectures for Big Data, machine learning and stream processing include not only recommended practices and interconnected building blocks but considerations for scalability, availability, manageability, and security as well. However, the automated deployment of multi-VM platforms on various clouds leveraging on such reference architectures may raise several issues. The paper focuses particularly on the widespread Apache Spark Big Data platform as the baseline and the Occopus cloud-agnostic orchestrator tool. The set of new generation reference architectures are configurable by human-readable descriptors according to available resources and cloud-providers, and offers various components such as Jupyter Notebook, RStudio, HDFS, and Kafka. These pre-configured reference architectures can be automatically deployed even by the data scientist on-demand, using a multi-cloud approach for a wide range of cloud systems like Amazon AWS, Microsoft Azure, Open-Stack, OpenNebula, CloudSigma, etc. Occopus enables the scaling of cluster-oriented components (such as Spark) of the instantiated reference architectures. The presented solution was successfully used in the Hungarian Comparative Agendas Project (CAP) by the Institute for Political Science to classify newspaper articles.

Toward a Quantum-Science Gateway: A Hybrid Reference Architecture Facilitating Quantum Computing Capabilities for Cloud Utilization

“…The goal of HUN-REN Cloud is supporting Hungarian scientists with elastic, virtualized computing resources. The lowlevel reference architectures offered by HUN-REN Cloud include amongst others blueprints for Horovod [32], Apache Spark [33], Jupyter development environments, and Kubernetes. For example the researchers can use these reference architectures to easily set up a Kubernetes cluster in HUN-REN Cloud, without the need to know the details of deployment or internal specifics.…”

“…In order to host this example, the reference architecture should include an Apache Spark cluster as well. For this we extended the existing Apache Spark reference architecture [33] provided by HUN-REN Cloud.…”

“…FIGURE 1.The Quantum -Science Gateway Hybrid Reference Architecture running on HUN-REN Cloud (components provided by the cloud are marked with yellow) utilizing the (i) Kubernetes Reference Architecture[64] (related components marked with blue grid); and extending the (ii) Apache Spark Reference Architecture[33] (related components marked with green stripes)…”

Fault detection in GPU-enabled Cloud Systems – An Overview

“…In this paper, we focused on the broad overview by referring to various materials and documentation. The effect of GPUs in fault detection and prediction in cloud systems should be analyzed in real-life monitoring processes, for example using the significantly upgraded ELKH Cloud infrastructure that has been serving a large number of machine learning and GPU accelerated scientific projects of the Hungarian research community [59]. The prepared overview opens new areas of research for investigation, and might serve some valuable contributions to the research related to debugging of orchestrated cloud applications [35].…”