Submarine: A subscription‐based data streaming framework for integrating large facilities and advanced cyberinfrastructure

Zamani, Ali; AbdelBaky, Moustafa; Balouek-Thomert, Daniel; Villalobos, J. J.; Rodero, Iván; Parashar, Manish

doi:10.1002/cpe.5256

Cited by 5 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Analyzing concurrently multiple data streams with limited resources forces resource-quality trade-offs [25]- [27]. As the incoming data are processed concurrently, resources available to each data stream are often unknown.…”

Section: Configuration Adaptation For Edge-based Systemsmentioning

confidence: 99%

Enabling microservices management for Deep Learning applications across the Edge-Cloud Continuum

Houmani

Balouek-Thomert

Caron

et al. 2021

2021 IEEE 33rd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)

Self Cite

View full text Add to dashboard Cite

Deep Learning has shifted the focus of traditional batch workflows to data-driven feature engineering on streaming data. In particular, the execution of Deep Learning workflows presents expectations of near-real-time results with user-defined acceptable accuracy. Meeting the objectives of such applications across heterogeneous resources located at the edge of the network, the core, and in-between requires managing trade-offs between the accuracy and the urgency of the results. However, current data analysis rarely manages the entire Deep Learning pipeline along the data path, making it complex for developers to implement strategies in realworld deployments. Driven by an object detection use case, this paper presents an architecture for time-critical Deep Learning workflows by providing a data-driven scheduling approach to distribute the pipeline across Edge to Cloud resources. Furthermore, it adopts a data management strategy that reduces the resolution of incoming data when potential trade-off optimizations are available. We illustrate the system's viability through a performance evaluation of the object detection use case on the Grid'5000 testbed. We demonstrate that in a multi-user scenario, with a standard frame rate of 25 frames per second, the system speed-up data analysis up to 54.4% compared to a Cloud-only-based scenario with an analysis accuracy higher than a fixed threshold.

show abstract

Section: Configuration Adaptation For Edge-based Systemsmentioning

confidence: 99%

Enabling microservices management for Deep Learning applications across the Edge-Cloud Continuum

Houmani

Balouek-Thomert

Caron

et al. 2021

2021 IEEE 33rd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In Reference , the authors proposed a subscription‐based data streaming service. The service integrates: (i) a messaging system to integrate large facilities with advanced cyberinfrastructure services; and (ii) a broker for workflow orchestration.…”

Section: Classification Of the Cloud Service Interoperability Approachesmentioning

confidence: 99%

A survey on the service interoperability in cloud computing: Client‐centric and provider‐centric perspectives

2020

View full text Add to dashboard Cite

The vendor lock-in is a prominent issue in cloud computing. It is caused by cloud providers who offer proprietary services, which hinders the cloud interoperability. Client-centric interoperability enables the migration of the data and applications across clouds; it gives the clients control over their workloads and a wider range of service choices. Whereas, provider-centric interoperability allows the providers to collaborate. Thus, providers, who have spare resources, can lend them to other providers who lack computational or storage capabilities to overcome the limitations of their local resources. In this article, we conduct a survey to differentiate between client-and provider-centric interoperability solutions. We aim to provide an up-to-date analysis of the current tendencies and the neglected areas of the cloud interoperability field. Thus, we study the cloud service interoperability evolution through the years. Furthermore, we propose definitions for the intra-cloud and inter-cloud interoperability. Moreover, we propose a taxonomy to classify the cloud interoperability approaches into client-centric and provider-centric categories. Then, for each category, we classify the approaches based on their interoperability environment into single cloud or interconnected clouds. Finally, we analyze and compare the approaches based on multiple criteria. The study reveals the focus on the client-centric solutions and the interoperability in interconnected clouds. We notice more interest in the data and application levels interoperability, mainly, in infrastructure as a service model. We also find that client-centric solutions are, mostly, semantic technologies and brokers. However, provider-centric solutions are middleware, protocols, and standards. We conclude that a generic cloud service interoperability model is needed.

show abstract

“…Messaging system can span across various devices such as VMs, bare metal servers, network appliances, clusters, and storage units. Publish/subscribe method is also leveraged for data movement between geo-distributed resources forming our messaging system (Zamani et al, 2019). The messaging system and additional content delivery network (CDN) servers and network appliances are operating as a delivery system that brings content closer to the users at remote locations.…”

Section: Submarine-workflow Management For Data Adaptation and Deliverymentioning

confidence: 99%

Towards a computing continuum: Enabling edge-to-cloud integration for data-driven workflows

Balouek-Thomert

Renart

Zamani

et al. 2019

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

Dramatic changes in the technology landscape marked by increasing scales and pervasiveness of compute and data have resulted in the proliferation of edge applications aimed at effectively processing data in a timely manner. As the levels and fidelity of instrumentation increases and the types and volumes of available data grow, new classes of applications are being explored that seamlessly combine real-time data with complex models and data analytics to monitor and manage systems of interest. However, these applications require a fluid integration of resources at the edge, the core, and along the data path to support dynamic and data-driven application workflows, that is, they need to leverage a computing continuum. In this article, we present our vision for enabling such a computing continuum and specifically focus on enabling edge-to-cloud integration to support data-driven workflows. The research is driven by an online data-driven tsunami warning use case that is supported by the deployment of large-scale national environment observation systems. This article presents our overall approach as well as current status and next steps.

show abstract

Submarine: A subscription‐based data streaming framework for integrating large facilities and advanced cyberinfrastructure

Cited by 5 publications

References 27 publications

Enabling microservices management for Deep Learning applications across the Edge-Cloud Continuum

Enabling microservices management for Deep Learning applications across the Edge-Cloud Continuum

A survey on the service interoperability in cloud computing: Client‐centric and provider‐centric perspectives

Towards a computing continuum: Enabling edge-to-cloud integration for data-driven workflows

Contact Info

Product

Resources

About