Low Latency Geo-distributed Data Analytics

PuQifan,; AnanthanarayananGanesh,; BodikPeter,; KandulaSrikanth,; AkellaAditya,; BahlParamvir,; StoicaIon,

doi:10.1145/2829988.2787505

Cited by 141 publications

(37 citation statements)

References 38 publications

(14 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The benefits of cloud computing are seen when the data reach the data centre and can be analysed. However, transmission of data has a high monetary cost, and can suffer from both excessive delays and security concerns [31]. This is where edge computing can step in.…”

Section: • Cloud and Edge Computingmentioning

confidence: 99%

Digital Twins in Civil Infrastructure Systems

et al. 2021

View full text Add to dashboard Cite

This research explores the existing definitions, concepts and applications surrounding the efficient implementation and use of digital twins (DTs) within civil infrastructure systems (CISs). The CISs within the scope of this research are as follows: transportation, energy, telecommunications, water and waste, as well as Smart Cities, which encompasses all of the previous. The research methodology consists of a review of current literature, a series of semi-structured interviews and a detailed survey. The outcome of this work is a refined definition of DTs within CISs, in addition to a set of recommendations for both future academic research and industry best practice.

show abstract

Section: • Cloud and Edge Computingmentioning

confidence: 99%

Digital Twins in Civil Infrastructure Systems

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Recently, scientific workflow task scheduling and data placement in distributed cloud datacenters has garnered much attention . Lin and Wu formulated a task‐scheduling problem to minimize the workflow end‐to‐end delay under a user‐specified financial constraint .…”

Section: Related Workmentioning

confidence: 99%

Graph partition–based data and task co‐scheduling of scientific workflow in geo‐distributed datacenters

Zhang

Chen

Zhan

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary Most large‐scale scientific workflows take place in multiple collaborative datacenters for access to community‐wide resources, while adhering to each datacenter's non‐uniform resource limits. However, moving both initial input datasets with predetermined locations and intermediate datasets needing placement decisions across geo‐distributed datacenters hinders efficient execution of large‐scale data‐intensive scientific workflows. Thus, scientific workflow's data and task co‐scheduling deal with situations such as pre‐placed initial input datasets, placement of intermediate datasets and each datacenter's non‐uniform computation and storage constraint, while minimizing the cross‐datacenter data transfer. Since this scheduling problem is known to be NP‐hard, here, we propose a novel approach, based on the multilevel graph coarsening and uncoarsening framework, together with a specialized hybrid genetic algorithm having distinctive graph partition driven features of repair and local improvement, for scheduling data‐intensive scientific workflows in geo‐distributed datacenters and optimizing the cross‐datacenter data transfer volume. Extensive simulations, based on four real‐world workflow traces, show that our algorithm significantly reduces the overall geo‐distributed data transfer and demonstrate its effectiveness.

show abstract

“…Collecting all data needed from different sites and then executing jobs at a central location would involve unacceptable time costs in data transmission. Hence, distributed data analysis jobs that could execute close to the input data receive attention recently [1,2]. Job execution requires system resources.…”

Section: Introductionmentioning

confidence: 99%

Towards Distributed Lexicographically Fair Resource Allocation with an Indivisible Constraint

Wan

Han

et al. 2022

Mathematics

View full text Add to dashboard Cite

In the cloud computing and big data era, data analysis jobs are usually executed over geo-distributed data centers to make use of data locality. When there are not enough resources to fully meet the demands of all the jobs, allocating resources fairly becomes critical. Meanwhile, it is worth noting that in many practical scenarios, resources waiting to be allocated are not infinitely divisible. In this paper, we focus on fair resource allocation for distributed job execution over multiple sites, where resources allocated each time have a minimum requirement. Aiming at the problem, we propose a novel scheme named Distributed Lexicographical Fairness (DLF) targeting to well specify the meaning of fairness in the new scenario considered. To well study DLF, we follow a common research approach that first analyzes its economic properties and then proposes algorithms to output concrete DLF allocations. In our study, we leverage a creative idea that transforms DLF equivalently to a special max flow problem in the integral field. The transformation facilitates our study in that by generalizing basic properties of DLF from the view of network flow, we prove that DLF satisfies Pareto efficiency, envy-freeness, strategy-proofness, relaxed sharing incentive and 12-maximin share. After that, we propose two algorithms. One is a basic algorithm that stimulates a water-filling process. However, our analysis shows that the time complexity is not strongly polynomial. Aiming at such inefficiency, we then propose a new iterative algorithm that comprehensively leverages parametric flow and push-relabel maximal flow techniques. By analyzing the steps of the iterative algorithm, we show that the time complexity is strongly polynomial.

show abstract

Low Latency Geo-distributed Data Analytics

Cited by 141 publications

References 38 publications

Digital Twins in Civil Infrastructure Systems

Digital Twins in Civil Infrastructure Systems

Graph partition–based data and task co‐scheduling of scientific workflow in geo‐distributed datacenters

Towards Distributed Lexicographically Fair Resource Allocation with an Indivisible Constraint

Contact Info

Product

Resources

About