A Coflow-Based Co-Optimization Framework for High-Performance Data Analytics

Cheng, Long; Wang, Ying; Pei, Yulong; Epema, Dick

doi:10.1109/icpp.2017.48

Cited by 12 publications

(6 citation statements)

References 29 publications

(61 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fu et al implemented the Seagull system to optimize the average task completion time while also providing deadlines for tasks with guarantees. Cheng et al raised a joint optimization framework for high performance data analysis applications based on Coflow. Jiang et al monitored the network status based on the SDN technology and rerouted the bottleneck flow in the task to the lightly loaded link to reduce the task completion time.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Rendering differential performance preference through intelligent network edge in cloud data centers

Shen

Zhou²,

Gao

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary Sharing the network infrastructure, the performance of emerging distributed applications and services in data centers is directly impacted by the network. As these applications are becoming more and more demanding, it is challenging to satisfy their requirements of low latency, high throughput, and low packet loss rate simultaneously. Prior approaches typically resort to flow control or scheduling mechanisms, prioritizing flows according to their demands. However, none of the methods can solely satisfy the various demands of data center applications. Addressing this challenge, we propose tasch, a preference aware flow scheduling mechanism equipped in the software network edge (ie, end‐host networking). This mechanism utilizes multiple separate queues for flows with different preferences, which guarantees low packet delay for latency‐sensitive flows and provides bandwidth guarantees for throughput‐sensitive flows. A coordinating algorithm is presented to share the network resource among multiple queues with pareto‐optimality. tasch is implemented as a thin and plugable kernel module in Linux based hypervisors, which lies between the complicated physical network and tenants VMs. Subsequently, based on the flow traces of real‐world applications, extensive experiments were conducted to verify the effectiveness of network management mechanism.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Fu et al 23 implemented the Seagull system to optimize the average task completion time while also providing deadlines for tasks with guarantees. Cheng et al 24 mechanism to satisfy the deadline requirements of flows in multi-resource environments.…”

Section: Of 14mentioning

confidence: 99%

Rendering differential performance preference through intelligent network edge in cloud data centers

Shen

Zhou²,

Gao

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…POPI [8] is a lightweight algorithm targeting for efficient processing large outer joins under Hadoop. In order to improve the performance in large distributed environments, a new framework called CCF [9] is proposed to co-optimize application-level data movement and network-level data communications for distributed operators. Rayon [10] is proposed to reserve resources for production jobs and best-effort jobs such that the SLAs for production jobs can be guaranteed and meanwhile the execution time of best-effort jobs can be reduced.…”

Section: 22mentioning

confidence: 99%

AutoAdmin: Automatic and Dynamic Resource Reservation Admission Control in Hadoop YARN Clusters

Yang¹,

Bhimani²,

Yao³

et al. 2018

SCPE

View full text Add to dashboard Cite

Hadoop YARN is an Apache Software Foundation's open project that provides a resource management framework for large scale parallel data processing, such as MapReduce jobs. Fair scheduler is a dispatcher which has been widely used in YARN to assign resources fairly and equally to applications. However, there exists a problem of the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. To solve this problem, we propose an automatic and dynamic admission control mechanism to prevent the ceasing situation happened when the requested amount of resources exceeds the cluster's resource capacity, and dynamically reserve resources for processing tasks in order to obtain good performance, e.g., reducing makespans of MapReduce jobs. After collecting resource usage information of each work node, our mechanism dynamically predicts the amount of reserved resources for processing tasks and automatically controls running jobs based on the prediction. We implement the new mechanism in Hadoop YARN and evaluate it with representative MapReduce benchmarks. The experimental results show the effectiveness and robustness of this mechanism under both homogeneous and heterogeneous workloads.

show abstract

“…Consequently, designing effective algorithms to do array redistribution is crucial for implementing distributed memory system software for these programming languages. In addition to array redistribution in parallel systems, in recent years, different data redistribution issues also received a lot of attention, including data redistribution in wireless sensor networks to minimize power consumption [5], data redistribution and retrieval in wireless sensor networks to preserve data [6], data redistribution in parallel join operations in distributed system [7], data distribution without breaking its semantics [8], data redistribution of data between processes [9], co-optimized application-level data movement and network-level data communications for distributed operators [10].…”

Section: Introductionmentioning

confidence: 99%

New Bipartite Graph Techniques for Irregular Data Redistribution Scheduling

2019

Algorithms

View full text Add to dashboard Cite

For many parallel and distributed systems, automatic data redistribution improves its locality and increases system performance for various computer problems and applications. In general, an array can be distributed to multiple processing systems by using regular or irregular distributions. Some data distribution adopts BLOCK, CYCLIC, or BLOCK-CYCLIC to specify data array decomposition and distribution. On the other hand, irregular distributions specify a different-size data array distribution according to user-defined commands or procedures. In this work, we propose three bipartite graph problems, including the “maximum edge coloring problem”, the “maximum degree edge coloring problem”, and the “cost-sharing maximum edge coloring problem” to formulate these kinds of distribution problems. Next, we propose an approximation algorithm with a ratio bound of two for the maximum edge coloring problem when the input graph is biplanar. Moreover, we also prove that the “cost-sharing maximum edge coloring problem” is an NP-complete problem even when the input graph is biplanar.

show abstract

A Coflow-Based Co-Optimization Framework for High-Performance Data Analytics

Cited by 12 publications

References 29 publications

Rendering differential performance preference through intelligent network edge in cloud data centers

Rendering differential performance preference through intelligent network edge in cloud data centers

AutoAdmin: Automatic and Dynamic Resource Reservation Admission Control in Hadoop YARN Clusters

New Bipartite Graph Techniques for Irregular Data Redistribution Scheduling

Contact Info

Product

Resources

About