Exploiting Efficient and Scalable Shuffle Transfers in Future Data Center Networks

Abstract

Distributed computing systems like MapReduce in data centers transfer massive amount of data across successive processing stages. Such shuffle transfers contribute most of the network traffic and make the network bandwidth become a bottleneck. In many commonly used workloads, data flows in such a transfer are highly correlated and aggregated at the receiver side. To lower down the network traffic and efficiently use the available network bandwidth, we propose to push the aggregation computation into the network and parallelize the shuffle and reduce phases. In this paper, we first examine the gain and feasibility of the in-network aggregation with BCube, a novel server-centric networking structure for future data centers. To exploit such a gain, we model the in-network aggregation problem that is NP-hard in BCube. We propose two approximate methods for building the efficient IRS-based incast aggregation tree and SRS-based shuffle aggregation subgraph, solely based on the labels of their members and the data center topology. We further design scalable forwarding schemes based on Bloom filters to implement in-network aggregation over massive concurrent shuffle transfers. Based on a prototype and large-scale simulations, we demonstrate that our approaches can significantly decrease the amount of network traffic and save the data center resources. Our approaches for BCube can be adapted to other servercentric network structures for future data centers after minimal modifications.

Toward SLAs Guaranteed Scalable VDC Provisioning in Cloud Data Centers

“…In order to meet the increasing demands for services and better performance, the physical infrastructure should scale gracefully to accommodate concurrent jobs enabling incremental expansion without affecting the existing services. Correspondingly, the scheduling strategy should be scalable and can be easily adapted to the new expanded cloud services [18], [19].…”

“…where λ v and λ e are the recovery costs of virtual node v and virtual link e; and µvē = max{svē, dvē}. The cost for embedding e onto backbone network can be defined as in Formula (18).…”

Delay Bounded Multi-Source Multicast in Software-Defined Networking

“…The algorithm firstly create a delay table and then iteratively find the delay cost between the adjacent nodes (lines 1-9). The sequence table is then created with all the delay paths between any two nodes and the minimum delay (lines [11][12][13][14][15][16][17][18]. In Algorithm 1, the complexity of all the iterations is O(|V|)…”

“…Despite such bandwidth superiority, multicast has not been optimally utilized by the Internet during the past decades. Recently, it has achieved some successful network-level deployments in datacenter networks [9], enterprise networks and IPTV networks [10][11][12][13]. Research proposals dealing with the construction of multicasting trees that satisfy various constraints under single source scenario are abundant in the literature.…”

TIO: A VLC-enabled hybrid data center network architecture

“…Data centers are dominating infrastructures that support various cloud computing applications [1] . To satisfy bandwidth demand, modern data centers depend heavily on fiber-optic links [2] .…”