Data center TCP (DCTCP)

Abstract

Cloud data centers host diverse applications, mixing workloads that require small predictable latency with others requiring large sustained throughput. In this environment, today's state-of-the-art TCP protocol falls short. We present measurements of a 6000 server production cluster and reveal impairments that lead to high application latencies, rooted in TCP's demands on the limited buffer space available in data center switches. For example, bandwidth hungry "background" flows build up queues at the switches, and thus impact the performance of latency sensitive "foreground" traffic.To address these problems, we propose DCTCP, a TCP-like protocol for data center networks. DCTCP leverages Explicit Congestion Notification (ECN) in the network to provide multi-bit feedback to the end hosts. We evaluate DCTCP at 1 and 10Gbps speeds using commodity, shallow buffered switches. We find DCTCP delivers the same or better throughput than TCP, while using 90% less buffer space. Unlike TCP, DCTCP also provides high burst tolerance and low latency for short flows. In handling workloads derived from operational measurements, we found DCTCP enables the applications to handle 10X the current background traffic, without impacting foreground traffic. Further, a 10X increase in foreground traffic does not cause any timeouts, thus largely eliminating incast problems.

Measuring the state of ECN readiness in servers, clients,and routers

“…Long-lived heavy bandwidth flows hamper this ability by inducing large queues. Recent proposals based on an analysis of real-world data center traffic rely on ECN to better handle such workloads [2].…”

Harsh RED: Improving RED for Limited Aggregate Traffic

“…Too slow response is subject to both falling back to tail drop and can cause even more dramatic performance drop through the th max cutter phenomenon if the gap between th max and the physical queue size is wide enough. Making RED "too fast" just brings it closer to FIFO behavior (ultimately this would result in what DCTCP [20] does). As random dropping, proactive response and some burst allowance are still happening, too fast RED does not nullify good features of RED.…”

“…SRED [19] focuses on finding out the number of flows which is significant for TCP congestion avoidance mode but quite irrelevant with slow start because the aggregated window growth rate in slow start does not depend on number of flows. In addition, DataCenter TCP (DCTCP) [20] is a TCP variant that includes endhost modifications and uses RED implemented on routers in a degenerated fashion. Because of the need for end-host modifications, we find DCTCP unsuitable for other than isolated environments such as data-centers as otherwise fairness issues would arise.…”

Dahu: Commodity switches for direct connect data center networks

“…A smaller 64 port switch would only need a total of 352 KB for a similar number of port groups and virtual ports. This memory may come at the expense of additional packet buffers (typically around 10 MB); however, recent trends in data center congestion management [4,5] indicate that trading a small amount of buffer memory for more adaptive routing may be worthwhile.…”