In-Datacenter Performance Analysis of a Tensor Processing Unit

Abstract

Many architects believe that major improvements in cost-energyperformance must now come from domain-specific hardware. This paper evaluates a custom ASIC-called a Tensor Processing Unit (TPU)-deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile responsetime requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power. We compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU, which are contemporaries deployed in the same datacenters. Our workload, written in the high-level TensorFlow framework, uses production NN applications (MLPs, CNNs, and LSTMs) that represent 95% of our datacenters' NN inference demand. Despite low utilization for some applications, the TPU is on average about 15X -30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X -80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU.

Emerging Accelerator Platforms for Data Centers

“…In addition, 28MB of software-managed on-chip memory is included to store the intermediate results and the inputs of the Matrix Multiply Unit. The datapath occupies 67% of the TPU floorplan, while the area occupied for control is only 2% [17]. This contrasts with the state-of-theart server CPUs and GPUs, in which the control structures occupy significant chip area and lead to increased power consumption.…”

“…Based on a projection that voice-based search will significantly increase the computational demands of Google's datacenters, a custom ASIC chip -called Tensor Processing Unit (TPU) --was designed and deployed by Google in 2015 [17]. TPU is aimed at accelerating the inference phase of different types of neural network applications, including multi-layer perceptrons (MLP), convolutional neural networks (CNN), and recurrent neural networks (RNN) [18].…”

AI, native supercomputing and the revival of Moore's Law

“…Mesh topology is a strikingly popular way to organize PEs, for example, Google's TPU [12], the DianNao family [15], MIT's Eyeriss [16] (see Fig. 6).…”

A design of the ALU for Convolution Neural Network of operation processing

“…이를 극복하기 위해 합성곱(convolution) 처리를 위한 특별한 프로세서 연구개발이 진행 중이다 [4]. 그 가운데 구글의 TPU(Tensor Processing Unit)에서는 특정 기능만을 수행하여 연 산 속도를 개선시킨 것에 대한 연구결과를 발표했다 [5] [6]. 이에 본 논문에서 합성곱 및 pooling 연 산에서 곱셈과 덧셈을 빠르게 계산 할 수 있고 병렬처리가 가능한 ALU 연산기를 제안한다.…”