Tale of Two Cs: Computation vs. Communication Scaling for Future Transformers on Future Hardware

T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives

“…For very large models (e.g., PALM, MT-NLG) we consider 32-way slicing, and for futuristic ones with one and ten trillion parameters, we consider 64-way sharding. The increasing TP slicing is necessary because these models' larger sizes cannot fit in 16 GPUs [64] and the increased slicing is also enabled by nodes with larger device counts [59,86]. Like prior work [36,49,64], we find that communication is a considerable fraction of the overall runtime: Megatron-GPT-2 (Mega-GPT-2) and T-NLG spend up to 34% and 43% of their training and inference (prompt phase) time on communication.…”

“…For larger Transformers like PALM [12], GPT-3 [9], and MT-NLG [80]) we use a higher slicing degree of 32 given their increasingly large memory capacity requirements [64] and availability of nodes with larger device counts that can enable this slicing [29,59,76]. We evaluate mixed-precision training which entails half-precision (FP16) forward and backpropagation and single-precision (FP32) weight updates.…”

“…These communication patterns are handled by collectives such as reduce-scatter, all-reduce, all-gather, all-to-all. While most of this communication can be hidden by independent compute operations [49,63,64], albeit with some resource contention [36,71], the all-reduce in TP is not (detailed in Section 2.4). Thus, we focus on all-reduce in TP and discuss other techniques/collectives in Sections 7.1 and 7.2.…”

“…Accordingly, their memory and computational demands have also increased, making them increasingly reliant on distributed techniques: multiple accelerators (e.g., GPUs) pooling their collective memory capacities and compute capabilities to collaboratively execute a DNN. However, the resulting communication between devices has become a significant proportion of their execution time and has limited the scaling efficiency with increasing device count [36,49,64].…”

“…Tensor-parallelism (TP), a type of MP, requires an all-reduce of layer outputs between devices as well. Among these distributed techniques, TP's communication typically lies on the critical path of model execution, as shown in Figure 1(a) and can be a significant proportion of runtime (∼45% [64]), resulting in a sub-linear increase in throughput as the number of devices increases.…”