Various distributed deep neural network (DNN) training technologies lead to increasingly complicated use of collective communications on GPU. The deadlock-prone collectives on GPU force researchers to guarantee that collectives are enqueued in a consistent order on each GPU to prevent deadlocks. In complex distributed DNN training scenarios, manual hardcoding is the only practical way for deadlock prevention, which poses significant challenges to the development of artificial intelligence. This paper presents OCCL, which is, to the best of our knowledge, the first deadlock-free collective communication library for GPU supporting dynamic decentralized preemption and gang-scheduling for collectives. Leveraging the preemption opportunity of collectives on GPU, OCCL dynamically preempts collectives in a decentralized way via the deadlock-free collective execution framework and allows dynamic decentralized gang-scheduling via the stickiness adjustment scheme. With the help of OCCL, researchers no longer have to struggle to get all GPUs to launch collectives in a consistent order to prevent deadlocks. We implement OCCL with several optimizations and integrate OCCL with a distributed deep learning framework OneFlow. Experimental results demonstrate that OCCL achieves comparable or better latency and bandwidth for collectives compared to NCCL, the state-of-the-art. When used in distributed DNN training, OCCL can improve the peak training throughput by up to 78% compared to statically sequenced NCCL, while introducing overheads of less than 6.5% across various distributed DNN training approaches. INTRODUCTIONRecent years have witnessed that the number of the state-of-theart (SOTA) deep neural network (DNN) models' parameters grows much faster than a single GPU's memory capacity and computational power [1,14,48]. This entails distributed DNN training, which includes various techniques such as data parallelism [27, 47], tensor parallelism [4, 50, 56], and pipeline parallelism [19,32,33], as well as hybrid parallelism [4,33,45], etc. Collective communication plays a critical role in distributed DNN training.Widely used collectives on GPU are deadlock-prone [40] because preemption is ill-supported on GPUs and collectives work in a resource-holding and busy-looping way on GPUs. As a result, the only chance to prevent collective-related deadlocks in distributed DNN training is to guarantee that all collectives are invoked in a consistent order on each of the GPUs.