Optimizing Convolutional Neural Networks on the Sunway TaihuLight Supercomputer

Zhao, Wenlai; Fu, Haohuan; Fang, Jiayuan; Zheng, Weijie; Gan, Lin; Yang, Guangwen

doi:10.1145/3177885

Cited by 20 publications

(15 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In the U pdate step, we first accumulate the c l j and count l j of all CPEs by performing two AllReduce operations, so that all CPEs can obtain the assignment results of the whole input dataset. We use register communication [43] to implement intra-CG AllReduce operation and use MPI AllReduce for inter-CG AllReduce. After the accumulation, the U pdate step is performed to calculate new centroids, as shown in line 15.…”

Section: Level 1 -Dataflow Partitionmentioning

confidence: 99%

Large-Scale Automatic K-Means Clustering for Heterogeneous Many-Core Supercomputer

Zhao

Liu

et al. 2020

IEEE Trans. Parallel Distrib. Syst.

Self Cite

View full text Add to dashboard Cite

This paper presents an automatic k-means clustering solution targeting the Sunway TaihuLight supercomputer. We first introduce a multi-level parallel partition approach that not only partitions by dataflow and centroid, but also by dimension, which unlocks the potential of the hierarchical parallelism in the heterogeneous many-core processor and the system architecture of the supercomputer. The parallel design is able to process large-scale clustering problems with up to 196,608 dimensions and over 160,000 targeting centroids, while maintaining high performance and high scalability. Furthermore, we propose an automatic hyper-parameter determination process for k-means clustering, by automatically generating and executing the clustering tasks with a set of candidate hyper-parameter, and then determining the optimal hyper-parameter using a proposed evaluation method. The proposed auto-clustering solution can not only achieve high performance and scalability for problems with massive high-dimensional data, but also support clustering without sufficient prior knowledge for the number of targeted clusters, which can potentially increase the scope of k-means algorithm to new application areas.

show abstract

Section: Level 1 -Dataflow Partitionmentioning

confidence: 99%

Large-Scale Automatic K-Means Clustering for Heterogeneous Many-Core Supercomputer

Zhao

Liu

et al. 2020

IEEE Trans. Parallel Distrib. Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…There're a few works exploiting architectural features on Sunway, e.g., heterogeneous computing cores, SIMD, register-level communication, SPM, and so on, which are either hand-tuned application-specific implementations [3,8,19,61], or domain-specific frameworks [18,36,75]. Specially, [38,62,76] perform hand-tuned tiling for parallelism.…”

Section: Related Workmentioning

confidence: 99%

Bandwidth-Aware Loop Tiling for DMA-Supported Scratchpad Memory

Liu

Cui

et al. 2020

Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques

View full text Add to dashboard Cite

Scratchpad Memory (SPM) is widely used in emerging domain-specific architectures and accelerators for improving energy efficiency and time predictability. Typically, SPM-based architectures use DMA for fetching data from off-chip memory and global load instructions for loading fine-grained data directly into registers. For such architectures, neither capacity-only nor bandwidthonly loop tiling can efficiently use the bandwidth and SPM. This paper introduces a bandwidth-aware loop tiling approach that enables a tradeoff between SPM space utilization and bandwidth utilization to be made, by leveraging a runtime tiling framework and a cross-host-kernel IPA. Experimental results demonstrate that our approach can achieve the performance improvement of up to 4x, with a geometric average of 26%.

show abstract

“…Performing reduction tree operations is thus both more efficient and scalable than the traditional parameter server approach. Several prior works [8], [20]- [23] all implement 'allreduce' operations, customized by cluster interconnect features, to optimize the transmission process.…”

Section: Communication Optimizationmentioning

confidence: 99%

An Efficient Method for Training Deep Learning Networks Distributed

Wang

Chen

et al. 2020

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Training deep learning (DL) is a computationally intensive process; as a result, training time can become so long that it impedes the development of DL. High performance computing clusters, especially supercomputers, are equipped with a large amount of computing resources, storage resources, and efficient interconnection ability, which can train DL networks better and faster. In this paper, we propose a method to train DL networks distributed with high efficiency. First, we propose a hierarchical synchronous Stochastic Gradient Descent (SGD) strategy, which can make full use of hardware resources and greatly increase computational efficiency. Second, we present a two-level parameter synchronization scheme which can reduce communication overhead by transmitting parameters of the first layer models in shared memory. Third, we optimize the parallel I/O by making each reader read data as continuously as possible to avoid the high overhead of discontinuous data reading. At last, we integrate the LARS algorithm into our system. The experimental results demonstrate that our approach has tremendous performance advantages relative to unoptimized methods. Compared with the native distributed strategy, our hierarchical synchronous SGD strategy (HSGD) can increase computing efficiency by about 20 times.

show abstract

Optimizing Convolutional Neural Networks on the Sunway TaihuLight Supercomputer

Cited by 20 publications

References 16 publications

Large-Scale Automatic K-Means Clustering for Heterogeneous Many-Core Supercomputer

Large-Scale Automatic K-Means Clustering for Heterogeneous Many-Core Supercomputer

Bandwidth-Aware Loop Tiling for DMA-Supported Scratchpad Memory

An Efficient Method for Training Deep Learning Networks Distributed

Contact Info

Product

Resources

About