APQ: Joint Search for Network Architecture, Pruning and Quantization Policy

Abstract

We present APQ for efficient deep learning inference on resource-constrained hardware. Unlike previous methods that separately search the neural architecture, pruning policy, and quantization policy, we optimize them in a joint manner. To deal with the larger design space it brings, a promising approach is to train a quantization-aware accuracy predictor to quickly get the accuracy of the quantized model and feed it to the search engine to select the best fit. However, training this quantization-aware accuracy predictor requires collecting a large number of quantized model, accuracy pairs, which involves quantization-aware finetuning and thus is highly time-consuming. To tackle this challenge, we propose to transfer the knowledge from a fullprecision (i.e., fp32) accuracy predictor to the quantizationaware (i.e., int8) accuracy predictor, which greatly improves the sample efficiency. Besides, collecting the dataset for the fp32 accuracy predictor only requires to evaluate neural networks without any training cost by sampling from a pretrained once-for-all [3] network, which is highly efficient. Extensive experiments on ImageNet demonstrate the benefits of our joint optimization approach. With the same accuracy, APQ reduces the latency/energy by 2×/1.3× over MobileNetV2+HAQ [30,36]. Compared to the separate optimization approach (ProxylessNAS+AMC+HAQ [5,12,36]), APQ achieves 2.3% higher ImageNet accuracy while reducing orders of magnitude GPU hours and CO 2 emission, pushing the frontier for green AI that is environmentalfriendly. The code and video are publicly available.

Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead

“…In AMC [208] and [209], learning-based approaches are adopted to prune and quantize the models for algorithm-hardware co-design. In APQ [210], pruning and quantization are optimized jointly with the NN model avoiding any accuracy loss.…”

One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search

“…The exponentially large search space consisting of billions of or even more architectures renders NAS a very challenging task [15,40,41,43,45,47]. The key reason is that evaluating and ranking the architectures in terms of metrics of interest (e.g., accuracy and latency) can be extremely time-consuming.…”

“…The key reason is that evaluating and ranking the architectures in terms of metrics of interest (e.g., accuracy and latency) can be extremely time-consuming. As a result, many studies have been focused on reducing the cost 1 of training and evaluating the architecture accuracy, including reinforcement learning-based NAS with accuracy evaluated based on a small proxy dataset [52], differentiable NAS [45], one-shot or few-shot NAS [4,9,51], NAS assisted with an accuracy predictor [15,43], among many others.…”

Evolutionary Algorithms in Approximate Computing: A Survey

“…The evolutionary design of neural networks, or neuroevolution, has recently led to the fully automated design of complex CNNs that are quite competitive in terms of accuracy and size, even for the most challenging datasets such as Im-ageNet [72]. In order to represent a candidate CNN in the genotype, a well-known CNN (such as MobileNetV2 in [73]) is usually taken as a template. The genotype then contains a set of parameters, each of them specifying possible values of the critical network's hyperparameters (the layer type, the number of filters, the kernel sizes, etc.).…”

“…In [74], the fixed-point quantization is applied as a postprocessing step after NAS is finished. However, for example, in APQ, a suitable quantization scheme is directly evolved during NAS [73]. APQ thus performs a joint search for architecture, pruning, and quantization policy starting with the MobileNetV2 network.…”