One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search

Abstract

Convolutional neural networks (CNNs) are used in numerous real-world applications such as vision-based autonomous driving and video content analysis. To run CNN inference on various target devices, hardware-aware neural architecture search (NAS) is crucial. A key requirement of efficient hardware-aware NAS is the fast evaluation of inference latencies in order to rank different architectures. While building a latency predictor for each target device has been commonly used in state of the art, this is a very time-consuming process, lacking scalability in the presence of extremely diverse devices. In this work, we address the scalability challenge by exploiting latency monotonicity --- the architecture latency rankings on different devices are often correlated. When strong latency monotonicity exists, we can re-use architectures searched for one proxy device on new target devices, without losing optimality. In the absence of strong latency monotonicity, we propose an efficient proxy adaptation technique to significantly boost the latency monotonicity. Finally, we validate our approach and conduct experiments with devices of different platforms on multiple mainstream search spaces, including MobileNet-V2, MobileNet-V3, NAS-Bench-201, ProxylessNAS and FBNet. Our results highlight that, by using just one proxy device, we can find almost the same Pareto-optimal architectures as the existing per-device NAS, while avoiding the prohibitive cost of building a latency predictor for each device.

A Semi-Decoupled Approach to Fast and Optimal Hardware-Software Co-Design of Neural Accelerators

“…As a result, we expect latency monotonicity to be satisfied in our problem. Additionally, beyond the findings in [18], we observe in our experiments that energy monotonicity also holds: if one architecture 𝑎 1 is more energyefficient than another architecture 𝑎 2 for one hardware choice, then it is very likely that 𝑎 1 is still more energy-efficient than 𝑎 2 for another hardware choice. Along with latency monotonicity, energy monotonicity will be later validated in our experiments.…”

“…Insights. The performance monotonicity leads to the following proposition, which generalizes the statement in [18] by considering both latency and energy monotonicity. We first note that, by solving the inner NAS problem under a set of latency and energy constraints in Eqns.…”

“…In [18], scalable hardware-aware NAS is proposed by utilizing latency monotonicity on various devices. Without considering energy consumption, a high SRCC (>0.9) for latency is needed to ensure that one proxy device's optimal architectures are still close to optimal on another device.…”

“…It has been shown in a recent hardware-aware NAS study [18] that the architectures' ranking orders in terms of inference latency are highly similar on different devices, with SRCCs often close to 0.9 or higher, especially among devices of the same platform (e.g., mobile phones). For example, if one architecture 𝑎 1 is faster than another architecture 𝑎 2 on one mobile phone, then it is very likely that 𝑎 1 is still faster than 𝑎 2 on another phone.…”

“…One reason is that architectures are typically either computing-bound or memorybound on devices of the same platform, which, by roofline analysis, results in similar rankings of their latencies [31]. Based on this property (a.k.a., latency monotonicity), it has been theoretically and empirically proved that the Pareto-optimal architectures on different devices are highly overlapping if not identical [18]. While the target hardware space chosen by the designer has many choices, it essentially covers one platform -neural accelerator under a set of hardware constraints.…”