HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark

Li, Chaojian; Yu, Zhongzhi; Fu, Yonggan; Zhang, Yongan; Zhao, Yang; You, Haoran; Yu, Qixuan; Wang, Yue; Lin, Yingyan

doi:10.48550/arxiv.2103.10584

Cited by 12 publications

(21 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also note a few limitations of our work. We have performed experiments over 14 NAS tasks, but there are new benchmarks that have been published recently [37,48,41,31]. We have adopted a conservative multi-fidelity technique, but less conservative multi-fidelity techniques are desired as long as robustness can be maintained.…”

Section: Conclusion and Limitationsmentioning

confidence: 99%

B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search

Cho¹,

Shin²,

Rhee³

2022

Preprint

View full text Add to dashboard Cite

The early pioneering Neural Architecture Search (NAS) works were multi-trial methods applicable to any general search space. The subsequent works took advantage of the early findings and developed weight-sharing methods that assume a structured search space typically with pre-fixed hyperparameters. Despite the amazing computational efficiency of the weight-sharing NAS algorithms, it is becoming apparent that multi-trial NAS algorithms are also needed for identifying very high-performance architectures, especially when exploring a general search space. In this work, we carefully review the latest multi-trial NAS algorithms and identify the key strategies including Evolutionary Algorithm (EA), Bayesian Optimization (BO), diversification, input and output transformations, and lower fidelity estimation. To accommodate the key strategies into a single framework, we develop B 2 EA that is a surrogate assisted EA with two BO surrogate models and a mutation step in between. To show that B 2 EA is robust and efficient, we evaluate three performance metrics over 14 benchmarks with general and cellbased search spaces. Comparisons with state-of-the-art multi-trial algorithms reveal that B 2 EA is robust and efficient over the 14 benchmarks for three difficulty levels of target performance. The B 2 EA code is publicly available at https: //github.com/snu-adsl/BBEA.Preprint. Under review.

show abstract

Section: Conclusion and Limitationsmentioning

confidence: 99%

B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search

Cho¹,

Shin²,

Rhee³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…overclocking, memory swapping, power usage maximization). Similar to HW-NAS-Bench [11], MAPLE-Edge leverages this domain knowledge to build an optimized, generic hardwarecost collection pipeline that automates the process of collecting latency measurements (as seen in Figure 1).…”

Section: Data Collection Pipelinementioning

confidence: 99%

“…Similar to MAPLE [1], BRP-NAS [7], HELP [10], and HW-NAS-Bench [11], MAPLE-Edge also uses the NAS-BENCH-201 [6] dataset for all experiments. NAS-BENCH-201 is a collection of 15,625 neural cell candidates with each architecture having a fixed cell topology with five possible operations in its search space including {none, skip-connection, conv1x1, conv3x3, avgpool3x3}.…”

Section: Edge Datasetmentioning

confidence: 99%

MAPLE-Edge: A Runtime Latency Predictor for Edge Devices

Nair¹,

Abbasi²,

Wong³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural Architecture Search (NAS) has enabled automatic discovery of more efficient neural network architectures, especially for mobile and embedded vision applications. Although recent research has proposed ways of quickly estimating latency on unseen hardware devices with just a few samples, little focus has been given to the challenges of estimating latency on runtimes using optimized graphs, such as TensorRT and specifically for edge devices. As devices like NVIDIA's Jetsons get more popular in embedded computing and robotics, we observe a pressing need to more accurately estimate inference latency of neural network architectures on diverse runtimes, including highly optimized ones. In this work, we propose MAPLE-Edge, an edge deviceoriented extension of MAPLE, the state-of-the-art latency predictor for general purpose hardware, where we train a regression network on architecture-latency pairs in conjunction with a hardware-runtime descriptor to effectively estimate latency on a diverse pool of edge devices. Compared to MAPLE, MAPLE-Edge can describe the runtime and target device platform using a much smaller set of CPU performance counters that are widely available on all Linux kernels, while still achieving up to +49.6% accuracy gains against previous state-of-the-art baseline methods on optimized edge device runtimes, using just 10 measurements from an unseen target device. We also demonstrate that unlike MAPLE which performs best when trained on a pool of devices sharing a common runtime, MAPLE-Edge can effectively generalize across runtimes by applying a trick of normalizing performance counters by the operator latency, in the measured hardware-runtime descriptor. Lastly, we show that for runtimes exhibiting lower than desired accuracy, performance can be boosted by collecting additional samples from the target device, with an extra 90 samples translating to gains of nearly +40%.

show abstract

“…For this reason, NAS-Bench-101 [19] and -201 [48] are proposed to solve the reproducibility issues and make a fair evaluation for NAS research. Recently, Hardware-NAS-Bench [72] was proposed to benchmark the inference latency and energy on several hardware devices. Despite the progress of benchmarks in NAS and other areas, model quantization lacks such standard to foster the reproducibility and deployability.…”

Section: Related Workmentioning

confidence: 99%

MQBench: Towards Reproducible and Deployable Model Quantization Benchmark

Li¹,

Shen²,

Ma³

et al. 2021

Preprint

View full text Add to dashboard Cite

Model quantization has emerged as an indispensable technique to accelerate deep learning inference. While researchers continue to push the frontier of quantization algorithms, existing quantization work is often unreproducible and undeployable. This is because researchers do not choose consistent training pipelines and ignore the requirements for hardware deployments. In this work, we propose Model Quantization Benchmark (MQBench), a first attempt to evaluate, analyze, and benchmark the reproducibility and deployability for model quantization algorithms. We choose multiple different platforms for real-world deployments, including CPU, GPU, ASIC, DSP, and evaluate extensive state-of-the-art quantization algorithms under a unified training pipeline. MQBench acts like a bridge to connect the algorithm and the hardware. We conduct a comprehensive analysis and find considerable intuitive or counter-intuitive insights. By aligning the training settings, we find existing algorithms have about the same performance on the conventional academic track. While for the hardware-deployable quantization, there is a huge accuracy gap which remains unsettled. Surprisingly, no existing algorithm wins every challenge in MQBench, and we hope this work could inspire future research directions.

show abstract

HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark

Cited by 12 publications

References 25 publications

B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search

B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization Modules for Neural Architecture Search

MAPLE-Edge: A Runtime Latency Predictor for Edge Devices

MQBench: Towards Reproducible and Deployable Model Quantization Benchmark

Contact Info

Product

Resources

About