Single Path One-Shot Neural Architecture Search with Uniform Sampling

Guo, Zichao; Zhang, Xiangyu; Mu, Haoyuan; Wen, Hui; Liu, Zechun; Wei, Yichen; Sun, Jian

doi:10.1007/978-3-030-58517-4_32

Cited by 440 publications

(513 citation statements)

References 16 publications

Supporting

Mentioning

513

Contrasting

Order By: Relevance

“…It explores a layer-wise space where each layer of the CNN can choose a different block, and the learning process is done by training the stochastic super net. The SPOS [163] uses in a similar way the supernet concept to perform NAS, where the constraints such as latency and number of FLOPs are applied. The HURRICANE framework [164] performs a two-stage search algorithm for the automatic hardware-aware NAS.…”

Section: E Hardware-aware Neural Architecture Searchmentioning

confidence: 99%

Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead

et al. 2020

View full text Add to dashboard Cite

Currently, Machine Learning (ML) is becoming ubiquitous in everyday life. Deep Learning (DL) is already present in many applications ranging from computer vision for medicine to autonomous driving of modern cars as well as other sectors in security, healthcare, and finance. However, to achieve impressive performance, these algorithms employ very deep networks, requiring a significant computational power, both during the training and inference time. A single inference of a DL model may require billions of multiply-and-accumulated operations, making the DL extremely compute-and energy-hungry. In a scenario where several sophisticated algorithms need to be executed with limited energy and low latency, the need for cost-effective hardware platforms capable of implementing energy-efficient DL execution arises. This paper first introduces the key properties of two brain-inspired models like Deep Neural Network (DNN), and Spiking Neural Network (SNN), and then analyzes techniques to produce efficient and high-performance designs. This work summarizes and compares the works for four leading platforms for the execution of algorithms such as CPU, GPU, FPGA and ASIC describing the main solutions of the state-of-the-art, giving much prominence to the last two solutions since they offer greater design flexibility and bear the potential of high energy-efficiency, especially for the inference process. In addition to hardware solutions, this paper discusses some of the important security issues that these DNN and SNN models may have during their execution, and offers a comprehensive section on benchmarking, explaining how to assess the quality of different networks and hardware systems designed for them.

show abstract

Section: E Hardware-aware Neural Architecture Searchmentioning

confidence: 99%

Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Because inefficient search strategies require a large number of GPUs, many NAS methods cannot be implemented given limited computational resources. To address these challenges, much recent work dedicates to developing effective methods which can reduce the computational costs of performance evaluation, e.g., surrogate-assisted evolutionary algorithms (SAEAs) [33,42,43], information reuse [44,45], one-shot neural architecture search [46][47][48][49][50], among many others.…”

Section: Neural Architecture Searchmentioning

confidence: 99%

“…Therefore, various techniques have been suggested to accelerate the fitness evaluation, such as information reuse [44,47] and SAEAs [88]. SAEAs have been popular to solve computationally expensive optimization problems, which use cheap classification and regression models, e.g., radial basis function networks (RBFNs) [89,90] and Gaussian process (GP) models [91,92], to replace the expensive fitness evaluation [93].…”

Section: Nas Based On Easmentioning

confidence: 99%

“…Knowledge inheritance [44,47] is another promising approach to accelerate fitness evaluations. Zhang et al [44] propose an EA based on asexual reproduction to find better typologies for deep CNNs and knowledge inheritance to reduce the computation cost.…”

Section: Nas Based On Easmentioning

confidence: 99%

“…To reduce the memory usage of local devices, a more light-weighted real-time evolutionary NAS framework (RT-FedEvoNAS) is proposed in [120]. Different from the previous gradient-based approach, RT-FedEvoNAS adopts model sampling technique [47,122] in federated learning, where only one path of repeated cells in the global model is sampled and downloaded to local clients. As a result, both local computation and uploading costs are significantly reduced.…”

Section: For Each Communication Round Every Group Of Clientsmentioning

confidence: 99%

See 2 more Smart Citations

From federated learning to federated neural architecture search: a survey

Zhu

Zhang

Jin

2021

Complex Intell. Syst.

View full text Add to dashboard Cite

Federated learning is a recently proposed distributed machine learning paradigm for privacy preservation, which has found a wide range of applications where data privacy is of primary concern. Meanwhile, neural architecture search has become very popular in deep learning for automatically tuning the architecture and hyperparameters of deep neural networks. While both federated learning and neural architecture search are faced with many open challenges, searching for optimized neural architectures in the federated learning framework is particularly demanding. This survey paper starts with a brief introduction to federated learning, including both horizontal, vertical, and hybrid federated learning. Then neural architecture search approaches based on reinforcement learning, evolutionary algorithms and gradient-based are presented. This is followed by a description of federated neural architecture search that has recently been proposed, which is categorized into online and offline implementations, and single- and multi-objective search approaches. Finally, remaining open research questions are outlined and promising research topics are suggested.

show abstract

Angle-Based Search Space Shrinking for Neural Architecture Search

Liang²,

Guo³

et al. 2020

Computer Vision – ECCV 2020

View full text Add to dashboard Cite

Single Path One-Shot Neural Architecture Search with Uniform Sampling

Cited by 440 publications

References 16 publications

Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead

Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead

From federated learning to federated neural architecture search: a survey

Angle-Based Search Space Shrinking for Neural Architecture Search

Contact Info

Product

Resources

About