A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities

Abstract

Recent advances in Neural Architecture Search (NAS) such as one-shot NAS offer the ability to extract specialized hardware-aware sub-network configurations from a task-specific super-network. While considerable effort has been employed towards improving the first stage, namely, the training of the super-network, the search for derivative high-performing sub-networks is still under-explored. Popular methods decouple the super-network training from the sub-network search and use performance predictors to reduce the computational burden of searching on different hardware platforms. We propose a flexible search framework that automatically and efficiently finds optimal sub-networks that are optimized for different performance metrics and hardware configurations. Specifically, we show how evolutionary algorithms can be paired with lightly trained objective predictors in an iterative cycle to accelerate architecture search in a multi-objective setting for various modalities including machine translation and image classification.

Evolution and Efficiency in Neural Architecture Search: Bridging the Gap between Expert Design and Automated Optimization

“…Hardware-Aware NAS: Recent approaches like hardwareaware frameworks have focused on optimizing NAS not only for model accuracy but also for hardware efficiency. These methods use evolutionary algorithms paired with objective predictors to efficiently find optimized architectures for various performance metrics and hardware configurations [14]. 2.…”

Neural Architecture Search for Transformers: A Survey

“…Lightweight Iterative NAS [208] is designed to accelerate the subnetwork search phase post the Supernetwork training, as the validation component in One-shot NAS methods comes with a huge computational cost, especially on large datasets. Besides, the main contribution of LINAS is the a generalizable framework to offer support for various search methods and model performance predictors in a multi-objective setting across multimodal environments.…”