RoHNAS: A Neural Architecture Search Framework With Conjoint Optimization for Adversarial Robustness and Hardware Efficiency of Convolutional and Capsule Networks

Abstract

Neural Architecture Search (NAS) algorithms aim at finding efficient Deep Neural Network (DNN) architectures for a given application under given system constraints. DNNs are computationallycomplex as well as vulnerable to adversarial attacks. In order to address multiple design objectives, we propose RoHNAS, a novel NAS framework that jointly optimizes for adversarial-robustness and hardwareefficiency of DNNs executed on specialized hardware accelerators. Besides the traditional convolutional DNNs, RoHNAS additionally accounts for complex types of DNNs such as Capsule Networks. For reducing the exploration time, RoHNAS analyzes and selects appropriate values of adversarial perturbation for each dataset to employ in the NAS flow. Extensive evaluations on multi -Graphics Processing Unit (GPU) -High Performance Computing (HPC) nodes provide a set of Pareto-optimal solutions, leveraging the tradeoff between the above-discussed design objectives. For example, a Pareto-optimal DNN for the CIFAR-10 dataset exhibits 86.07% accuracy, while having an energy of 38.63 mJ, a memory footprint of 11.85 MiB, and a latency of 4.47 ms.

Neural architecture search for resource constrained hardware devices: A survey

“…In addition, the macro architecture of the same network type also varies. For example, NASCaps [26] changed its macro architecture to allow cells to be defined. Ref.…”

“…Finally, the fitness of the offspring is assessed and the new generation is updated by adding the optimal mutations to the population. Some researchers have used evolutionary algorithms to integrate hardware constraints into NAS algorithms [26,33,42,43].…”

“…NASCaps [26] proposed a NAS framework for generative cellular networks and CNNs that uses a multi-objective evolutionary algorithm based on NSGA-II to obtain Pareto solutions.…”

U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search

“…Thus, researchers have put considerable effort into developing effective neural architecture search methods to help automate the design of DNNs. Early work on neural architecture search adopted frameworks like reinforcement learning [10,12,17,18], evolutionary algorithms [9,11], and Bayesian optimization [2]. Because these methods operate on a discrete search space and need to perform many trials while searching for an optimal architecture in an exponentially-increasing hyperparameter space, they require thousands of GPU-hours to find optimal DNN architectures, which greatly limits their applicability.…”

“…The early versions of neural architecture search frameworks have resorted to reinforcement learning [10,12,17,18], evolutionary algorithms [9,11], and Bayesian optimization [2] to search for optimal DNN architectures. Unfortunately, these methods have time and space complexities that increase combinatorially with the number of options that are defined in the search space, requiring excessive amounts of computational resources.…”