Accelerating Neural Architecture Search via Proxy Data

Na, Byunggook; Mok, Jisoo; Choe, Hyeokjun; Yoon, Sungroh

doi:10.24963/ijcai.2021/392

Cited by 9 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The current state of the art in the automatic design of CSNN architectures are the works of Kim et al [19] and AutoSNN by Na et al [32]. Both works focus on Neural Architecture Search (NAS), with an evolutionary search component implemented in AutoSNN, and attain state-of-the-art performances in the CIFAR-10, CIFAR-100 [21], and TinyImageNet datasets.…”

Section: Related Workmentioning

confidence: 99%

SPENSER: Towards a NeuroEvolutionary Approach for Convolutional Spiking Neural Networks

Branquinho

Lourenço

Costa

2023

Proceedings of the Companion Conference on Genetic and Evolutionary Computation

View full text Add to dashboard Cite

Spiking Neural Networks (SNNs) have attracted recent interest due to their energy efficiency and biological plausibility. However, the performance of SNNs still lags behind traditional Artificial Neural Networks (ANNs), as there is no consensus on the best learning algorithm for SNNs. Best-performing SNNs are based on ANN to SNN conversion or learning with spike-based backpropagation through surrogate gradients. The focus of recent research has been on developing and testing different learning strategies, with hand-tailored architectures and parameter tuning. Neuroevolution (NE), has proven successful as a way to automatically design ANNs and tune parameters, but its applications to SNNs are still at an early stage. DENSER is a NE framework for the automatic design and parametrization of ANNs, based on the principles of Genetic Algorithms (GA) and Structured Grammatical Evolution (SGE). In this paper, we propose SPENSER, a NE framework for SNN generation based on DENSER, for image classification on the MNIST and Fashion-MNIST datasets. SPENSER generates competitive performing networks with a test accuracy of 99.42% and 91.65% respectively. CCS CONCEPTS• Computing methodologies → Neural networks; Computer vision; • Theory of computation → Evolutionary algorithms; Grammars and context-free languages.

show abstract

Section: Related Workmentioning

confidence: 99%

SPENSER: Towards a NeuroEvolutionary Approach for Convolutional Spiking Neural Networks

Branquinho

Lourenço

Costa

2023

Proceedings of the Companion Conference on Genetic and Evolutionary Computation

View full text Add to dashboard Cite

show abstract

“…EcoNAS [66] explores four common reduction factors -the number of channels, the resolution of input images, the number of training epochs, and the sample ratio of the full training set -and determine which one of these proxies can be used to reliably estimate the final test accuracy. Na et al [44] show that it is possible to only use a subset of the target dataset for execute NAS and propose a novel proxy dataset selection algorithm.…”

Section: Appendices A1 Related Work A11 Neural Architecture Searchmentioning

confidence: 99%

“…The family of architecture comparators replaces the deterministic evaluation of neural architectures with a relativistic approach that compares two architectures and determines which one yields better performance [14,60]. Apart from weight sharing and performance prediction, some works propose more general proxies for architecture evaluation [1,37,42,44,66]. White et al [56] offer a comprehensive survey of performance predictors in NAS, and in the Appendix, we discuss related works in more detail.…”

Section: Introductionmentioning

confidence: 99%

Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?

Mok¹,

Na²,

Kim³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In Neural Architecture Search (NAS), reducing the cost of architecture evaluation remains one of the most crucial challenges. Among a plethora of efforts to bypass training of each candidate architecture to convergence for evaluation, the Neural Tangent Kernel (NTK) is emerging as a promising theoretical framework that can be utilized to estimate the performance of a neural architecture at initialization. In this work, we revisit several at-initialization metrics that can be derived from the NTK and reveal their key shortcomings. Then, through the empirical analysis of the time evolution of NTK, we deduce that modern neural architectures exhibit highly non-linear characteristics, making the NTK-based metrics incapable of reliably estimating the performance of an architecture without some amount of training. To take such non-linear characteristics into account, we introduce Label-Gradient Alignment (LGA), a novel NTKbased metric whose inherent formulation allows it to capture the large amount of non-linear advantage present in modern neural architectures. With minimal amount of training, LGA obtains a meaningful level of rank correlation with the final test accuracy of an architecture. Lastly, we demonstrate that LGA, complemented with few epochs of training, successfully guides existing search algorithms to achieve competitive search performances with significantly less search cost. The code is available at: https:// github.com/nutellamok/DemystifyingNTK.

show abstract

“…Recently, Kim et al ( 2022 ) explored a cell-based neural architecture search method on SNNs, but did not involve large-scale recurrent connections. Na et al ( 2022 ) introduced a spike-aware NAS framework called AutoSNN to investigate the impact of architectural components on SNNs' performance and energy efficiency. Overall, NAS for RSNNs is still rarely explored.…”

Section: Introductionmentioning

confidence: 99%

Composing recurrent spiking neural networks using locally-recurrent motifs and risk-mitigating architectural optimization

Zhang,

Geng,

2024

Front. Neurosci.

View full text Add to dashboard Cite

In neural circuits, recurrent connectivity plays a crucial role in network function and stability. However, existing recurrent spiking neural networks (RSNNs) are often constructed by random connections without optimization. While RSNNs can produce rich dynamics that are critical for memory formation and learning, systemic architectural optimization of RSNNs is still an open challenge. We aim to enable systematic design of large RSNNs via a new scalable RSNN architecture and automated architectural optimization. We compose RSNNs based on a layer architecture called Sparsely-Connected Recurrent Motif Layer (SC-ML) that consists of multiple small recurrent motifs wired together by sparse lateral connections. The small size of the motifs and sparse inter-motif connectivity leads to an RSNN architecture scalable to large network sizes. We further propose a method called Hybrid Risk-Mitigating Architectural Search (HRMAS) to systematically optimize the topology of the proposed recurrent motifs and SC-ML layer architecture. HRMAS is an alternating two-step optimization process by which we mitigate the risk of network instability and performance degradation caused by architectural change by introducing a novel biologically-inspired “self-repairing” mechanism through intrinsic plasticity. The intrinsic plasticity is introduced to the second step of each HRMAS iteration and acts as unsupervised fast self-adaptation to structural and synaptic weight modifications introduced by the first step during the RSNN architectural “evolution.” We demonstrate that the proposed automatic architecture optimization leads to significant performance gains over existing manually designed RSNNs: we achieve 96.44% on TI46-Alpha, 94.66% on N-TIDIGITS, 90.28% on DVS-Gesture, and 98.72% on N-MNIST. To the best of the authors' knowledge, this is the first work to perform systematic architecture optimization on RSNNs.

show abstract

Accelerating Neural Architecture Search via Proxy Data

Cited by 9 publications

References 0 publications

SPENSER: Towards a NeuroEvolutionary Approach for Convolutional Spiking Neural Networks

SPENSER: Towards a NeuroEvolutionary Approach for Convolutional Spiking Neural Networks

Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?

Composing recurrent spiking neural networks using locally-recurrent motifs and risk-mitigating architectural optimization

Contact Info

Product

Resources

About