Generalized Weight Agnostic Neural Networks for Configurable and Continual Autonomous Systems

Gadikar, Pranav; Ganesan, Vinod; Panda, Pratyush Kumar

doi:10.1145/3486001.3486226

Cited by 8 publications

(15 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Weight agnostic neural networks. One of the pioneering works in zero-shot NAS is presented by Gaier and Ha [2019]. They demonstrate a constructor that builds up neural architectures based on the mean accuracy over several initialisations with constant shared weights and the number of parameters contained within the model.…”

Section: Zero-cost Nasmentioning

confidence: 99%

“…Two existing NAS methods inspire the metric that we share in the present work: CV [Gracheva, 2021] and weight agnostic neural networks [Gaier and Ha, 2019]. Both metrics aim to exclude individual weight values from consideration when evaluating networks: the former cancels out the individual weights via multiple random initialisations, while the latter sets them to the same value across the network.…”

Section: Epsilon Metricmentioning

confidence: 99%

“…It can be, to some degree, explained by the fact that random weight initialisations bring in some noise. Our idea is that replacing random initialisations with single shared weight initialisations, similarly to Gaier and Ha [2019], should improve the method's performance.…”

Section: Epsilon Metricmentioning

confidence: 99%

See 2 more Smart Citations

Light and Accurate: Neural Architecture Search via Two Constant Shared Weights Initialisations

Gracheva¹

2023

Preprint

View full text Add to dashboard Cite

In recent years, zero-cost proxies are gaining ground in neural architecture search (NAS). These methods allow finding the optimal neural network for a given task faster and with a lesser computational load than conventional NAS methods. Equally important is the fact that they also shed some light on the internal workings of neural architectures. This paper presents a zero-cost metric that highly correlates with the train set accuracy across the NAS-Bench-101, NAS-Bench-201 and NAS-Bench-NLP benchmark datasets. Architectures are initialised with two distinct constant shared weights, one at a time. Then, a fixed random mini-batch of data is passed forward through each initialisation. We observe that the dispersion of the outputs between two initialisations positively correlates with trained accuracy. The correlation further improves when we normalise dispersion by average output magnitude. Our metric, epsilon, does not require gradients computation or labels. It thus unbinds the NAS procedure from training hyperparameters, loss metrics and human-labelled data. Our method is easy to integrate within existing NAS algorithms and takes a fraction of a second to evaluate a single network.

show abstract

Section: Zero-cost Nasmentioning

confidence: 99%

Section: Epsilon Metricmentioning

confidence: 99%

See 1 more Smart Citation

Light and Accurate: Neural Architecture Search via Two Constant Shared Weights Initialisations

Gracheva¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…ANNs using these structure operators can then be applied in different spatial or temporal information processing tasks, such as image recognition (Frankle et al, 2019;Chen et al, 2020), auditory recognition, and heterogeneous graph recognition (Hu et al, 2022). Furthermore, when only focusing on the learning of weight, the weight agnostic neural network (Gaier and Ha, 2019;Aladago and Torresani, 2021) is a representative of the methods that only train the connections instead of weights.…”

Section: Introductionmentioning

confidence: 99%

Explaining cocktail party effect and McGurk effect with a spiking neural network improved by Motif-topology

et al. 2023

View full text Add to dashboard Cite

Network architectures and learning principles have been critical in developing complex cognitive capabilities in artificial neural networks (ANNs). Spiking neural networks (SNNs) are a subset of ANNs that incorporate additional biological features such as dynamic spiking neurons, biologically specified architectures, and efficient and useful paradigms. Here we focus more on network architectures in SNNs, such as the meta operator called 3-node network motifs, which is borrowed from the biological network. We proposed a Motif-topology improved SNN (M-SNN), which is further verified efficient in explaining key cognitive phenomenon such as the cocktail party effect (a typical noise-robust speech-recognition task) and McGurk effect (a typical multi-sensory integration task). For M-SNN, the Motif topology is obtained by integrating the spatial and temporal motifs. These spatial and temporal motifs are first generated from the pre-training of spatial (e.g., MNIST) and temporal (e.g., TIDigits) datasets, respectively, and then applied to the previously introduced two cognitive effect tasks. The experimental results showed a lower computational cost and higher accuracy and a better explanation of some key phenomena of these two effects, such as new concept generation and anti-background noise. This mesoscale network motifs topology has much room for the future.

show abstract

“…To test the inductive bias of using anatomy information, we compare the features that the models produce at random initialization [433]. From Figure 6.4 we can see AAGN is able to achieve a decent separation of AD and CN MRI images even without training.…”

Section: Model Analysismentioning

confidence: 99%

Self-supervised learning for early detection of neurodegenerative diseases with small data

Jiang¹

View full text Add to dashboard Cite

would also like to thank co-supervisor Prof. Chin and mentor Prof. Yu for sharing their knowledge and improving me as a researcher. This journey would have been impossible without their guidance and supervision.My sincere gratitude to my friends, Bryan, Jonathan, and Han Yue for providing peer-support and livening up the journey during the tough times. The memories made over the different seasons will be fondly remembered. I would also like to thank the staff of LILY and the Alibaba-NTU JRI who have helped me along the way. Special thanks to Huang Bo for providing the technical support with servers and Zhiwei for spending time to go through the initial outline of this thesis.Last but not least, my eternal gratitude to my family for their unwavering support since day one of this journey. vii

show abstract

Generalized Weight Agnostic Neural Networks for Configurable and Continual Autonomous Systems

Cited by 8 publications

References 4 publications

Light and Accurate: Neural Architecture Search via Two Constant Shared Weights Initialisations

Light and Accurate: Neural Architecture Search via Two Constant Shared Weights Initialisations

Explaining cocktail party effect and McGurk effect with a spiking neural network improved by Motif-topology

Self-supervised learning for early detection of neurodegenerative diseases with small data

Contact Info

Product

Resources

About