Robust high-dimensional memory-augmented neural networks

Abstract

Traditional neural networks require enormous amounts of data to build their complex mappings during a slow training procedure that hinders their abilities for relearning and adapting to new data. Memory-augmented neural networks enhance neural networks with an explicit memory to overcome these issues. Access to this explicit memory, however, occurs via soft read and write operations involving every individual memory entry, resulting in a bottleneck when implemented using the conventional von Neumann computer architecture. To overcome this bottleneck, we propose a robust architecture that employs a computational memory unit as the explicit memory performing analog in-memory computation on high-dimensional (HD) vectors, while closely matching 32-bit software-equivalent accuracy. This is achieved by a content-based attention mechanism that represents unrelated items in the computational memory with uncorrelated HD vectors, whose real-valued components can be readily approximated by binary, or bipolar components. Experimental results demonstrate the efficacy of our approach on few-shot image classification tasks on the Omniglot dataset using more than 256,000 phase-change memory devices. Our approach effectively merges the richness of deep neural network representations with HD computing that paves the way for robust vector-symbolic manipulations applicable in reasoning, fusion, and compression.

A Primer on Hyperdimensional Computing for iEEG Seizure Detection

“…On a more technical side, a very active area of research are memory-augmented neural networks (MANNs), in which a deep neural network is connected to an associative memory for fast and lifelong learning. Computing with HD vectors can reduce the complexity of MANNs by computing with binary vectors ( 80 ). This recently proposed method reduced the number of parameters by replacing the fully connected layer of a convolutional neural network with a binary associative memory for EEG-based motor imagery brain–machine interfaces ( 81 ).…”

A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

“…Many other types of classifiers such as k-Nearest Neighbors [Karunaratne et al, 2021b] are also likely to work with HVs as their input. As mentioned in the previous section, linear classifiers can be used since HVs are formed with a nonlinear transformation (in case of [Karunaratne et al, 2021b] with an HDC/VSA-guided convolutional neural network feature extractor). For example, the ridge regression, which is commonly used for randomized neural networks [Scardapane and Wang, 2017], performed well with HVs [Rosato et al, 2021].…”

“…Some neural networks already produce binary vectors (see [Mitrokhin et al, 2020]), and the transformation to HVs was in randomly repeated those components to get the necessary dimensionality. In [Karunaratne et al, 2021b], the authors first guided a convolutional neural network to produce HDC/VSA-conforming vectors with the aid of proper attention and sharpening functions, and then just used the sign function to transform those real-valued vectors to bipolar HVs (of the same dimensionality). A rather general way to transform format and/or dimensionality is using RP, possibly with the subsequent binarization by thresholding [Hersche et al, 2020a], .…”

“…In [Mitrokhin et al, 2020], , the authors superimposed HVs obtained from several neural networks that improved the results in applications. There are also proposals to train neural networks from the scratch (as in [Karunaratne et al, 2021b] with meta-learning) such that the activations of the network would resemble pseudo-orthogonal HVs for, e.g., images of unrelated classes. Yet another promising avenue is to use neural networks as a way to form HVs from raw sensory information and then also integrate learning in HDC/VSA domain with these HVs.…”

A comparison of vector symbolic architectures

“…So far, they have been applied in various fields including medical diagnosis (Widdows and Cohen 2015), image feature aggregation , semantic image retrieval , robotics (Neubert et al 2019b), to address catastrophic forgetting in deep neural networks (Cheung et al 2019), fault detection , analogy mapping (Rachkovskij and Slipchenko 2012), reinforcement learning , long-short term memory (Danihelka et al 2016), pattern recognition ), text classification (Joshi et al 2017), synthesis of finite state automata (Osipov et al 2017), and for creating hyperdimensional stack machines (Yerxa et al 2018). Interestingly, also the intermediate and output layers of deep artificial neural networks can provide high-dimensional vector embeddings for symbolic processing with a VSA (Neubert et al 2019b;Yilmaz 2015;Karunaratne et al 2021). Although processing of vectors with thousands of dimensions is currently not very time efficient on standard CPUs, typically, VSA operations can be highly parallelized.…”