Hardware implementation methods in Random Vector Functional-Link Networks

Martínez-Villena, José M.; Muñoz, Andrés Pedreño; Soria-Olivas, Emilio

doi:10.1007/s10489-013-0501-1

Cited by 22 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With respect to hyperdimensional computing, the best starting point is the tutorial-like article [5] by Kanerva. Since even the conventional RVFLs are considered as one of the simplest approaches for machine learning, it explains why the efforts on pushing their resource-efficiency to the extreme are limited. The most relevant works in this direction are [8], [18]. Similar to the present study, both works use FPGA for hardware experiments.…”

Section: Related Workmentioning

confidence: 84%

Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks

Kleyko,

Kheffache,

Frady

et al. 2019

Preprint

View full text Add to dashboard Cite

The deployment of machine learning algorithms on resource-constrained edge devices is an important challenge from both theoretical and applied points of view. In this article, we focus on resource-efficient randomly connected neural networks known as Random Vector Functional Link (RVFL) networks since their simple design and extremely fast training time make them very attractive for solving many applied classification tasks. We propose to represent input features via the densitybased encoding known in the area of stochastic computing and use the operations of binding and bundling from the area of hyperdimensional computing for obtaining the activations of the hidden neurons. Using a collection of 121 real-world datasets from the UCI Machine Learning Repository, we empirically show that the proposed approach demonstrates higher average accuracy than the conventional RVFL. We also demonstrate that it is possible to represent the readout matrix using only integers in a limited range with minimal loss in the accuracy. In this case, the proposed approach operates only on small n-bits integers, which results in a computationally efficient architecture. Finally, through hardware FPGA implementations, we show that such an approach consumes approximately eleven times less energy than that of the conventional RVFL.

show abstract

Section: Related Workmentioning

confidence: 84%

Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks

Kleyko,

Kheffache,

Frady

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The computational complexity of the RLS is dominated by the N ×N matrix inversion in (2), though the other matrix operations are also important factors. These matrix calculations typically place a high computational burden on hardware and may cause numerical instabilities [16]. Methods for computing (H H + λI) −1 (with the commonly used QR decomposition, for example) can be parallelized using graphical processing unit (GPU) clusters [5].…”

Section: Methodsmentioning

confidence: 99%

“…However, directly computing the matrix inverse (H H + λI) −1 is numerically unstable and ill-advised. Instead, we consider a different approach that uses QR decomposition to factor (H H + λI) −1 [16]; this step allows us to solve for the individual rows w (j) of W out without computing matrix inverses. Please consult [16] for more details on this approach.…”

Section: Computational Cost Of Rls and Glvq Classifiersmentioning

confidence: 99%

“…Instead, we consider a different approach that uses QR decomposition to factor (H H + λI) −1 [16]; this step allows us to solve for the individual rows w (j) of W out without computing matrix inverses. Please consult [16] for more details on this approach. We note that computations for this approach can be parallelized using GPU clusters, as mentioned in Section II.…”

Section: Computational Cost Of Rls and Glvq Classifiersmentioning

confidence: 99%

See 1 more Smart Citation

Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing

Diao,

Kleyko,

Rabaey

et al. 2021

Preprint

View full text Add to dashboard Cite

Machine learning algorithms deployed on edge devices must meet certain resource constraints and efficiency requirements. Random Vector Functional Link (RVFL) networks are favored for such applications due to their simple design and training efficiency. We propose a modified RVFL network that avoids computationally expensive matrix operations during training, thus expanding the network's range of potential applications. Our modification replaces the least-squares classifier with the Generalized Learning Vector Quantization (GLVQ) classifier, which only employs simple vector and distance calculations. The GLVQ classifier can also be considered an improvement upon certain classification algorithms popularly used in the area of Hyperdimensional Computing. The proposed approach achieved state-of-the-art accuracy on a collection of datasets from the UCI Machine Learning Repository-higher than previously proposed RVFL networks. We further demonstrate that our approach still achieves high accuracy while severely limited in training iterations (using on average only 21% of the least-squares classifier computational costs).

show abstract

“…To this end, they also investigated how RVFL networks can be customized to offer a practical tool for density estimation . RVFL networks have gained increasing popularity in the last few years, and numerous additional applications were explored, which are briefly summarized in Section 1 of Martnez‐Villena et al An extensive evaluation of different RVFL variants is given instead in Zhang and Suganthan, particularly showing significant improvements by the inclusion of the input/output links. RVFL networks have also been explored in semisupervised scenarios, wherein only a part of the training data is labeled, and more recently in a multilayered configuration using a variation of the autoencoder network …”

Section: Feedforward Network With Random Weightsmentioning

confidence: 99%

Randomness in neural networks: an overview

Scardapane

Wang

2017

WIREs Data Min & Knowl

239

144

View full text Add to dashboard Cite

Neural networks, as powerful tools for data mining and knowledge engineering, can learn from data to build feature-based classifiers and nonlinear predictive models. Training neural networks involves the optimization of nonconvex objective functions, and usually, the learning process is costly and infeasible for applications associated with data streams. A possible, albeit counterintuitive, alternative is to randomly assign a subset of the networks' weights so that the resulting optimization task can be formulated as a linear least-squares problem. This methodology can be applied to both feedforward and recurrent networks, and similar techniques can be used to approximate kernel functions. Many experimental results indicate that such randomized models can reach sound performance compared to fully adaptable ones, with a number of favorable benefits, including (1) simplicity of implementation, (2) faster learning with less intervention from human beings, and (3) possibility of leveraging overall linear regression and classification algorithms (e.g., ℓ 1 norm minimization for obtaining sparse formulations). This class of neural networks attractive and valuable to the data mining community, particularly for handling large scale data mining in real-time. However, the literature in the field is extremely vast and fragmented, with many results being reintroduced multiple times under different names. This overview aims to provide a self-contained, uniform introduction to the different ways in which randomization can be applied to the design of neural networks and kernel functions. A clear exposition of the basic framework underlying all these approaches helps to clarify innovative lines of research, open problems, and most importantly, foster the exchanges of well-known results throughout different communities.

show abstract

Hardware implementation methods in Random Vector Functional-Link Networks

Cited by 22 publications

References 36 publications

Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks

Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks

Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing

Randomness in neural networks: an overview

Contact Info

Product

Resources

About