HyperEmbed: Tradeoffs Between Resources and Performance in NLP Tasks with Hyperdimensional Computing Enabled Embedding of n-gram Statistics

Alonso, Pedro; Shridhar, Kumar; Kleyko, Denis; Osipov, Evgeny; Liwicki, Marcus

doi:10.1109/ijcnn52387.2021.9534359

Cited by 12 publications

(12 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The works in this domain dealt with different tasks such as text categorization, news identification, and intent classification. Most of the works [Rachkovskij, 2007], [Shridhar et al, 2020], [Alonso et al, 2021] used HVs as a way to represent data for some conventional machine learning classification algorithms. 1.…”

Section: Classification Based On Feature Vectorsmentioning

confidence: 99%

“…For example, the ridge regression, which is commonly used for randomized neural networks [Scardapane and Wang, 2017], performed well with HVs [Rosato et al, 2021]. However, not all conventional classifiers work well with HVs [Alonso et al, 2021]. That is because some of the algorithms (e.g., decision tree or Naïve Bayes) assume that a component of a vector can be interpreted on its own.…”

Section: Classificationmentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

Kleyko,

Rachkovskij,

Osipov

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This is Part II of the two-part comprehensive survey devoted to a computing framework most commonly known under the names Hyperdimensional Computing and Vector Symbolic Architectures (HDC/VSA). Both names refer to a family of computational models that use high-dimensional distributed representations and rely on the algebraic properties of their key operations to incorporate the advantages of structured symbolic representations and vector distributed representations. Holographic Reduced Representations [Plate, 1995], [Plate, 2003] is an influential HDC/VSA model that is well-known in the machine learning domain and often used to refer to the whole family. However, for the sake of consistency, we use HDC/VSA to refer to the area.Part I of this survey [Kleyko et al., 2021c] covered foundational aspects of the area, such as historical context leading to the development of HDC/VSA, key elements of any HDC/VSA model, known HDC/VSA models, and transforming input data of various types into high-dimensional vectors suitable for HDC/VSA. This second part surveys existing applications, the role of HDC/VSA in cognitive computing and architectures, as well as directions for future work. Most of the applications lie within the machine learning/artificial intelligence domain, however we also cover other applications to provide a thorough picture. The survey is written to be useful for both newcomers and practitioners.

show abstract

Section: Classification Based On Feature Vectorsmentioning

confidence: 99%

Section: Classificationmentioning

confidence: 99%

A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

Kleyko,

Rachkovskij,

Osipov

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Using these symbolic operations, HDC can solve a variety of learning tasks with comparable performance to conventional machine learning algorithms [26]- [28]. In addition, hypervectors can be used as inputs to conventional machine learning algorithms [29]- [33].…”

Section: B Hyperdimensional Computingmentioning

confidence: 99%

Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing

Diao,

Kleyko,

Rabaey

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning algorithms deployed on edge devices must meet certain resource constraints and efficiency requirements. Random Vector Functional Link (RVFL) networks are favored for such applications due to their simple design and training efficiency. We propose a modified RVFL network that avoids computationally expensive matrix operations during training, thus expanding the network's range of potential applications. Our modification replaces the least-squares classifier with the Generalized Learning Vector Quantization (GLVQ) classifier, which only employs simple vector and distance calculations. The GLVQ classifier can also be considered an improvement upon certain classification algorithms popularly used in the area of Hyperdimensional Computing. The proposed approach achieved state-of-the-art accuracy on a collection of datasets from the UCI Machine Learning Repository-higher than previously proposed RVFL networks. We further demonstrate that our approach still achieves high accuracy while severely limited in training iterations (using on average only 21% of the least-squares classifier computational costs).

show abstract

“…Please consult [24] for a general overview. What is more important here is that HDC/VSA can solve a variety of learning tasks with comparable performance to conventional machine learning algorithms [25]- [27] or alternatively hypervectors can be used as an input to conventional machine learning algorithms [28]- [32].…”

Section: A Hyperdimensional Computingmentioning

confidence: 99%

Hyperdimensional Computing for Efficient Distributed Classification with Randomized Neural Networks

Rosato,

Panella,

Kleyko

2021

Preprint

Self Cite

View full text Add to dashboard Cite

In the supervised learning domain, considering the recent prevalence of algorithms with high computational cost, the attention is steering towards simpler, lighter, and less computationally extensive training and inference approaches. In particular, randomized algorithms are currently having a resurgence, given their generalized elementary approach. By using randomized neural networks, we study distributed classification, which can be employed in situations were data cannot be stored at a central location nor shared. We propose a more efficient solution for distributed classification by making use of a lossy compression approach applied when sharing the local classifiers with other agents. This approach originates from the framework of hyperdimensional computing, and is adapted herein. The results of experiments on a collection of datasets demonstrate that the proposed approach has usually higher accuracy than local classifiers and getting close to the benchmark -the centralized classifier. This work can be considered as the first step towards analyzing the variegated horizon of distributed randomized neural networks.

show abstract

HyperEmbed: Tradeoffs Between Resources and Performance in NLP Tasks with Hyperdimensional Computing Enabled Embedding of n-gram Statistics

Cited by 12 publications

References 34 publications

A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing

Hyperdimensional Computing for Efficient Distributed Classification with Randomized Neural Networks

Contact Info

Product

Resources

About