SpatialHD: Spatial Transformer Fused with Hyperdimensional Computing for AI Applications

Bettayeb, Meriem; Hassan, Eman; Mohammad, Baker; Saleh, Hani

doi:10.1109/aicas57966.2023.10168629

Cited by 6 publications

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Large-scale transformer networks have significant power and computational requirements, preventing them from being deployed on edge devices or in resource-constrained environments such as Internet of Things (IoT) systems 3 . Recent attempts to compress and speed Natural Language Processing (NLP) paradigms on embedded systems like Field Programmable Gateway Arrays (FPGAs) have been reported 4 .…”

Section: Introductionmentioning

confidence: 99%

SAFMem: Accelerating Transformer Self-Attention Functionality via Memristor-Based Hardware

Bettayeb,

Halawani,

Khan

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

The adoption of transformer networks has experienced a notable surge in various AI applications. However, the increasedcomputational complexity, stemming primarily from the self-attention mechanism, parallels the manner in which convolutionoperations constrain the capabilities and speed of Convolutional Neural Networks (CNNs). The self-attention algorithm,specifically the Matrix-matrix Multiplication (MatMul) operations, demands a substantial amount of memory and computationalcomplexity, thereby restricting the overall performance of the transformer. This paper introduces an efficient hardwareaccelerator for the transformer network, leveraging memristor-based in-memory computing. The design targets the memorybottleneck associated with MatMul operations in the self-attention process, utilizing approximate analog computation andthe highly parallel computations facilitated by the memristor crossbar architecture. Remarkably, this approach resulted in areduction of approximately 10 times in the number of Multiply-Accumulate (MAC) operations in transformer networks, whilemaintaining 93.37% accuracy for the MNIST dataset, as validated by a comprehensive circuit simulator employing NeuroSim3.0. Simulation outcomes indicate an area utilization of 6895.7 μm2, a latency of 15.52 seconds, an energy consumption of3 mJ, and a leakage power of 59.55 μW . The methodology outlined in this paper represents a substantial stride towards ahardware-friendly transformer architecture for edge device, poised to achieve real-time performance.

show abstract

Section: Introductionmentioning

confidence: 99%

SAFMem: Accelerating Transformer Self-Attention Functionality via Memristor-Based Hardware

Bettayeb,

Halawani,

Khan

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

How does ChatGPT evaluate the value of spatial information in the 4th industrial revolution?

Hwang,

Um,

Pradhan

et al. 2023

Spat. Inf. Res.

View full text Add to dashboard Cite

Chat Generative Pre-trained Transformer (ChatGPT), developed by OpenAI, is a prominent AI model capable of understanding and generating human-like text based on input. Since terms and concepts of spatial information are contextual, the applications of ChatGPT on spatial information disciplines can be biased by the perceptions and perspectives of ChatGPT towards spatial information. Therefore, a thorough understanding of the real magnitude and level of comprehension of spatial information by ChatGPT is essential before exploring its potential applications in spatial information disciplines. This article aims to investigate how ChatGPT evaluates spatial information and its potential contributions to 4th Industrial Revolution (Industry 4.0). ChatGPT has summarized a notable perspective on evaluating and utilizing spatial information in the context of the Industry 4.0. The result of this study shows that ChatGPT has a good understanding on contextual concepts related to spatial information. However, it exhibits potential biases and challenges, as its responses lean towards the technological and analytical aspects. The results provide a crucial understanding on how to leverage ChatGPT’s benefits to the fullest while recognizing its constraints, with the aim to enhance the efficacy from the perspective of applications linked to spatial information.

show abstract

Adapting Spatial Transformer Networks Across Diverse Hardware Platforms: A Comprehensive Implementation Study

Bettayeb,

Hassan,

Khan

et al. 2024

2024 IEEE 6th International Conference on AI Circuits and Systems (AICAS)

View full text Add to dashboard Cite

SpatialHD: Spatial Transformer Fused with Hyperdimensional Computing for AI Applications

Cited by 6 publications

References 10 publications

SAFMem: Accelerating Transformer Self-Attention Functionality via Memristor-Based Hardware

SAFMem: Accelerating Transformer Self-Attention Functionality via Memristor-Based Hardware

How does ChatGPT evaluate the value of spatial information in the 4th industrial revolution?

Adapting Spatial Transformer Networks Across Diverse Hardware Platforms: A Comprehensive Implementation Study

Contact Info

Product

Resources

About