Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism

Vasyltsov, Ihor; Chang, Wooseok

doi:10.48550/arxiv.2111.10770

Cited by 3 publications

(5 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The GAM-MLP model concludes with a fully connected layer followed by a Soft-Max layer for classification purposes [32]. The diagram illustrating the classification and recognition process in the MLP of layer plus SoftMax layer is shown in Figure 7 below.…”

Section: Activity Classification and Recognitionmentioning

confidence: 99%

A Hybrid Human Activity Recognition Method Using an MLP Neural Network and Euler Angle Extraction Based on IMU Sensors

Mao,

Yan,

Guo

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Inertial measurement unit (IMU) technology has gained popularity in human activity recognition (HAR) due to its ability to identify human activity by measuring acceleration, angular velocity, and magnetic flux in key body areas like the wrist and knee. It has propelled the extensive application of HAR across various domains. In the healthcare sector, HAR finds utility in monitoring and assessing movements during rehabilitation processes, while in the sports science field, it contributes to enhancing training outcomes and preventing exercise-related injuries. However, traditional sensor fusion algorithms often require intricate mathematical and statistical processing, resulting in higher algorithmic complexity. Additionally, in dynamic environments, sensor states may undergo changes, posing challenges for real-time adjustments within conventional fusion algorithms to cater to the requirements of prolonged observations. To address these limitations, we propose a novel hybrid human pose recognition method based on IMU sensors. The proposed method initially calculates Euler angles and subsequently refines them using magnetometer and gyroscope data to obtain the accurate attitude angle. Furthermore, the application of FFT (Fast Fourier Transform) feature extraction facilitates the transition of the signal from its time-based representation to its frequency-based representation, enhancing the practical significance of the data. To optimize feature fusion and information exchange, a group attention module is introduced, leveraging the capabilities of a Multi-Layer Perceptron which is called the Feature Fusion Enrichment Multi-Layer Perceptron (GAM-MLP) to effectively combine features and generate precise classification results. Experimental results demonstrated the superior performance of the proposed method, achieving an impressive accuracy rate of 96.13% across 19 different human pose recognition tasks. The proposed hybrid human pose recognition method is capable of meeting the demands of real-world motion monitoring and health assessment.

show abstract

Section: Activity Classification and Recognitionmentioning

confidence: 99%

A Hybrid Human Activity Recognition Method Using an MLP Neural Network and Euler Angle Extraction Based on IMU Sensors

Mao,

Yan,

Guo

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…• Approximate attention: This method uses a low-rank matrix or a random feature map to approximate the encoder output sequence, thereby reducing the amount of computation and memory consumption, while maintaining a certain degree of accuracy and effect (Vasyltsov and Chang, 2021). The formula for approximate attention is as follows:…”

Section: Attention Mechanismmentioning

confidence: 99%

Research on the application and promotion of the carbon neutral concept based on the attention mechanism in football under the end-to-end architecture

Hou,

Mu,

Liu

2023

Front. Ecol. Evol.

View full text Add to dashboard Cite

IntroductionIn light of escalating concerns regarding global warming and environmental pollution, the pursuit of carbon neutrality has emerged as a pivotal strategy to address climate change on a global scale. As society becomes increasingly conscious of its ecological impact, various sectors, including sports, are urged to embrace environmental responsibility. This study seeks to explore the integration of a carbon neutral framework utilizing artificial intelligence's attention mechanism within the realm of football, with the aim of contributing to football's adoption of carbon neutrality.MethodsThe study commences by introducing an end-to-end architectural framework capable of unifying and optimizing all facets of football to realize a comprehensive carbon-neutral objective. This architecture serves as a consolidated platform for enhancing carbon emission reduction within football pedagogical activities, fostering synergy among diverse constituents while concurrently assessing the equilibrium between carbon reduction and pedagogical effectiveness. Subsequently, attention mechanisms are leveraged to heighten the efficacy and comprehensibility of carbon-neutral strategies. The application of attention mechanisms enables the model to autonomously focus on attributes or regions closely associated with carbon neutrality objectives, thereby facilitating precision and efficacy in recommending carbon neutral strategies. By employing attention mechanisms in football, a more thorough understanding of carbon emissions' dynamics is attained, allowing for the identification of pivotal emission contributors and tailored suggestions for emission mitigation. Furthermore, the Long Short-Term Memory (LSTM) method is employed to analyze football time-series data. Given football's intricate sequence of actions, the LSTM technique adeptly captures long-term dependencies, offering improved analysis and optimization of carbon emissions during football activities.ResultsThe integrated end-to-end architectural framework offers a holistic approach to carbon-neutral football strategies. Attention mechanisms effectively enhance the focus and interpretation of carbon-neutral strategies, contributing to precise and impactful recommendations. Employing LSTM for time-series analysis aids in comprehending carbon emission dynamics, enabling the identification of efficacious carbon neutral strategies. The study underscores the potential of AI-driven attention mechanisms and LSTM in fostering carbon neutrality within football.DiscussionThe study's findings underscore the viability of integrating AI-driven methodologies, specifically attention mechanisms and LSTM, to promote carbon neutrality within the football domain. The end-to-end architecture serves as a foundational platform for comprehensive carbon emission reduction, offering potential for broader application in other sectors. The combination of attention mechanisms and LSTM engenders deeper insights into carbon emissions' intricate temporal dynamics, informing the development of targeted strategies for emission mitigation. The study's outcomes provide theoretical underpinnings for advancing sustainable football practices and inspire the broader adoption of carbon neutrality principles across diverse domains.

show abstract

“…In the work [28], a precision-adjustable architecture for the Softmax function was developed, with all inputs and outputs represented in 16-bit format, achieving both efficiency and adjustability. Furthermore, [27] even explored the use of 8bit quantization for Softmax function computations, achieving minimal precision loss while working with attention mechanisms in deep neural networks.…”

Section: Quantification Methodsmentioning

confidence: 99%

“…Exponential and division computations require substantial computational resources and time, potentially leading to increased hardware resource consumption and computation latency. Some researchers have made contributions in this regard: [27] proposed two methods using 8-bit fixed-point approximations based on lookup tables to compute Softmax, achieving an accuracy loss of less than 1%. In [28], a method utilizing lookup tables to implement a Precision-Adjustable approach for the Softmax function was employed.…”

Section: B Approximate Calculation Of Nonlinear Functionmentioning

confidence: 99%

See 1 more Smart Citation

A Study Of Civil Engineering Education At Singapore Nanyang Technology University And At The University Of Florida

Liu¹,

Najafi²

2002 Annual Conference Proceedings

View full text Add to dashboard Cite

offer civil engineering courses to undergraduate and graduate students. This paper discusses and compares Civil Engineering curriculum in both Universities. The Civil Engineering courses at the University of Florida are divided into two phases of general education and upper division/civil engineering core education. At Nanyang Technology University, the Civil Engineering offers courses of a prescribed core of essential subjects and both institutions offer excellent program of study. The Civil Engineering program at Nanyang Technology University offers courses closely related to industry and government needs. Similarly, at the University of Florida, Civil Engineering program also offers excellent education program in line with government and industries needs. At both institutions, efforts should be focused on revising curriculum that is more responsible to existing and future rapid technological changes.

show abstract

Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism

Cited by 3 publications

References 28 publications

A Hybrid Human Activity Recognition Method Using an MLP Neural Network and Euler Angle Extraction Based on IMU Sensors

A Hybrid Human Activity Recognition Method Using an MLP Neural Network and Euler Angle Extraction Based on IMU Sensors

Research on the application and promotion of the carbon neutral concept based on the attention mechanism in football under the end-to-end architecture

A Study Of Civil Engineering Education At Singapore Nanyang Technology University And At The University Of Florida

Contact Info

Product

Resources

About