Approximate LSTM Computing for Energy-Efficient Speech Recognition

Jo, Junseo; Kung, Jaeha; Lee, Youngjoo

doi:10.3390/electronics9122004

Cited by 24 publications

(19 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…in the denominator term of Equation (3). From this equation, the source voltage, V S,i is expected to be lowered, as the number of LRS cells for Row #i becomes large.…”

Section: Methodsmentioning

confidence: 99%

“…By doing so, the source voltage loss due to source resistance can be compensated in Equation (3). Here it should be noted that R R,i of Row #i in Figure 3 can be made of a memristor.…”

Section: Methodsmentioning

confidence: 99%

“…Neural networks draw many interests nowadays, as they have been verified useful in various cognitive tasks such as natural language processing, image recognition, object classification, etc. [1][2][3]. As the applications of neural networks become more complex, the demand for high-performance computing becomes increasingly more.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Memristor-CMOS Hybrid Neuron Circuit with Nonideal-Effect Correction Related to Parasitic Resistance for Binary-Memristor-Crossbar Neural Networks

Nguyen

Min

2021

Micromachines

View full text Add to dashboard Cite

Voltages and currents in a memristor crossbar can be significantly affected due to nonideal effects such as parasitic source, line, and neuron resistance. These nonideal effects related to the parasitic resistance can cause the degradation of the neural network’s performance realized with the nonideal memristor crossbar. To avoid performance degradation due to the parasitic-resistance-related nonideal effects, adaptive training methods were proposed previously. However, the complicated training algorithm could add a heavy computational burden to the neural network hardware. Especially, the hardware and algorithmic burden can be more serious for edge intelligence applications such as Internet of Things (IoT) sensors. In this paper, a memristor-CMOS hybrid neuron circuit is proposed for compensating the parasitic-resistance-related nonideal effects during not the training phase but the inference one, where the complicated adaptive training is not needed. Moreover, unlike the previous linear correction method performed by the external hardware, the proposed correction circuit can be included in the memristor crossbar to minimize the power and hardware overheads for compensating the nonideal effects. The proposed correction circuit has been verified to be able to restore the degradation of source and output voltages in the nonideal crossbar. For the source voltage, the average percentage error of the uncompensated crossbar is as large as 36.7%. If the correction circuit is used, the percentage error in the source voltage can be reduced from 36.7% to 7.5%. For the output voltage, the average percentage error of the uncompensated crossbar is as large as 65.2%. The correction circuit can improve the percentage error in the output voltage from 65.2% to 8.6%. Almost the percentage error can be reduced to ~1/7 if the correction circuit is used. The nonideal memristor crossbar with the correction circuit has been tested for MNIST and CIFAR-10 datasets in this paper. For MNIST, the uncompensated and compensated crossbars indicate the recognition rate of 90.4% and 95.1%, respectively, compared to 95.5% of the ideal crossbar. For CIFAR-10, the nonideal crossbars without and with the nonideal-effect correction show the rate of 85.3% and 88.1%, respectively, compared to the ideal crossbar achieving the rate as large as 88.9%.

show abstract

“…in the denominator term of Equation (3). From this equation, the source voltage, V S,i is expected to be lowered, as the number of LRS cells for Row #i becomes large.…”

Section: Methodsmentioning

confidence: 99%

“…By doing so, the source voltage loss due to source resistance can be compensated in Equation (3). Here it should be noted that R R,i of Row #i in Figure 3 can be made of a memristor.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Memristor-CMOS Hybrid Neuron Circuit with Nonideal-Effect Correction Related to Parasitic Resistance for Binary-Memristor-Crossbar Neural Networks

Nguyen

Min

2021

Micromachines

View full text Add to dashboard Cite

show abstract

“…Due to remembering information for a long time and removing the vanishing gradient problem of RNN, LSTM has appeared to be an effective model in solving problems with sequential data containing long-term dependencies. Some of the examples of LSTM applications are speech recognition [36], machine translation [37], time series forecasting [38,39], and sentiment analysis [40]. Hochreiter and Schmidhuber [41] in 1997 first proposed the LSTM model in which each LSTM unit contains only input and output gates.…”

Section: Long Short-term Memory (Lstm)mentioning

confidence: 99%

Comparative Study of Univariate and Multivariate Long Short-Term Memory for Very Short-Term Forecasting of Global Horizontal Irradiance

et al. 2021

View full text Add to dashboard Cite

Accurate global horizontal irradiance (GHI) forecasting is crucial for efficient management and forecasting of the output power of photovoltaic power plants. However, developing a reliable GHI forecasting model is challenging because GHI varies over time, and its variation is affected by changes in weather patterns. Recently, the long short-term memory (LSTM) deep learning network has become a powerful tool for modeling complex time series problems. This work aims to develop and compare univariate and several multivariate LSTM models that can predict GHI in Guntur, India on a very short-term basis. To build the multivariate time series models, we considered all possible combinations of temperature, humidity, and wind direction variables along with GHI as inputs and developed seven multivariate models, while in the univariate model, we considered only GHI variability. We collected the meteorological data for Guntur from 1 January 2016 to 31 December 2016 and built 12 datasets, each containing variability of GHI, temperature, humidity, and wind direction of a month. We then constructed the models, each of which measures up to 2 h ahead of forecasting of GHI. Finally, to measure the symmetry among the models, we evaluated the performances of the prediction models using root mean square error (RMSE) and mean absolute error (MAE). The results indicate that, compared to the univariate method, each multivariate LSTM performs better in the very short-term GHI prediction task. Moreover, among the multivariate LSTM models, the model that incorporates the temperature variable with GHI as input has outweighed others, achieving average RMSE values 0.74 W/m2–1.5 W/m2.

show abstract

“…Approximate computing is a new paradigm where an acceptable error is induced in the computing to achieve more energy-efficient processing [ 28 , 29 , 30 , 31 , 32 , 33 ]. It has been introduced at different system levels [ 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 ], and a large number of approximate arithmetic circuits have been designed to save chip area and energy [ 35 , 38 , 46 , 47 , 48 , 49 , 50 , 51 ]. Multiplication is a very common, but expensive operation, with exact multipliers being large circuits that consume a significant amount of energy.…”

Section: Introductionmentioning

confidence: 99%

An Approximate GEMM Unit for Energy-Efficient Object Detection

Pilipović

Risojević

Božič

et al. 2021

Sensors

View full text Add to dashboard Cite

Edge computing brings artificial intelligence algorithms and graphics processing units closer to data sources, making autonomy and energy-efficient processing vital for their design. Approximate computing has emerged as a popular strategy for energy-efficient circuit design, where the challenge is to achieve the best tradeoff between design efficiency and accuracy. The essential operation in artificial intelligence algorithms is the general matrix multiplication (GEMM) operation comprised of matrix multiplication and accumulation. This paper presents an approximate general matrix multiplication (AGEMM) unit that employs approximate multipliers to perform matrix–matrix operations on four-by-four matrices given in sixteen-bit signed fixed-point format. The synthesis of the proposed AGEMM unit to the 45 nm Nangate Open Cell Library revealed that it consumed only up to 36% of the area and 25% of the energy required by the exact general matrix multiplication unit. The AGEMM unit is ideally suited to convolutional neural networks, which can adapt to the error induced in the computation. We evaluated the AGEMM units’ usability for honeybee detection with the YOLOv4-tiny convolutional neural network. The results implied that we can deploy the AGEMM units in convolutional neural networks without noticeable performance degradation. Moreover, the AGEMM unit’s employment can lead to more area- and energy-efficient convolutional neural network processing, which in turn could prolong sensors’ and edge nodes’ autonomy.

show abstract

Approximate LSTM Computing for Energy-Efficient Speech Recognition

Cited by 24 publications

References 20 publications

Memristor-CMOS Hybrid Neuron Circuit with Nonideal-Effect Correction Related to Parasitic Resistance for Binary-Memristor-Crossbar Neural Networks

Memristor-CMOS Hybrid Neuron Circuit with Nonideal-Effect Correction Related to Parasitic Resistance for Binary-Memristor-Crossbar Neural Networks

Comparative Study of Univariate and Multivariate Long Short-Term Memory for Very Short-Term Forecasting of Global Horizontal Irradiance

An Approximate GEMM Unit for Energy-Efficient Object Detection

Contact Info

Product

Resources

About