On-chip training of recurrent neural networks with limited numerical precision

Na, Taesik; Ko, Jong Hwan; Kung, Jaeha; Mukhopadhyay, Saibal

doi:10.1109/ijcnn.2017.7966324

Cited by 37 publications

(28 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is growing interest in stochastic rounding in various domains [6,7,10,[14][15][16][17][18], and this rounding mode has started appearing in hardware devices produced by Graphcore [19] and Intel [20]. In this work we proposed and compared several algorithms for simulating stochastically rounded elementary arithmetic operations via software.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Algorithms for Stochastically Rounded Elementary Arithmetic Operations in IEEE 754 Floating-Point Arithmetic

Fasi

Mikaitis

2021

2021 IEEE 28th Symposium on Computer Arithmetic (ARITH)

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

“…Furthermore, stochastic rounding is being increasingly used in machine learning [13][14][15][16][17][18]. When training neural networks, in particular, it can help compensate for the loss of accuracy caused by reducing the precision at which deep neural networks are trained in fixed-point [14] as well as floating-point [17] arithmetic.…”

Section: Motivationmentioning

confidence: 99%

Algorithms for Stochastically Rounded Elementary Arithmetic Operations in IEEE 754 Floating-Point Arithmetic

Fasi

Mikaitis

2021

2021 IEEE 28th Symposium on Computer Arithmetic (ARITH)

View full text Add to dashboard Cite

“…It's also a little smoother; moreover, it trains somewhat quicker than an LSTM. Due to its relative simplicity [60]. GRUs are combined into a single update gate which acts as both an i and a forget gate.…”

Section: Gated Recurrent Unit (Gru)mentioning

confidence: 99%

The Prediction Process Based on Deep Recurrent Neural Networks: A Review

Zeebaree

Abdulazeez

Abdullrhman

et al. 2021

AJRCoS

View full text Add to dashboard Cite

Prediction is vital in our daily lives, as it is used in various ways, such as learning, adapting, predicting, and classifying. The prediction of parameters capacity of RNNs is very high; it provides more accurate results than the conventional statistical methods for prediction. The impact of a hierarchy of recurrent neural networks on Predicting process is studied in this paper. A recurrent network takes the hidden state of the previous layer as input and generates as output the hidden state of the current layer. Some of deep Learning algorithms can be utilized in as prediction tools in video analysis, musical information retrieval and time series applications. Recurrent networks may process examples simultaneously, maintaining a state or memory that recreates an arbitrarily long background window. Long Short-Term Memory (LSTM) and Bidirectional RNN (BRNN) are examples of recurrent networks. This paper aims to give a comprehensive assessment of predictions based on RNN. Additionally, each paper presents all relevant facts, such as dataset, method, architecture, and the accuracy of the predictions they deliver.

show abstract

“…Benefiting from the good fault tolerance of neural networks [27], [28], the data format of the predictor is represented by signed integers, not by floating-point numbers. In order to study the effect of signed integers with different digits on the branch prediction accuracy, the control variable method is used to control the size of the PHT table to 512 and the length of the GHR register to 32, gradually increasing the data representation precision of signed integers, and the comparison figure shown in Figure 7 is obtained.…”

Section: Data Representation Precisionmentioning

confidence: 99%

A Dynamic Branch Predictor Based on Parallel Structure of SRNN

Zhang

Feng

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Branch predictor is a key component of processor, which can improve the efficiency of instruction execution. The branch predictor based on machine learning algorithm can achieve high branch prediction accuracy, but it has the disadvantages of long training time and high access delay. As a neural network algorithm, Recurrent Neural Network (RNN) is good at processing data related to time series, and can learn the correlation between data faster. Sliced Recurrent Neural Network (SRNN) parallelizes the RNN algorithm, effectively reducing the access delay of the RNN algorithm. In this paper, a dynamic branch predictor based on parallel structure of SRNN is proposed to accelerate the training time and reduces the computing delay. The optimal design parameters of predictor, which has prediction accuracy with lower source cost, are selected through a serial simulations. The experimental results show that the branch predictor proposed in this paper has higher prediction accuracy than the traditional Bimod and Gshare branch predictors under the same hardware consumption, and its branch prediction rate is 2.34% higher than the traditional Perceptron neural predictor in the short learning period. INDEX TERMS Branch predictor, machine learning, recurrent neural network (RNN), sliced recurrent neural network (SRNN).

show abstract

On-chip training of recurrent neural networks with limited numerical precision

Cited by 37 publications

References 10 publications

Algorithms for Stochastically Rounded Elementary Arithmetic Operations in IEEE 754 Floating-Point Arithmetic

Algorithms for Stochastically Rounded Elementary Arithmetic Operations in IEEE 754 Floating-Point Arithmetic

The Prediction Process Based on Deep Recurrent Neural Networks: A Review

A Dynamic Branch Predictor Based on Parallel Structure of SRNN

Contact Info

Product

Resources

About