FPGA Acceleration of Recurrent Neural Network Based Language Model

Abstract

Recurrent neural network (RNN) based language model (RNNLM) is a biologically inspired model for natural language processing. It records the historical information through additional recurrent connections and therefore is very effective in capturing semantics of sentences. However, the use of RNNLM has been greatly hindered for the high computation cost in training. This work presents an FPGA implementation framework for RNNLM training acceleration. At architectural level, we improve the parallelism of RNN training scheme and reduce the computing resource requirement for computation efficiency enhancement. The hardware implementation primarily targets at reducing data communication load. A multi-thread based computation engine is utilized which can successfully mask the long memory latency and reuse frequent accessed data. The evaluation based on the Microsoft Research Sentence Completion Challenge shows that the proposed FPGA implementation outperforms traditional class-based modest-size recurrent networks and obtains 46.2% in training accuracy. Moreover, experiments at different network sizes demonstrate a great scalability of the proposed framework.

FastWave: Accelerating Autoregressive Convolutional Neural Networks on FPGA

“…], [28] while maintaining stateof-the-art accuracy. Prior efforts have been made in FPGA acceleration of speech recognition, classification and language modelling using Recurrent Neural Networks [14], [16], [27]; however the challenges in generation of sequences with longterm dependencies, particularly in audio domain have not been addressed.…”

Accelerated Artificial Neural Networks on FPGA for Fault Detection in Automotive Systems

“…Deep learning forms of ANN like Convolutional Neural Networks (CNN) and Recurrant Neural Networks (RNN) are widely acknowledged for their compute performance and energy efficiency in classification and machine learning tasks on large datasets (typically in datacenters) [3], [4], [8]. Many ANN applications that interact with physical systems require the accuracy and dynamic range offered by floating point representations, resulting in increased complexity at each neuron.…”

“…Many ANN applications that interact with physical systems require the accuracy and dynamic range offered by floating point representations, resulting in increased complexity at each neuron. FPGAs represent an ideal platform for accelerating ANN-based systems because they enable large scale parallelism while also supporting high throughput floating point computations [3], [4], [9].…”

“…Artificial Neural Networks (ANNs) are a promising approach in this regard, since they can be used to mimic these physical relations (by training). Such adaptability enables ANNs to be employed for a wide range of applications like classification [3], [4], adaptive communication [5] and invehicle fault detection [6], [7], among many others. These applications often employ multiple layers of pre-trained neurons referred to as multi-layer perceptrons (MLPs).…”

EERA-DNN: An energy-efficient reconfigurable architecture for DNNs with hybrid bit-width and logarithmic multiplier

“…In work [20] and [21], the network are implemented and evaluated on their original form without compression. In work [9] and our work, all the networks are first compressed, so the equivalent throughput should be multiplied by its compression rates which is as same as [9].…”