Robust and Scalable Differentiable Neural Computer for Question Answering

Abstract

Deep learning models are often not easily adaptable to new tasks and require task-specific adjustments. The differentiable neural computer (DNC), a memoryaugmented neural network, is designed as a general problem solver which can be used in a wide range of tasks. But in reality, it is hard to apply this model to new tasks. We analyze the DNC and identify possible improvements within the application of question answering. This motivates a more robust and scalable DNC (rsDNC). The objective precondition is to keep the general character of this model intact while making its application more reliable and speeding up its required training time. The rsDNC is distinguished by a more robust training, a slim memory unit and a bidirectional architecture. We not only achieve new state-of-the-art performance on the bAbI task, but also minimize the performance variance between different initializations. Furthermore, we demonstrate the simplified applicability of the rsDNC to new tasks with passable results on the CNN RC task without adaptions.

Explainable Inference on Sequential Data via Memory-Tracking

“…This suggests that the agent improves its performance when it learns to exploit the relevant step information stored in the memory. Note that the matching accuracy is influenced both by the high variability of the learning process in MANNs [Franke et al, 2018] and by the fact that the memory output can be sometimes ignored, in favor of the LSTM output. This is highlighted by the variance -although it remains reliable (accuracy ≥ 0.6) also in the worst-case scenarios.…”

“…Differential Neural Computers (DNCs) [Graves et al, 2016] have been recently introduced as a memory-augmented machine learning model based on the usage of a controller network. A controller is a neural network -typically recurrent with LSTM units -that (1) has read and write access to an external memory through multiple read and write heads, and (2) learns both how to perform these operations and on which data to run them.…”

“…Finally, based on the analysis from [Franke et al, 2018], we apply the bypass dropout to the output of the LSTM to force an earlier memory usage during training, and a layer normalization to the input of memory to stabilize the training process.…”

Context and event-based cognitive memory constructs for embodied intelligence machines

“…Various types of learning were considered: supervised, unsupervised, semi-supervised, weakly-supervised, transfer learning, reinforcement learning, active learning, one-shot learning etc. The Differential Neural Computer model (it augments a neural network with memory) [26], structural equation models [27], correlations with p-value testing, Granger causality tests and an abductive reasoning approach (logical inferencing that starts with one or more observations and searches for the simplest explanation for it) were also considered.…”

Neural Stored-program Memory

“…These works keep using memory for storing data rather than the weights of the network and thus parallel to our approach. Other DNC modifications [8,9] are also orthogonal to our work.…”