Extraction of rules from discrete-time recurrent neural networks

Omlin, Christian W.; Giles, C. Lee

doi:10.1016/0893-6080(95)00086-0

Cited by 158 publications

(92 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…LSTM learned to solve many previously unlearnable DL tasks involving: Recognition of the temporal order of widely separated events in noisy input streams; Robust storage of high-precision real numbers across extended time intervals; Arithmetic operations on continuous input streams; Extraction of information conveyed by the temporal distance between events; Recognition of temporally extended patterns in noisy input sequences (Hochreiter and Schmidhuber, 1997b;; Stable generation of precisely timed rhythms, as well as smooth and non-smooth periodic trajectories . LSTM clearly outperformed previous RNNs on tasks that require learning the rules of regular languages describable by deterministic Finite State Automata (FSAs) (Watrous and Kuhn, 1992;Casey, 1996;Siegelmann, 1992;Blair and Pollack, 1997;Kalinke and Lehmann, 1998;Zeng et al, 1994;Manolios and Fanelli, 1994;Omlin and Giles, 1996;Vahed and Omlin, 2004), both in terms of reliability and speed.…”

Section: : Supervised Recurrent Very Deep Learner (Lstm Rnn)mentioning

confidence: 99%

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.LATEX source: http://www.idsia.ch/˜juergen/DeepLearning8Oct2014.tex Complete BIBTEX file (888 kB): http://www.idsia.ch/˜juergen/deep.bib Preface This is the preprint of an invited Deep Learning (DL) overview. One of its goals is to assign credit to those who contributed to the present state of the art. I acknowledge the limitations of attempting to achieve this goal. The DL research community itself may be viewed as a continually evolving, deep network of scientists who have influenced each other in complex ways. Starting from recent DL results, I tried to trace back the origins of relevant ideas through the past half century and beyond, sometimes using "local search" to follow citations of citations backwards in time. Since not all DL publications properly acknowledge earlier relevant work, additional global search strategies were employed, aided by consulting numerous neural network experts. As a result, the present preprint mostly consists of references. Nevertheless, through an expert selection bias I may have missed important work. A related bias was surely introduced by my special familiarity with the work of my own DL research group in the past quarter-century. For these reasons, this work should be viewed as merely a snapshot of an ongoing credit assignment process. To help improve it, please do not hesitate to send corrections and suggestions to juergen@idsia.ch.

show abstract

Section: : Supervised Recurrent Very Deep Learner (Lstm Rnn)mentioning

confidence: 99%

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

show abstract

“…This was verified empirically [57] through information extraction methods [46], [37] where recurrent networks trained on fuzzy strings develop a crisp internal representation of FFA, i.e., that they represent FFA in the form of equivalent deterministic acceptors. 3 Thus, our theoretical analysis correctly predicted the knowledge representation for such trained networks.…”

Section: B Backgroundmentioning

confidence: 99%

“…Neural networks that have been trained to behave like FFA do not necessarily share this property, i.e., their internal representation of states and transitions may become unstable for sufficiently long input sequences [37]. Finally, with the extraction of knowledge from trained neural networks, the methods discussed here could potentially be applied to incorporating and refining [38] fuzzy knowledge previously encoded into recurrent neural networks.…”

mentioning

confidence: 99%

Equivalence in knowledge representation: automata, recurrent neural networks, and dynamical fuzzy systems

1999

View full text Add to dashboard Cite

show abstract

“…These two mechanisms could conceivably use two different representations: effectively a task-dependent, and a taskindependent representation, e.g. the learning mechanism could be a recurrent neural network, and knowledge could be extracted in the form of finite state automata [12]. People have also attempted to combine feed-forward neural networks with symbolic rules [13].…”

Section: Cumulative Learningmentioning

confidence: 99%

Symbol Grounding Through Cumulative Learning

Swarup¹,

Lakkaraju²,

Ray³

et al. 2006

Symbol Grounding and Beyond

View full text Add to dashboard Cite

Abstract. We suggest that the primary motivation for an agent to construct a symbol-meaning mapping is to solve a task. The meaning space of an agent should be derived from the tasks that it faces during the course of its lifetime. We outline a process in which agents learn to solve multiple tasks and extract a store of "cumulative knowledge" that helps them to solve each new task more quickly and accurately. This cumulative knowledge then forms the ontology or meaning space of the agent. We suggest that by grounding symbols to this extracted cumulative knowledge agents can gain a further performance benefit because they can guide each others' learning process. In this version of the symbol grounding problem meanings cannot be directly communicated because they are internal to the agents, and they will be different for each agent. Also, the meanings may not correspond directly to objects in the environment. The communication process can also allow a symbol meaning mapping that is dynamic. We posit that these properties make this version of the symbol grounding problem realistic and natural. Finally, we discuss how symbols could be grounded to cumulative knowledge via a situation where a teacher selects tasks for a student to perform.

show abstract

Extraction of rules from discrete-time recurrent neural networks

Abstract: Absa'act--The extraction of symbolic knowledge from trained neural networks and the direct encoding of (partial)

Cited by 158 publications

References 25 publications

Deep learning in neural networks: An overview

Deep learning in neural networks: An overview

Equivalence in knowledge representation: automata, recurrent neural networks, and dynamical fuzzy systems

Symbol Grounding Through Cumulative Learning

Contact Info

Product

Resources

About