Bridging CNNs, RNNs, and Weighted Finite-State Machines

Abstract

Recurrent and convolutional neural networks comprise two distinct families of models that have proven to be useful for encoding natural language utterances. In this paper we present SoPa, a new model that aims to bridge these two approaches. SoPa combines neural representation learning with weighted finite-state automata (WFSAs) to learn a soft version of traditional surface patterns. We show that SoPa is an extension of a one-layer CNN, and that such CNNs are equivalent to a restricted version of SoPa, and accordingly, to a restricted form of WFSA. Empirically, on three text classification tasks, SoPa is comparable or better than both a BiLSTM (RNN) baseline and a CNN baseline, and is particularly useful in small data settings.

Weighted Automata Extraction from Recurrent Neural Networks via Regression on State Spaces

“…To apply their method, one has to modify the structure of an RNN before training, while our method does not need any special structure to RNNs and can be applied to already trained RNNs. Schwartz, Thomson, and Smith (2018) introduce a neural network architecture that can represent (restricted forms of) CNNs and RNNs. WFAs could also be expressed by their architecture, but extraction of automata is out of their interest.…”

Rational Recurrences

“…Our definition is equivalent, giving the weight functions value0 wherever they were undefined. 3 ε-transitions can be handled with a slight modification (Schwartz et al, 2018). Note though that if A contains a cycle of ε-transitions, then either K must follow the star semiring laws (Kuich and Salomaa, 1986), or the number of consecutive ε-transitions allowed must be capped.…”

“…So far our discussion has centered on B, a twostate WFSA capturing unigram patterns (Example 5). In the same spirit as going from unigram to n-gram features, one can use WFSAs with more states to capture longer patterns (Schwartz et al, 2018). In this section we augment B by introducing more states, and explore its relationship to some neural architectures motivated by n-gram features.…”

“…It is a direct application of the Forward algorithm ( §2), though now in a form that deals with the "-transition. Such an approach applies, of course, to more general cases, as noted by Schwartz et al (2018). Given an input string x 2 ⌃ + , let z (j) t denote the total score of all paths landing in state q j just after consuming x t .…”

“…B can be seen as capturing soft unigram patterns (Davidov et al, 2010), in the sense that it consumes one input symbol to reach the final state from the initial state. It is straightforward to design WFSAs capturing longer patterns by including more states (Schwartz et al, 2018), as we will discuss later in §4 and §5.…”