Distributional Learning of Context-Free and Multiple Context-Free Grammars

Clark, Alexander; Yoshinaka, Ryo

doi:10.1007/978-3-662-48395-4_6

Cited by 13 publications

(9 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This condition is very close to a number of conditions that have been proposed in the literature both for topic modeling and for grammatical inference: We use here the terminology of Stratos et al (2016), but similar ideas occur in, for example, Adriaans's (1999) approach to learning CFGs and Denis et al's (2004) approach to learning regular languages. This is also very closely related to what is called the 1-Finite Kernel Property in distributional learning of CFGs (Clark and Yoshinaka, 2016).…”

Section: Structural Conditions On Grammarsmentioning

confidence: 74%

“…This can be naturally extended, mutatis mutandis, to sets of exemplars, and to exemplars with length greater than 1. The extension beyond CFGs to mildly context sensitive grammars such as MCFGs (Seki et al, 1991) seems to present some problems that do not occur in the nonprobabilistic case (Clark and Yoshinaka, 2016); although the same bounds on the bottom up parameters can be derived, identifying the set of anchors seems to be challenging.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Consistent Unsupervised Estimators for Anchored PCFGs

Clark

Fijalkow

2020

Transactions of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Learning probabilistic context-free grammars (PCFGs) from strings is a classic problem in computational linguistics since Horning ( 1969 ). Here we present an algorithm based on distributional learning that is a consistent estimator for a large class of PCFGs that satisfy certain natural conditions including being anchored (Stratos et al., 2016 ). We proceed via a reparameterization of (top–down) PCFGs that we call a bottom–up weighted context-free grammar. We show that if the grammar is anchored and satisfies additional restrictions on its ambiguity, then the parameters can be directly related to distributional properties of the anchoring strings; we show the asymptotic correctness of a naive estimator and present some simulations using synthetic data that show that algorithms based on this approach have good finite sample behavior.

show abstract

Section: Structural Conditions On Grammarsmentioning

confidence: 74%

Section: Discussionmentioning

confidence: 99%

Consistent Unsupervised Estimators for Anchored PCFGs

Clark

Fijalkow

2020

Transactions of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Because of the works on learning context-free grammars Eyraud, 2007, Clark et al, 2010] and beyond context-free [Clark and Yoshinaka, 2014], we conjecture that it would not be the case. Indeed, bi-directional RNN use information from both the prefix and the suffix of a given element, which positions them within the spectrum of distributional learning [Clark and Yoshinaka, 2016]. The work on spectral learning of non finite state models [Bailly et al, 2013] would then be then a good starting point for a distillation process for these networks.…”

Section: Discussionmentioning

confidence: 99%

Distillation of Weighted Automata from Recurrent Neural Networks using a Spectral Approach

Eyraud¹,

Ayache²

2020

Preprint

View full text Add to dashboard Cite

This paper is an attempt to bridge the gap between deep learning and grammatical inference. Indeed, it provides an algorithm to extract a (stochastic) formal language from any recurrent neural network trained for language modelling. In detail, the algorithm uses the already trained network as an oracle -and thus does not require the access to the inner representation of the black-box -and applies a spectral approach to infer a weighted automaton.As weighted automata compute linear functions, they are computationally more efficient than neural networks and thus the nature of the approach is the one of knowledge distillation. We detail experiments on 62 data sets (both synthetic and from real-world applications) that allow an in-depth study of the abilities of the proposed algorithm. The results show the WA we extract are good approximations of the RNN, validating the approach. Moreover, we show how the process provides interesting insights toward the behavior of RNN learned on data, enlarging the scope of this work to the one of explainability of deep learning models.

show abstract

“…However, it turns out that MIX is substitutable, so such an account would not be supported by Distributional Learning as a model of language acquisition. Proposition 13 (Clark and Yoshinaka, 2016).…”

Section: [)]mentioning

confidence: 99%

Learnability and Overgeneration in Computational Syntax

Hao¹

2019

View full text Add to dashboard Cite

Distributional Learning of Context-Free and Multiple Context-Free Grammars

Cited by 13 publications

References 57 publications

Consistent Unsupervised Estimators for Anchored PCFGs

Consistent Unsupervised Estimators for Anchored PCFGs

Distillation of Weighted Automata from Recurrent Neural Networks using a Spectral Approach

Learnability and Overgeneration in Computational Syntax

Contact Info

Product

Resources

About