Visually Grounded Neural Syntax Acquisition

Shi, Haoyue; Mao, Jiayuan; Gimpel, Kevin; Livescu, Karen

doi:10.18653/v1/p19-1180

Cited by 64 publications

(93 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Shen et al (2018Shen et al ( , 2019 learn tree structures through soft gating layers within neural language models, while Drozdov et al (2019) combine recursive autoencoders with the inside-outside algorithm. Kim et al (2019) train unsupervised recurrent neural network grammars with a structured inference network to induce latent trees, and Shi et al (2019) utilize image captions to identify and ground constituents.…”

Section: Related Workmentioning

confidence: 99%

Compound Probabilistic Context-Free Grammars for Grammar Induction

Kim

Dyer

Rushton

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

216

View full text Add to dashboard Cite

We study a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context-free grammar.In contrast to traditional formulations which learn a single stochastic grammar, our context-free rule probabilities are modulated by a per-sentence continuous latent variable, which induces marginal dependencies beyond the traditional context-free assumptions. Inference in this grammar is performed by collapsed variational inference, in which an amortized variational posterior is placed on the continuous variable, and the latent trees are marginalized with dynamic programming. Experiments on English and Chinese show the effectiveness of our approach compared to recent state-of-theart methods for grammar induction. probabilities on previous history, i.e. πz,T →w t ∝ exp(u w f2([wT ; z; ht])) where ht is the hidden state from an LSTM over x

show abstract

Section: Related Workmentioning

confidence: 99%

Compound Probabilistic Context-Free Grammars for Grammar Induction

Kim

Dyer

Rushton

2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

216

View full text Add to dashboard Cite

show abstract

“…The next step is to derive the query representation based on the recursively extracted constituent nodes in the LST. In previous work [4,36], only the last constituent node is used for task-specific inference. However, as mentioned previously, the complex query usually consists of multiple visual concepts and their reference descriptions, in which some concepts or reference descriptions may not have clear visual evidence or just have very short temporal durations in the videos.…”

Section: Tree-augmented Query Encodermentioning

confidence: 99%

Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval

Yang

Dong

Cao

et al. 2020

Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

106

View full text Add to dashboard Cite

The rapid growth of user-generated videos on the Internet has intensified the need for text-based video retrieval systems. Traditional methods mainly favor the concept-based paradigm on retrieval with simple queries, which are usually ineffective for complex queries that carry far more complex semantics. Recently, embedding-based paradigm has emerged as a popular approach. It aims to map the queries and videos into a shared embedding space where semantically-similar texts and videos are much closer to each other. Despite its simplicity, it forgoes the exploitation of the syntactic structure of text queries, making it suboptimal to model the complex queries. To facilitate video retrieval with complex queries, we propose a Tree-augmented Cross-modal Encoding method by jointly learning the linguistic structure of queries and the temporal representation of videos. Specifically, given a complex user query, we first recursively compose a latent semantic tree to structurally describe the text query. We then design a tree-augmented query encoder to derive structure-aware query representation and a temporal attentive video encoder to model the temporal characteristics of videos. Finally, both the query and videos are mapped into a joint embedding space for matching and ranking. In this approach, we have a better understanding and modeling of the complex queries, thereby achieving a better video retrieval performance. Extensive experiments on large scale video retrieval benchmark datasets demonstrate the effectiveness of our approach. CCS CONCEPTS • Information systems → Multimedia and multimodal retrieval; Video search.

show abstract

“…Kim et al (2019b) employ unsupervised recurrent neural network grammars, and Kim et al (2019a) employ compound probabilistic context free grammars. Shi et al (2019) show how image captions can be successfully leveraged to identify constituents in sentences. None of these papers performs an explicit analysis of differences between languages.…”

Section: Related Workmentioning

confidence: 99%

Neural Unsupervised Parsing Beyond English

Kann

Mohananey

Bowman

et al. 2019

Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)

View full text Add to dashboard Cite

Recently, neural network models which automatically infer syntactic structure from raw text have started to achieve promising results. However, earlier work on unsupervised parsing shows large performance differences between non-neural models trained on corpora in different languages, even for comparable amounts of data. With that in mind, we train instances of the PRPN architecture (Shen et al., 2018a)-one of these unsupervised neural network parsers-for Arabic, Chinese, English, and German. We find that (i) the model strongly outperforms trivial baselines and, thus, acquires at least some parsing ability for all languages; (ii) good hyperparameter values seem to be universal; (iii) how the model benefits from larger training set sizes depends on the corpus, with the model achieving the largest performance gains when increasing the number of sentences from 2,500 to 12,500 for English. In addition, we show that, by sharing parameters between the related languages German and English, we can improve the model's unsupervised parsing F1 score by up to 4% in the low-resource setting.

show abstract

Visually Grounded Neural Syntax Acquisition

Cited by 64 publications

References 55 publications

Compound Probabilistic Context-Free Grammars for Grammar Induction

Compound Probabilistic Context-Free Grammars for Grammar Induction

Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval

Neural Unsupervised Parsing Beyond English

Contact Info

Product

Resources

About