When Are Tree Structures Necessary for Deep Learning of Representations?

Abstract

Recursive neural models, which use syntactic parse trees to recursively generate representations bottom-up, are a popular architecture. But there have not been rigorous evaluations showing for exactly which tasks this syntax-based method is appropriate. In this paper we benchmark recursive neural models against sequential recurrent neural models (simple recurrent and LSTM models), enforcing apples-to-apples comparison as much as possible. We investigate 4 tasks: (1) sentiment classification at the sentence level and phrase level; (2) matching questions to answer-phrases; (3) discourse parsing; (4) semantic relation extraction (e.g., component-whole between nouns).Our goal is to understand better when, and why, recursive models can outperform simpler models. We find that recursive models help mainly on tasks (like semantic relation extraction) that require associating headwords across a long distance, particularly on very long sequences. We then introduce a method for allowing recurrent models to achieve similar performance: breaking long sentences into clause-like units at punctuation and processing them separately before combining. Our results thus help understand the limitations of both classes of models, and suggest directions for improving recurrent models.

Dictionaries and distributions: Combining expert knowledge and large scale textual data content analysis

“…This has allowed for the simplified creation of large labeled datasets, ideal for the application of unsupervised learning methods. Approaches have continued to evolve, both in terms of problem formulation (Chen et al, 2015) and as the full weight of modern machine learning techniques have been brought to bear (Socher et al, 2013;Tai et al, 2015;Li et al, 2015).…”

Large-scale Opinion Relation Extraction with Distantly Supervised Neural Network

“…Neural network classifiers are popular for relation extraction recently. Many of them focus on fully supervised settings, recurrent neural networks (RNN) and convolutional neural networks (CNN) (Vu et al, 2016;Zeng et al, 2015;Xu et al, 2015a;Xu et al, 2015b;Zhang and Wang, 2015), sequence models and tree models are investigated (Li et al, 2015;dos Santos et al, 2015). One similar network structure to our model is proposed in (Miwa and Bansal, 2016).…”

A Language-Independent Neural Network for Event Detection

“…Compared with previous neural models, we keep the advantage of convolutional neural network (Nguyen and Grishman, 2015) in capturing local contexts. Besides, we also incorporate a Bi-directional LSTM to model the preceding and following information of a word as it has been commonly accepted that LSTM is good at capturing long-term dependencies in a sequence (Tang et al, 2015b;Li et al, 2015a).…”