Cooperative Learning of Disjoint Syntax and Semantics

Havrylov, Serhii; Kruszewski, Germán; Joulin, Armand

doi:10.48550/arxiv.1902.09393

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Models such as a flat RNN will fail to capture the hierarchical structure of this task. However, if a model can induce an explicit latent z, the parse tree of the expression, then the task is easy to learn by a tree-RNN model p(y|x, z) (Yogatama et al, 2016;Havrylov et al, 2019).…”

Section: Motivating Case Studymentioning

confidence: 99%

Torch-Struct: Deep Structured Prediction Library

Rushton¹

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations

View full text Add to dashboard Cite

The literature on structured prediction for NLP describes a rich collection of distributions and algorithms over sequences, segmentations, alignments, and trees; however, these algorithms are difficult to utilize in deep learning frameworks. We introduce Torch-Struct, a library for structured prediction designed to take advantage of and integrate with vectorized, auto-differentiation based frameworks. Torch-Struct includes a broad collection of probabilistic structures accessed through a simple and flexible distribution-based API that connects to any deep learning model. The library utilizes batched, vectorized operations and exploits auto-differentiation to produce readable, fast, and testable code. Internally, we also include a number of general-purpose optimizations to provide cross-algorithm efficiency. Experiments show significant performance gains over fast baselines. Case studies demonstrate the benefits of the library. Torch-Struct is available at https://github.com/ harvardnlp/pytorch-struct.

show abstract

Section: Motivating Case Studymentioning

confidence: 99%

Torch-Struct: Deep Structured Prediction Library

Rushton¹

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations

View full text Add to dashboard Cite

show abstract

“…The contextual transfer boosts the performance of both the participating models by enriching the intermediate-level representations that share the backpropagation from both tasks. We note works [7,14,39] that recommend joint training of tasks on complementary contexts like emotion and sentiment classification. However, in contrast to these, we use joint training to leverage more complex modalities such as syntax and semantics.…”

Section: Introductionmentioning

confidence: 99%

DeSCoVeR: Debiased Semantic Context Prior for Venue Recommendation

Rajanala

Pal

Singh

et al. 2022

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

We present a novel semantic context prior-based venue recommendation system that uses only the title and the abstract of a paper. Based on the intuition that the text in the title and abstract have both semantic and syntactic components, we demonstrate that joint training of a semantic feature extractor and syntactic feature extractor collaboratively leverages meaningful information that helps in recommending venues for paper publication. The proposed methodology that we call DeSCoVeR first elicits these semantic and syntactic features using a Neural Topic Model and text classifier, respectively. The model then executes a transfer learning optimization procedure to perform a contextual transfer between the feature distributions of the Neural Topic Model and the text classifier during the training phase. DeSCoVeR also mitigates the document-level label bias using a Causal back-door path criterion and a sentence-level keyword bias removal technique. Experiments on the DBLP dataset show that DeSCoVeR outperforms the stateof-the-art methods.

show abstract

“…In fact, the Gumbel-Softmax trick naturally translates to structured variables when arg max operator is applied over a structured domain rather than component-wise [34]. In contrast, score function estimators are now less common in structured domain, with a few exceptions such as [50,14]. The primary difficulty is the sample score function: neither Gibbs distributions, nor distribution defined through a generative process have a general shortcut to compute it.…”

Section: Introductionmentioning

confidence: 99%

Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces

Struminsky¹,

Gadetsky²,

Rakitin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Structured latent variables allow incorporating meaningful prior knowledge into deep learning models. However, learning with such variables remains challenging because of their discrete nature. Nowadays, the standard learning approach is to define a latent variable as a perturbed algorithm output and to use a differentiable surrogate for training. In general, the surrogate puts additional constraints on the model and inevitably leads to biased gradients. To alleviate these shortcomings, we extend the Gumbel-Max trick to define distributions over structured domains. We avoid the differentiable surrogates by leveraging the score function estimators for optimization. In particular, we highlight a family of recursive algorithms with a common feature we call stochastic invariant. The feature allows us to construct reliable gradient estimates and control variates without additional constraints on the model. In our experiments, we consider various structured latent variable models and achieve results competitive with relaxation-based counterparts.

show abstract

Cooperative Learning of Disjoint Syntax and Semantics

Cited by 3 publications

References 0 publications

Torch-Struct: Deep Structured Prediction Library

Torch-Struct: Deep Structured Prediction Library

DeSCoVeR: Debiased Semantic Context Prior for Venue Recommendation

Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces

Contact Info

Product

Resources

About