Multi-space Variational Encoder-Decoders for Semi-supervised Labeled Sequence Transduction

Abstract

Labeled sequence transduction is a task of transforming one sequence into another sequence that satisfies desiderata specified by a set of labels. In this paper we propose multi-space variational encoderdecoders, a new model for labeled sequence transduction with semi-supervised learning. The generative model can use neural networks to handle both discrete and continuous latent variables to exploit various features of data. Experiments show that our model provides not only a powerful supervised framework but also can effectively take advantage of the unlabeled data. On the SIGMORPHON morphological inflection benchmark, our model outperforms single-model state-ofart results by a large margin for the majority of languages. 1

Morphological Inflection Generation with Multi-space Variational Encoder-Decoders

“…In this work, we further examine the method proposed in (Zhou and Neubig, 2017) for the shared task of SIGMORPHON 2017 on 52 languages and demonstrate the effectiveness of this approach. We will further improve our model's sophistication by investigating strategies for choosing appropriate semi-supervised data, and examining the model's performance on languages with a high inflection level.…”

Exact Hard Monotonic Attention for Character-Level Transduction

“…, y |Σy| }, respectively. In tasks where the tag is provided, i.e., labeled transduction (Zhou and Neubig, 2017), we denote the tag as an ordered set t ∈ Σ * t with a finite tag vocabulary Σ t = {t 1 , . .…”

Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection

“…com/sigmorphon/conll2017/tree/master/ evaluation Zhou and Neubig (2017), where even after 150k unlabeled examples, performance still appears to be increasing.) After controlling for the amount of additional data, we see only a small benefit from autoencoding corpus words (AE-CW) rather than random strings (AE- Figure 2: The accuracy of our best systems on all languages.…”

“…We train our system using Stochastic Gradient Descent. Our system is implemented using the Dynet toolkit (Neubig et al, 2017) 4 and our code is freely available. 5 There are three hyper-parameters in our system: the character embedding dimension, the size of the hidden layer of the LSTM models and the size of the hidden layer of the attention network.…”

“…This makes a variety of natural language processing tasks more challenging, as it increases the number of words in a language drastically; a problem morphological analysis and generation help to mitigate. However, a big issue when developing methods for morphological processing is that for many morphologically rich languages, there are only few or no relevant training data available, making it impossible to train state-of-the-art machine learning models (e.g., Kann and Schütze, 2016b;Aharoni et al, 2016;Zhou and Neubig, 2017)). This is the motivation for the CoNLL-SIGMORPHON-2017 shared task on universal morphological reinflection (Cotterell et al, 2017a), which animates the development of systems for as many as 52 different languages in 6 different low-resource settings for morphological reinflection: to generate an inflected form, given a target morphological tag and either the lemma (task 1) or a partial paradigm (task 2).…”