One Wug, Two Wug+s Transformer Inflection Models Hallucinate Affixes

Samir, Farhan; Silfverberg, Miikka

doi:10.18653/v1/2022.computel-1.5

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our experiment with restricting the hallucination process to generate forms that are phonotactically attested (bigram) in the training data revealed that its benefit was found only in very restricted conditions depending on the amount of hallucinated samples and the specific language (and presumably the inflectional pattern). Our findings are in agreement with the detailed error analyses of data hallucination techniques by Samir and Silfverberg (2022) which concluded that hallucination is not a one-size-fits-all technique and it must be used with caution and requires closer inspection depending on the type of morphological inflections.…”

Section: Discussionsupporting

confidence: 91%

HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories

Ramarao¹,

Zinova²,

Tang³

et al. 2022

Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

View full text Add to dashboard Cite

This paper presents the submission by the HeiMorph team to the SIGMORPHON 2022 task 2 of Morphological Acquisition Trajectories. Across all experimental conditions, we have found no evidence for the so-called Ushaped development trajectory. Our submitted systems achieve an average test accuracies of 55.5% on Arabic, 67% on German and 73.38% on English. We found that, bigram hallucination provides better inferences only for English and Arabic and only when the number of hallucinations remains low.

show abstract

Section: Discussionsupporting

confidence: 91%

HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories

Ramarao¹,

Zinova²,

Tang³

et al. 2022

Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

View full text Add to dashboard Cite

show abstract

“…The data hallucination method introduced by Anastasopoulos and Neubig (2019) can sometimes create invalid examples due to phonological alternations as noted by Samir and Silfverberg (2022). For example, given the English inflection example like+VERB+PAST → liked, their approach will first identify the longest common subsequence of the lemma and word form, that is, like and will then replace this with a random character sequence, for example xyz.…”

Section: Lemma Copyingmentioning

confidence: 99%

Generalizing Morphological Inflection Systems to Unseen Lemmas

Yang¹,

Yang²,

Nicolai³

et al. 2022

Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

Self Cite

View full text Add to dashboard Cite

This paper presents experiments on morphological inflection using data from the SIGMORPHON-UniMorph 2022 Shared Task 0: Generalization and Typologically Diverse Morphological Inflection. We present a transformer inflection system, which enriches the standard transformer architecture with reverse positional encoding and type embeddings. We further apply data hallucination and lemma copying to augment training data. We train models using a two-stage procedure: (1) We first train on the augmented training data using standard backpropagation and teacher forcing.(2) We then continue training with a variant of the scheduled sampling algorithm dubbed student forcing. Our system delivers competitive performance under the small and large data conditions on the shared task datasets. * *The first two authors contributed equally. 1 Note, our system is not an official shared task submission because we submitted our final results after the shared task deadline.

show abstract

“…Nevertheless, there is reason for optimism. Several works have shown that automatic inflection models come much closer to a compositional solution when the human-annotated dataset is complimented by a synthetic data-augmentation procedure (Liu and Hulden, 2022;Silfverberg et al, 2017;Anastasopoulos and Neubig, 2019;Lane and Bird, 2020;Samir and Silfverberg, 2022), where morphological affixes are identified and attached to synthetic lexemes distinct from those in the training dataset (Fig. 2).…”

Section: Introductionmentioning

confidence: 99%

Understanding Compositional Data Augmentation in Typologically Diverse Morphological Inflection

Samir,

Silfverberg

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Data augmentation techniques are widely used in low-resource automatic morphological inflection to overcome data sparsity. However, the full implications of these techniques remain poorly understood. In this study, we aim to shed light on the theoretical aspects of the prominent data augmentation strategy STEM-CORRUPT (Silfverberg et al., 2017;Anastasopoulos and Neubig, 2019), a method that generates synthetic examples by randomly substituting stem characters in gold standard training examples. To begin, we conduct an information-theoretic analysis, arguing that STEMCORRUPT improves compositional generalization by eliminating spurious correlations between morphemes, specifically between the stem and the affixes. Our theoretical analysis further leads us to study the sampleefficiency with which STEMCORRUPT reduces these spurious correlations. Through evaluation across seven typologically distinct languages, we demonstrate that selecting a subset of datapoints with both high diversity and high predictive uncertainty significantly enhances the data-efficiency of STEMCORRUPT. However, we also explore the impact of typological features on the choice of the data selection strategy and find that languages incorporating a high degree of allomorphy and phonological alternations derive less benefit from synthetic examples with high uncertainty. We attribute this effect to phonotactic violations induced by STEMCORRUPT, emphasizing the need for further research to ensure optimal performance across the entire spectrum of natural language morphology. 1

show abstract

One Wug, Two Wug+s Transformer Inflection Models Hallucinate Affixes

Cited by 3 publications

References 0 publications

HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories

HeiMorph at SIGMORPHON 2022 Shared Task on Morphological Acquisition Trajectories

Generalizing Morphological Inflection Systems to Unseen Lemmas

Understanding Compositional Data Augmentation in Typologically Diverse Morphological Inflection

Contact Info

Product

Resources

About