Subword Sampling for Low Resource Word Alignment

Abstract

Annotation projection is an important area in NLP that can greatly contribute to creating language resources for low-resource languages. Word alignment plays a key role in this setting. However, most of the existing word alignment methods are designed for a high resource setting in machine translation where millions of parallel sentences are available. This amount reduces to a few thousands of sentences when dealing with lowresource languages failing the existing established IBM models. In this paper, we propose subword sampling-based alignment of text units. This method's hypothesis is that the aggregation of different granularities of text for certain language pairs can help word-level alignment. For certain languages for which gold-standard alignments exist, we propose an iterative Bayesian optimization framework to optimize selecting possible subwords from the space of possible subword representations of the source and target sentences. We show that the subword sampling method consistently outperforms word-level alignment on six language pairs: English-German, English-French, English-Romanian, English-Persian, English-Hindi, and English-Inuktitut. In addition, we show that the hyperparameters learned for certain language pairs can be applied to other languages at no supervision and consistently improve the alignment results. We observe that using 5K parallel sentences together with our proposed subword sampling approach, we obtain similar F1 scores to the use of 100K's of parallel sentences in existing word-level fastalign/eflomal alignment methods.

SilverAlign: MT-Based Silver Data Algorithm For Evaluating Word Alignment

“…They also compare them with respect to unaligned words, rare words, Part-of-Speech (PoS) tags, and distortion errors. Asgari et al (2020) study word alignment results when using subword-level tokenization and show improved performance with respect to word level. analyze the performance of word aligners regarding different PoS for English/German and show that Eflomal has low performance when aligning links with high distortion.…”

Exploring Enhanced Code-Switched Noising for Pretraining in Neural Machine Translation

“…Awesome-align uses representations from mBERT (Devlin et al, 2019) so, it could scale to the languages the latter is pretrained on. For other languages, such as very low-resource pairs, it could be worth exploring lowresource word aligners (Asgari et al, 2020;Poerner et al, 2018) -though we leave the exploration of the same as part of future work. As for the `base' model, we could use models trained from scratch as a viable alternative (see Table 4) and potentially obtain comparable performance.…”

Incorporating Context into Subword Vocabularies

“…Previous work towards this goal includes algorithms which offer robustness within an existing subword vocabulary (Provilkov et al, 2020;He et al, 2020;Hiraoka, 2022), necessitating modification of either training, inference, or both procedures in the context of LLMs. Others have considered tuning the size of a subword vocabulary (Salesky et al, 2020), or selecting from an enlarged set of possible segmentations (Asgari et al, 2020), for optimizing performance on downstream tasks.…”