Meta-Learning a Cross-lingual Manifold for Semantic Parsing

Sherborne, Tom; Lapata, Mirella

doi:10.1162/tacl_a_00533

Cited by 6 publications

(2 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use the MultiATIS++SQL dataset from Sherborne and Lapata (2022) comprising gold parallel utterances in English, French, Portuguese, Spanish, German and Chinese (from Xu et al (2020)) paired to executable SQL output logical forms (from Iyer et al (2017)). The model follows Sherborne and Lapata (2023), as an encoder-decoder Transformer model based on mBART50 (Tang et al, 2021). The parser generates valid SQL queries and performance is measured as exact-match denotation accuracy-the proportion of output queries returning identical database results relative to gold SQL queries.…”

Section: Semantic Parsing (Sp)mentioning

confidence: 99%

Extrinsic Evaluation of Machine Translation Metrics

Moghe¹,

Sherborne²,

Steedman³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Automatic machine translation (MT) metrics are widely used to distinguish the quality of machine translation systems across large test sets (i.e., system-level evaluation). However, it is unclear if automatic metrics can reliably distinguish good translations from bad at the sentence level (i.e., segment-level evaluation). We investigate how useful MT metrics are at detecting segment-level quality by correlating metrics with the translation utility for downstream tasks. We evaluate the segment-level performance of widespread MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks (dialogue state tracking, question answering, and semantic parsing). For each task, we have access to a monolingual task-specific model and a translation model. We calculate the correlation between the metric's ability to predict a good/bad translation with the success/failure on the final task for machine-translated test sentences. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of downstream outcomes. We also find that the scores provided by neural metrics are not interpretable, in large part due to having undefined ranges. We synthesise our analysis into recommendations for future MT metrics to produce labels rather than scores for more informative interaction between machine translation and multilingual language understanding.

show abstract

Section: Semantic Parsing (Sp)mentioning

confidence: 99%

Extrinsic Evaluation of Machine Translation Metrics

Moghe¹,

Sherborne²,

Steedman³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

show abstract

“…We focus on two threads of related work: (1) metalearning for cross-lingual transfer and (2) training data selection. Sherborne and Lapata (2023) and Wu et al (2020b) use meta-learning for crosslingual NER and Semantic Parsing with a slight enhancement in minimal resources. X-MAML (Nooralahzadeh et al, 2020) combines the MAML and cross-lingual transfer method based on PLM and demonstrates improvement in zero-shot and few-shot settings.…”

Section: Related Workmentioning

confidence: 99%

Good Meta-tasks Make A Better Cross-lingual Meta-transfer Learning for Low-resource Languages

Wu,

Guo,

Cui

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Model-agnostic meta-learning has garnered attention as a promising technique for enhancing few-shot cross-lingual transfer learning in lowresource scenarios. However, little attention was paid to the impact of data selection strategies on this cross-lingual meta-transfer method, particularly the sampling of cross-lingual metatraining data (i.e. meta-tasks) at the syntactic level to reduce language gaps. In this paper, we propose a Meta-Task Collector-based Crosslingual Meta-Transfer framework (MeTaCo-XMT) to adapt different data selection strategies to construct meta-tasks for meta-transfer learning. Syntactic differences have an effect on transfer performance, so we consider a syntactic similarity sampling strategy and propose a syntactic distance metric model consisting of a syntactic encoder block based on the pretrained model and a distance metric block using Word Move's Distance (WMD). Additionally, we conduct experiments with three different data selection strategies to instantiate our framework and analyze their performance impact. Experimental results on two multilingual NLP datasets, WikiAnn and TydiQA, demonstrate the significant superiority of our approach compared to existing strong baselines 1 .

show abstract