Challenges and Strategies in Cross-Cultural NLP

Hershcovich, Daniel; Frank, Stella; Lent, Heather; Lhoneux, Miryam de; Abdou, Mostafa; Brandl, Stephanie; Bugliarello, Emanuele; Piqueras, Laura Cabello; Chalkidis, Ilias; Cui, Ruixiang; Fierro, Constanza; Margatina, Katerina; Rust, Phillip; Søgaard, Anders

doi:10.18653/v1/2022.acl-long.482

Cited by 30 publications

(42 citation statements)

References 78 publications

(85 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…sal patterns in translation (simplification, normalisation, and explicitation) and linguistic interference: the source language spills its lexical and structural properties over the target language (Lembersky et al, 2012;Volansky et al, 2015). These artefacts, introduced by the translation procedure, could make the dataset not representative of real-life dialogue and cultural context of the target language (Hershcovich, Frank, Lent, de Lhoneux, Abdou, Brandl, Bugliarello, Piqueras, Chalkidis, Cui, et al, 2022) and instead give an edge to translation-based cross-lingual transfer. Hence, the evaluation performance becomes unreliable and excessively optimistic (Artetxe et al, 2020).…”

Section: Outlook For Multilingual Tod Datasetsmentioning

confidence: 99%

“…In the first stage, native speakers were asked to translate and localise slot values; in the second stage, another group of human subjects translated or localised the entire phrase using the slot task output provided by the first worker. Going beyond research in ToD, Hershcovich et al (2022) suggest that collecting multilingual data within (large) local communities results in culturally richer data and avoids imposing English-driven use cases. Additionally, it can reduce the per-person manual effort (by dividing the work between more people) which is often a bottleneck in the data collection process.…”

Section: Outlook For Multilingual Tod Datasetsmentioning

confidence: 99%

“…linguistically versus their inability to transfer between different cultural contexts and realities (Hershcovich et al, 2022). Evaluation of ToD systems still adds another layer of difficulty, typically not encountered in the above-mentioned benchmarks for general-purpose NLP tasks.…”

Section: Human-centred Evaluation Of Multilingual Tod Systemsmentioning

confidence: 99%

“…These notions are highly culture-dependent (Kendall, Linden, & Murray, 2005). It means that an appropriate response of a dialogue system in the Anglosphere can be wrong, impolite and even aggressive when ported without any adaptation to a different culture (Hershcovich et al, 2022). Thus, accounting for ethics, cross-cultural differences and social norms is crucial when evaluating multilingual NLP in general (Solaiman & Dennison, 2021) and dialogue systems in particular.…”

Section: Human-centred Evaluation Of Multilingual Tod Systemsmentioning

confidence: 99%

See 3 more Smart Citations

Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

Razumovskaia

Glavaš

Majewska

et al. 2022

jair

View full text Add to dashboard Cite

In task-oriented dialogue (ToD), a user holds a conversation with an artificial agent with the aim of completing a concrete task. Although this technology represents one of the central objectives of AI and has been the focus of ever more intense research and development efforts, it is currently limited to a few narrow domains (e.g., food ordering, ticket booking) and a handful of languages (e.g., English, Chinese). This work provides an extensive overview of existing methods and resources in multilingual ToD as an entry point to this exciting and emerging field. We find that the most critical factor preventing the creation of truly multilingual ToD systems is the lack of datasets in most languages for both training and evaluation. In fact, acquiring annotations or human feedback for each component of modular systems or for data-hungry end-to-end systems is expensive and tedious. Hence, state-of-the-art approaches to multilingual ToD mostly rely on (zero- or few-shot) cross-lingual transfer from resource-rich languages (almost exclusively English), either by means of (i) machine translation or (ii) multilingual representations. These approaches are currently viable only for typologically similar languages and languages with parallel / monolingual corpora available. On the other hand, their effectiveness beyond these boundaries is doubtful or hard to assess due to the lack of linguistically diverse benchmarks (especially for natural language generation and end-to-end evaluation). To overcome this limitation, we draw parallels between components of the ToD pipeline and other NLP tasks, which can inspire solutions for learning in low-resource scenarios. Finally, we list additional challenges that multilinguality poses for related areas (such as speech, fluency in generated text, and human-centred evaluation), and indicate future directions that hold promise to further expand language coverage and dialogue capabilities of current ToD systems.

show abstract

Section: Outlook For Multilingual Tod Datasetsmentioning

confidence: 99%

Section: Outlook For Multilingual Tod Datasetsmentioning

confidence: 99%

Section: Human-centred Evaluation Of Multilingual Tod Systemsmentioning

confidence: 99%

Section: Human-centred Evaluation Of Multilingual Tod Systemsmentioning

confidence: 99%

See 2 more Smart Citations

Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

Razumovskaia

Glavaš

Majewska

et al. 2022

jair

View full text Add to dashboard Cite

show abstract

“…Furthermore, the assumption that an English KB is a ''canonical'' conceptualization is unjustified, as speakers of other languages may know and care about other entities and relationships (Liu et al, 2021a;Hershcovich et al, 2022a). Therefore, future work must create multilingual SP datasets by sourcing questions from native speakers rather than translating them.…”

Section: Limitationsmentioning

confidence: 99%

Compositional Generalization in Multilingual Semantic Parsing over Wikidata

Cui

Aralikatte

Lent

et al. 2022

Transactions of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Semantic parsing (SP) allows humans to leverage vast knowledge resources through natural interaction. However, parsers are mostly designed for and evaluated on English resources, such as CFQ (Keysers et al., 2020), the current standard benchmark based on English data generated from grammar rules and oriented towards Freebase, an outdated knowledge base. We propose a method for creating a multilingual, parallel dataset of question-query pairs, grounded in Wikidata. We introduce such a dataset, which we call Multilingual Compositional Wikidata Questions (MCWQ), and use it to analyze the compositional generalization of semantic parsers in Hebrew, Kannada, Chinese, and English. While within- language generalization is comparable across languages, experiments on zero-shot cross- lingual transfer demonstrate that cross-lingual compositional generalization fails, even with state-of-the-art pretrained multilingual encoders. Furthermore, our methodology, dataset, and results will facilitate future research on SP in more realistic and diverse settings than has been possible with existing resources.

show abstract

Regional bias in monolingual English language models

Lyu,

Dost,

Koh

et al. 2024

Mach Learn

View full text Add to dashboard Cite

In Natural Language Processing (NLP), pre-trained language models (LLMs) are widely employed and refined for various tasks. These models have shown considerable social and geographic biases creating skewed or even unfair representations of certain groups. Research focuses on biases toward L2 (English as a second language) regions but neglects bias within L1 (first language) regions. In this work, we ask if there is regional bias within L1 regions already inherent in pre-trained LLMs and, if so, what the consequences are in terms of downstream model performance. We contribute an investigation framework specifically tailored for low-resource regions, offering a method to identify bias without imposing strict requirements for labeled datasets. Our research reveals subtle geographic variations in the word embeddings of BERT, even in cultures traditionally perceived as similar. These nuanced features, once captured, have the potential to significantly impact downstream tasks. Generally, models exhibit comparable performance on datasets that share similarities, and conversely, performance may diverge when datasets differ in their nuanced features embedded within the language. It is crucial to note that estimating model performance solely based on standard benchmark datasets may not necessarily apply to the datasets with distinct features from the benchmark datasets. Our proposed framework plays a pivotal role in identifying and addressing biases detected in word embeddings, particularly evident in low-resource regions such as New Zealand.

show abstract

Challenges and Strategies in Cross-Cultural NLP

Cited by 30 publications

References 78 publications

Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems

Compositional Generalization in Multilingual Semantic Parsing over Wikidata

Regional bias in monolingual English language models

Contact Info

Product

Resources

About