MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages

FitzGerald, Jack; Hench, Christopher; Peris, Charith; Mackie, Scott; Rottmann, Kay; Sanchez, Ana M.; Nash, Aaron; Urbach, Liam; Kakarala, Vishesh; Singh, Rajesh; Swetha, Ranganath,; Crist, Laurie; Britan, Misha; Leeuwis, Wouter; Tür, Gökhan; Natarajan, Prem

doi:10.18653/v1/2023.acl-long.235

Cited by 10 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Crowdsourcing has been used to gather utterances and dialogues within particular contexts of use [15,36,38], translate and localize existing data sets [39,40], and elicit reactions to voice stimuli, especially social and paralinguistic characteristics [41,42]. A notable example is the translation and localization of the Amazon MASSIVE data set into 51 languages with Amazon Mechanical Turk [27]. This signals a shift in how co-design is being approached: from the classical model of small-scale focus groups and jams to a larger-scale, online, crowd-driven model that has global reach.…”

Section: Related Workmentioning

confidence: 99%

“…We analyzed patterns related to people, notably gender, as well as machines, notably scorn and abusive conduct [49]. However, we avoided removing "inappropriate" material, such as swear words [27,36]. These forms of exchanges need to be trained into VAs so that VAs can recognize and respond appropriately [37].…”

Section: Related Workmentioning

confidence: 99%

“…The US is a typical oversampled Western nation [9,12], while Japan differs by only one letter on the WEIRD spectrum, i.e., language and culture. Notably, translations of English and Japanese NLP data sets are common, such as the crowdsourcing initiatives of Tatoeba (https://tatoeba .org/en/downloads) and MASSIVE [27]. Yet, biases have been found within these data sets: "missteps" resulting from the crowdsourced translation process [55].…”

Section: Approaching the Co-design Of Vas Cross-culturallymentioning

confidence: 99%

“…Co-design can offer insights that spark new ideas grounded in user needs and mental models [25]. In VA design, crowdsourcing is an emerging solution [15-17, 23, 24, 26] for translating and "coimagining" new NLP data sets, for example, Amazon's MASSIVE project [27]. Still, these efforts have been limited to creation for or translation from a single lingo-cultural context, usually "WEIRD" American English.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Coimagining the Future of Voice Assistants with Cultural Sensitivity

Seaborn,

Sawa,

Watanabe

2024

Human Behavior and Emerging Technologies

View full text Add to dashboard Cite

Voice assistants (VAs) are becoming a feature of our everyday life. Yet, the user experience (UX) is often limited, leading to underuse, disengagement, and abandonment. Co-designing interactions for VAs with potential end-users can be useful. Crowdsourcing this process online and anonymously may add value. However, most work has been done in the English-speaking West on dialogue data sets. We must be sensitive to cultural differences in language, social interactions, and attitudes towards technology. Our aims were to explore the value of co-designing VAs in the non-Western context of Japan and demonstrate the necessity of cultural sensitivity. We conducted an online elicitation study (N=135) where Americans (n=64) and Japanese people (n=71) imagined dialogues (N=282) and activities (N=73) with future VAs. We discuss the implications for coimagining interactions with future VAs, offer design guidelines for the Japanese and English-speaking US contexts, and suggest opportunities for cultural plurality in VA design and scholarship.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Approaching the Co-design Of Vas Cross-culturallymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Coimagining the Future of Voice Assistants with Cultural Sensitivity

Seaborn,

Sawa,

Watanabe

2024

Human Behavior and Emerging Technologies

View full text Add to dashboard Cite

show abstract

“…NusaX (Winata et al, 2023) is a multilingual sentiment analysis dataset comprising 12 languages, including 10 Indonesian regional languages. MASSIVE (FitzGerald et al, 2023) is a multilingual natural language understanding dataset with 51 languages for which we use the intent detection data.…”

Section: Datasetsmentioning

confidence: 99%

Efficient Zero-Shot Cross-lingual Inference via Retrieval

Winata,

Xie,

Radhakrishnan

et al. 2023

Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacifi

View full text Add to dashboard Cite

Resources for building NLP applications, such as data and models, are usually only created and curated for a limited set of high resource languages. Thus, the ability to transfer knowledge to a new language is a key way in which to enable access to NLP technology for a wider population. This paper presents a framework to perform zero-shot inference in a target language by using cross-lingual retrieval from another language where limited annotated data for a comparable domain is available. Results on two large-scale multilingual datasets show that, in this setup, this framework improves over fine-tuning multilingual models or translating annotated data, and achieves results relatively close to fine-tuning the model on the target language directly. These results show that models can be transferred efficiently across languages for a given task and domain, even for languages not covered by multilingual model training approaches.

show abstract