XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages

Abstract

Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs)-languages for which NLP research is particularly far behind in meeting user needsit is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarcedata scenario rather than zero-shot; its focus on user-centric tasks-tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario is most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, question answering, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides a methodology for evaluating many modeling scenarios including text-only, multimodal (vision, audio, and text), supervised parameter tuning, and in-context learning. 1 We evaluate commonly used models on the benchmark. 2 * Equal contribution. We list detailed contributions in §A. 1 Our results in §4 indicate that few-shot in-context learning is less effective than fine-tuning on 100s of examples for ULs. We advocate for comparing such approaches directly as the community explores XTREME-UP.2 https://github.com/google-research/xtreme-up * Average time for read speech. † Based on mean typing speed (Dhakal et al., 2018) and average sentence length in Universal Dependencies (de Marneffe et al., 2021). ‡ An annotated example can be used for both task directions. For document OCR, each example is a whole page; for example, 9 pages of training images per language corresponds to approximately 290 lines of output text per language.