Finite-state script normalization and processing utilities: The Nisaba Brahmic library

Johny, Cibu C; Wolf-Sonkin, Lawrence; Gutkin, Alexander; Roark, Brian

doi:10.18653/v1/2021.eacl-demos.3

Cited by 5 publications

(5 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Firstly, normalize the information attributes of the power grid around each airport in the unit [7][8], which means that all attribute values are mapped between 0 and 1. The processing formula is as follows:…”

Section: Information Fusion Of the Power Grid Around The Airportmentioning

confidence: 99%

A transmission line selection algorithm for airport surrounding power grid based on information fusion model and Dijkstra algorithm

Yin,

Zhang,

Zang

et al. 2024

Fourth International Conference on Mechanical, Electronics, and Electrical and Automation Control (METMS 2024)

View full text Add to dashboard Cite

Section: Information Fusion Of the Power Grid Around The Airportmentioning

confidence: 99%

A transmission line selection algorithm for airport surrounding power grid based on information fusion model and Dijkstra algorithm

Yin,

Zhang,

Zang

et al. 2024

Fourth International Conference on Mechanical, Electronics, and Electrical and Automation Control (METMS 2024)

View full text Add to dashboard Cite

“…All the native script data was normalized using Unicode NFC (Whistler, 2021). The data was then further transformed using language-specific visual normalization for Brahmic and Perso-Arabic writing systems using the Nisaba script normalization library (Johny et al, 2021;Gutkin et al, 2022). Both NFC and visual normalization operations preserve visual invariance of the input text, with visual normalization handling many ambiguous cases that fall outside the scope of standard NFC.…”

Section: D43 Data Preparationmentioning

confidence: 99%

XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages

Ruder,

Clark,

Gutkin

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Data scarcity is a crucial issue for the development of highly multilingual NLP systems. Yet for many under-represented languages (ULs)-languages for which NLP research is particularly far behind in meeting user needsit is feasible to annotate small amounts of data. Motivated by this, we propose XTREME-UP, a benchmark defined by: its focus on the scarcedata scenario rather than zero-shot; its focus on user-centric tasks-tasks with broad adoption by speakers of high-resource languages; and its focus on under-represented languages where this scarce-data scenario is most realistic. XTREME-UP evaluates the capabilities of language models across 88 under-represented languages over 9 key user-centric technologies including ASR, OCR, MT, and information access tasks that are of general utility. We create new datasets for OCR, autocomplete, question answering, semantic parsing, and transliteration, and build on and refine existing datasets for other tasks. XTREME-UP provides a methodology for evaluating many modeling scenarios including text-only, multimodal (vision, audio, and text), supervised parameter tuning, and in-context learning. 1 We evaluate commonly used models on the benchmark. 2 * Equal contribution. We list detailed contributions in §A. 1 Our results in §4 indicate that few-shot in-context learning is less effective than fine-tuning on 100s of examples for ULs. We advocate for comparing such approaches directly as the community explores XTREME-UP.2 https://github.com/google-research/xtreme-up * Average time for read speech. † Based on mean typing speed (Dhakal et al., 2018) and average sentence length in Universal Dependencies (de Marneffe et al., 2021). ‡ An annotated example can be used for both task directions. For document OCR, each example is a whole page; for example, 9 pages of training images per language corresponds to approximately 290 lines of output text per language.

show abstract

“…Our transliterator uses the open-source Nisaba library of finite-state script normalization and processing utilities (Johny et al, 2021). 12 The script operations in Nisaba are efficiently and succinctly represented as weighted finite-state transducers (WFSTs) using Pynini finite-state grammars (Gorman, 2016;Gorman & Sproat, 2021).…”

Section: Transliteration For Meiteilonmentioning

confidence: 99%

“…The four component WFSTs are compiled into the final transliteration WFST T = N • R • P • R −1 , where "•" denotes FST composition operation (Mohri, 2009). The first component transducer N implements visual normalization of the Bengali script input that consists of visually invariant normalization transformations including NFC (Johny et al, 2021). This is followed by the Meiteilon-specific Bengali to Latin script many-to-one mapping R that produces Latin script output in ISO 15919 format (ISO, 2001) augmented with some placeholder markers required for the next processing stage.…”

Section: Transliteration For Meiteilonmentioning

confidence: 99%

Building Machine Translation Systems for the Next Thousand Languages

Bapna¹,

Caswell²,

Kreutzer³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper we share findings from our effort to build practical machine translation (MT) systems capable of translating across over one thousand languages. We describe results in three research domains: (i) Building clean, web-mined datasets for 1500+ languages by leveraging semi-supervised pre-training for language identification and developing data-driven filtering techniques; (ii) Developing practical MT models for under-served languages by leveraging massively multilingual models trained with supervised parallel data for over 100 high-resource languages and monolingual datasets for an additional 1000+ languages; and (iii) Studying the limitations of evaluation metrics for these languages and conducting qualitative analysis of the outputs from our MT models, highlighting several frequent error modes of these types of models. Using this approach, we add 24 new languages to Google Translate, the product's largest increase in language coverage to-date. We hope that our work provides useful insights to practitioners working towards building MT systems for currently understudied languages, and highlights research directions that can complement the weaknesses of massively multilingual models in data-sparse settings.

show abstract

Finite-state script normalization and processing utilities: The Nisaba Brahmic library

Cited by 5 publications

References 23 publications

A transmission line selection algorithm for airport surrounding power grid based on information fusion model and Dijkstra algorithm

A transmission line selection algorithm for airport surrounding power grid based on information fusion model and Dijkstra algorithm

XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages

Building Machine Translation Systems for the Next Thousand Languages

Contact Info

Product

Resources

About