The Korean Morphologically Tight-Fitting Tokenizer for Noisy User-Generated Texts

Lee, Sangah; Shin, Hyopil

doi:10.18653/v1/2021.wnut-1.45

Cited by 3 publications

(1 citation statement)

References 13 publications

(7 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some have tried to tackle it as a text normalization problem during preprocessing (Han et al, 2013;Supranovich and Patsepnia, 2015;Benamar et al, 2021;Demir and Topcu, 2022). Others have attempted to get to the root of the problem by improving the tokenization algorithm or using a character-based model (Hofmann et al, 2021;Lee and Shin, 2021;Tay et al, 2021;Wang et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

BERTwich: Extending BERT’s Capabilities to Model Dialectal and Noisy Text

Srivastava,

Chiang

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Real-world NLP applications often deal with nonstandard text (e.g., dialectal, informal, or misspelled text). However, language models like BERT deteriorate in the face of dialect variation or noise. How do we push BERT's modeling capabilities to encompass nonstandard text? Fine-tuning helps, but it is designed for specializing a model to a task and does not seem to bring about the deeper, more pervasive changes needed to adapt a model to nonstandard language. In this paper, we introduce the novel idea of sandwiching BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. We find that our approach, paired with recent work on including character-level noise in fine-tuning data, can promote zero-shot transfer to dialectal text, as well as reduce the distance in the embedding space between words and their noisy counterparts.

show abstract