Neural Multi-task Text Normalization and Sanitization with Pointer-Generator

Abstract

Text normalization and sanitization are intrinsic components of Natural Language Inferences. In Information Retrieval or Dialogue Generation, normalization of user queries or utterances enhances linguistic understanding by translating non-canonical text to its canonical form, on which many state-of-the-art language models are trained. On the other hand, text sanitization removes sensitive information to guarantee user privacy and anonymity. Existing approaches to normalization and sanitization mainly rely on hand-crafted heuristics and syntactic features of individual tokens while disregarding the linguistic context. Moreover, such context-unaware solutions cannot dynamically determine whether out-of-vocab tokens are misspelt or are entity names. In this work, we formulate text normalization and sanitization as a multi-task text generation approach and propose a neural pointer-generator network based on multihead attention. Its generator effectively captures linguistic context during normalization and sanitization while its pointer dynamically preserves the entities that are generally missing in the vocabulary. Experiments show that our generation approach outperforms both token-based text normalization and sanitization, while the pointer-generator improves the generator-only baseline in terms of BLEU4 score, and classical attentional pointer networks in terms of pointing accuracy.

LexiClean: An annotation tool for rapid multi-task lexical normalisation

“…Lexical normalisation is typically tackled as one of two formulations, either as a sequence-tosequence (seq2seq) (Muller et al, 2019;Nguyen and Cavallari, 2020) or token classification problem (van der Goot and van Noord, 2017;Stewart et al, 2018Stewart et al, , 2019b. Seq2seq structures the learning task similar to neural machine translation (NMT) (Bahdanau et al, 2014) whereby an encoder receives a sequence of noisy text, X = (x 1 , .…”

“…More recently, attention has shifted towards neural techniques that i) contextually normalise tokens based on high-level classifications (Stewart et al, 2019b), ii) modify and fine-tune large pre-trained transformer based representations (Muller et al, 2019), or iii) perform joint normalisation and sanitisation (e.g. masking sensitive tokens) (Nguyen and Cavallari, 2020).…”