Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models

Abstract

Chinese pre-trained language models usually process text as a sequence of characters, while ignoring more coarse granularity, e.g., words. In this work, we propose a novel pre-training paradigm for Chinese -Lattice-BERT, which explicitly incorporates word representations along with characters, thus can model a sentence in a multi-granularity manner. Specifically, we construct a lattice graph from the characters and words in a sentence and feed all these text units into transformers. We design a lattice position attention mechanism to exploit the lattice structures in self-attention layers. We further propose a masked segment prediction task to push the model to learn from rich but redundant information inherent in lattices, while avoiding learning unexpected tricks. Experiments on 11 Chinese natural language understanding tasks show that our model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. Further analysis shows that Lattice-BERT can harness the lattice structures, and the improvement comes from the exploration of redundant information and multigranularity representations. 1

MarkBERT: Marking Word Boundaries Improves Chinese BERT

“…Existing Chinese BERT models that incorporate word information can be divided into two categories. The first category uses word information in the pretraining stage but represents a text as a sequence of characters when the pretrained model is applied to downstream tasks (Cui et al, 2019a;Lai et al, 2021). The second category uses word information when the pretrained model is used in downstream tasks (Su, 2020;Guo et al, 2021).…”

“…• Lattice BERT: Lai et al (2021) uses lexicons to enhance the character-level encodings (left side of the encoder in Figure . 3 (b) ). It uses the parallel structure in the transformers to discriminate characters and additional lexicons.…”

“…-Lattice-BERT (Lai et al, 2021): the state-ofthe-art multi-granularity model that uses lexicons as word-level knowledge concatenated to the original input context.…”

“…There are several attempts at building Chinese BERT models where word information is considered. Existing studies tokenize a word as a basic unit (Su, 2020), as multiple characters (Cui et al, 2019a) or a combination of both Lai et al, 2021;Guo et al, 2021). However, due to the limit of the vocabulary size of BERT, these models only learn for a limited number (e.g., 40K) of words with high frequency.…”

"Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction

“…Language features are considered in more recent works. For example, AMBERT (Zhang and Li, 2020) and Lattice-BERT (Lai et al, 2021) both take word information into consideration. Chinese-BERT (Sun et al, 2021) utilizes pinyin and glyph of characters.…”