RobBERT: a Dutch RoBERTa-based Language Model

Abstract

Pre-trained language models have been dominating the field of natural language processing in recent years, and have led to significant performance gains for various complex natural language tasks. One of the most prominent pre-trained language models is BERT (Bidirectional Encoders for Transformers), which was released as an English as well as a multilingual version. Although multilingual BERT performs well on many tasks, recent studies showed that BERT models trained on a single language significantly outperform the multilingual results. Training a Dutch BERT model thus has a lot of potential for a wide range of Dutch NLP tasks. While previous approaches have used earlier implementations of BERT to train their Dutch BERT, we used RoBERTa, a robustly optimized BERT approach, to train a Dutch language model called RobBERT. We show that RobBERT improves state of the art results in Dutch-specific language tasks, and also outperforms other existing Dutch BERTbased models in sentiment analysis. These results indicate that RobBERT is a powerful pretrained model for fine-tuning for a large variety of Dutch language tasks. We publicly release this pre-trained model in hope of supporting further downstream Dutch NLP applications.

CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

“…While shared subword vocabularies proved to be a practical compromise that allows handling multiple languages within the same network, they are suboptimal when targeting a specific language; recent work reports gains from customized singlelanguage vocabularies (Delobelle et al, 2020).…”

SICKNL: A Dataset for Dutch Natural Language Inference

“…In the second setup, we use BERTje, the Dutch BERT model of de Vries et al (2019) and RobBERT, the Dutch RoBERTa model of Delobelle et al (2020), with their corresponding English counterparts, as well as multilingual BERT (mBERT), as sequence classifiers on the Entailment task of SICK(-NL). Here we observe a similar pattern in the results in Table 3: while there are individual difference on the same task, the main surprise is that the Dutch dataset is harder, even when exactly the same model (mBERT) is used.…”

“…Moreover, the syntactically parsed LASSY corpus of written Dutch (van Noord et al, 2013), and the SONAR corpus of written Dutch (Oostdijk et al, 2013) provide rich resources on which NLP systems may be developed. Indeed, Dutch is in the scope of the multilingual BERT models published by Google (Devlin et al, 2019), and two monolingual Dutch BERT models have been published as part of Hugging-Face's transformers library (de Vries et al, 2019;Delobelle et al, 2020).…”

GottBERT: a pure German Language Model

“…Initially, most of the research took place in English followed by multilingual approaches (Conneau et al, 2019). Although, multilingual approaches were trained on large texts of many languages, they were outperformed by single language models (de Vries et al, 2019;Martin et al, 2020;Le et al, 2020;Delobelle et al, 2020). Single language models trained with the Open Super-large Crawled ALMAnaCH coRpus (OSCAR) showed good performance due to the size and variance of the OS-CAR (Martin et al, 2020;Delobelle et al, 2020).…”