DagoBERT: Generating Derivational Morphology with a Pretrained Language Model

Abstract

Can pretrained language models (PLMs) generate derivationally complex words? We present the first study investigating this question, taking BERT as the example PLM. We examine BERT's derivational capabilities in different settings, ranging from using the unmodified pretrained model to full finetuning. Our best model, DagoBERT (Derivationally and generatively optimized BERT), clearly outperforms the previous state of the art in derivation generation (DG). Furthermore, our experiments show that the input segmentation crucially impacts BERT's derivational knowledge, suggesting that the performance of PLMs could be further improved if a morphologically informed vocabulary of units were used.

Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words

“…It is divided into smaller communities, so-called subreddits, which have been shown to be a rich source of derivationally complex words (Hofmann et al, 2020c). Hofmann et al (2020a) have published a dataset of derivatives found on Reddit annotated with the subreddits in which they occur. 8 Inspired by a content-based subreddit categorization scheme, 9 we define two groups of subreddits, an entertainment set (ent) consisting of the subreddits anime, DestinyTheGame, funny, Games, gaming, leagueoflegends, movies, Music, pics, and videos, as well as a discussion set (dis) consisting of the subred-8 https://github.com/valentinhofmann/ dagobert 9 https://www.reddit.com/r/ TheoryOfReddit/comments/1f7hqc/the_200_ most_active_subreddits_categorized_by dits askscience, atheism, conspiracy, news, Libertarian, politics, science, technology, TwoXChromosomes, and worldnews, and extract all derivationally complex words occurring in them.…”

“…The specific BERT variant we use is BERT BASE (uncased) (Devlin et al, 2019). For the derivational segmentation, we follow previous work by Hofmann et al (2020a) in separating stem and prefixes by a hyphen. We further follow Casanueva et al (2020) and in mean-pooling the output representations for all subwords, excluding BERT's special tokens.…”

“…, Deutsch et al (2018) propose sequence-to-sequence models for the generation of derivationally complex words. Hofmann et al (2020a) address the same task using BERT. In contrast, we analyze how different input segmentations affect the semantic representations of derivationally complex words in PLMs, a question that has not been addressed before.…”

“…One common characteristic of PLMs is their input segmentation: PLMs are based on fixed-size vocabularies of words and subwords that are generated by compression algorithms such as bytepair encoding (Gage, 1994;Sennrich et al, 2016) and WordPiece (Schuster and Nakajima, 2012;Wu et al, 2016). The segmentations produced by these algorithms are linguistically questionable at times (Church, 2020), which has been shown to worsen performance on certain downstream tasks (Bostrom and Durrett, 2020;Hofmann et al, 2020a). However, the wider implications of these findings, particularly with regard to the generalization capabilities of PLMs, are still poorly understood.…”

This is a BERT. Now there are several of them. Can they generalize to novel words?

“…Bostrom and Durrett (2020) argue that byte-pair encoding less faithfully expresses English morphology than Unigram segmentation, and show a performance improvement in downstream tasks with a unigramsegmentation-based BERT model. Hofmann et al (2020) show that BERT can be fine-tuned with a classification layer to complete a derivational morphology cloze task, finding that imposing morpheme boundaries with hyphenation on the input side ultimately improved BERT's performance at this task. Finally, Edmiston (2020) investigates several monolingual BERT models for representations of morphological information.…”