Explaining and Improving BERT Performance on Lexical Semantic Change Detection

Abstract

Type-and token-based embedding architectures are still competing in lexical semantic change detection. The recent success of typebased models in SemEval-2020 Task 1 has raised the question why the success of tokenbased models on a variety of other NLP tasks does not translate to our field. We investigate the influence of a range of variables on clusterings of BERT vectors and show that its low performance is largely due to orthographic information on the target word, which is encoded even in the higher layers of BERT representations. By reducing the influence of orthography we considerably improve BERT's performance.

Text analysis and deep learning: A network approach

“…Due to their contextual resolution, transformer models such as BERT are able to attend to more features of the text, including orthographic information (Laicher et al 2021) and, as we show in this paper, syntactic information. Furthermore, such models also encode positional information in the focal word embeddings and in those of the contextual sequence.…”

“…Since our training and inference task are identical, we do not risk a gap between training and analysis (Rogers, Kovaleva, and Rumshisky 2020) and the accuracy of our approach directly depends on the model's native training performance. On the other hand, any technique to increase performance of lexical substitutions (e.g., Amrami and Goldberg 2019;Laicher et al 2021;Schick and Schütze 2019) apply to our model as well.…”

“…The condition is sufficient for syntax and semantics jointly. Research indicates (Laicher et al 2021) that for contextual embeddings, orthographic similarity is a further condition. We do not observe such relations while employing our network approach.…”