Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations

Abstract

Recent research has adopted a new experimental field centered around the concept of text perturbations which has revealed that shuffled word order has little to no impact on the downstream performance of Transformer-based language models across many NLP tasks. These findings contradict the common understanding of how the models encode hierarchical and structural information and even question if the word order is modeled with position embeddings. To this end, this paper proposes nine probing datasets organized by the type of controllable text perturbation for three Indo-European languages with a varying degree of word order flexibility: English, Swedish and Russian. Based on the probing analysis of the M-BERT and M-BART models, we report that the syntactic sensitivity depends on the language and model pre-training objectives. We also find that the sensitivity grows across layers together with the increase of the perturbation granularity. Last but not least, we show that the models barely use the positional information to induce syntactic trees from their intermediate self-attention and contextualized representations.

Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models

“…There have been several different approaches to quantifying the linguistic information that is learned by multilingual models. One direction has performed layer-wise analyses to quantify what information is stored at different layers in the model (de Vries et al, 2020;Taktasheva et al, 2021;Papadimitriou et al, 2021). Others have examined the extent to which the different training languages are captured by the model, finding that some languages suffer in the multilingual setting despite overall good performance from the models (Conneau et al, 2020a;.…”

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics

“…This work follows the same experimental direction where text perturbations serve to explore the sensitivity of language models to specific phenomena (Futrell et al, 2019;Ettinger, 2020;Taktasheva et al, 2021;Dankers et al, 2021). It has been shown, for example, that shuffling word order causes significant performance drops on a wide range of QA tasks (Si et al, 2019;Sugawara et al, 2019), but that state-of-the-art NLU models are not sensitive to word order (Pham et al, 2020;.…”

Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023)

“…The existing work devoted to the cross-lingual probing showed that grammatical knowledge of Transformer LMs is adapted to the downstream language; in the case of Russian, the interpretation of results cannot be easily explained (Ravishankar et al, 2019). However, LMs are more insensitive towards granular perturbations when processing texts in languages with free word order, such as Russian (Taktasheva et al, 2021).…”