Probing Multilingual Sentence Representations With X-Probe

Abstract

This paper extends the task of probing sentence representations for linguistic insight in a multilingual domain. In doing so, we make two contributions: first, we provide datasets for multilingual probing, derived from Wikipedia, in five languages, viz. English, French, German, Spanish and Russian. Second, we evaluate six sentence encoders for each language, each trained by mapping sentence representations to English sentence representations, using sentences in a parallel corpus. We discover that cross-lingually mapped representations are often better at retaining certain linguistic information than representations derived from English encoders trained on natural language inference (NLI) as a downstream task.

Discrete Cosine Transform as Universal Sentence Encoder

“…2 The benchmark covers five languages: English, French, German, Spanish and Russian, derived from Wikipedia. The task set comprises 9 probing tasks, summarized in Table 1, that address varieties of linguistic properties including surface, syntactic, and semantic information Ravishankar et al, 2019). Ravishankar et al (2019) used the datasets to evaluate different sentence encoders trained by mapping sentence representations to English.…”

“…The task set comprises 9 probing tasks, summarized in Table 1, that address varieties of linguistic properties including surface, syntactic, and semantic information Ravishankar et al, 2019). Ravishankar et al (2019) used the datasets to evaluate different sentence encoders trained by mapping sentence representations to English. Unlike Ravishankar et al ( 2019), we use the datasets to evaluate DCT embeddings for each language independently.…”

“…(Shao and Johnson, 2008). Finally, a fixed-length sentence vector of size Kd is generated by concatenating the first Ravishankar et al, 2019).…”

“…Unlike(Almarwani et al, 2019), we note no further improvements with larger coefficients, thus, we only report the results of 1 ≤ K ≤ 4.2 Refer to(Ravishankar et al, 2019) for more details about the probing tasks.…”

How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation

“…The majority of approaches for probing sentence embeddings target English, but recently some works have also addressed other languages such as Polish, Russian, or Spanish in a multiand cross-lingual setup (Krasnowska-Kieraś and Wróblewska, 2019;Ravishankar et al, 2019). Motivations for considering a multi-lingual analysis include knowing whether findings from English transfer to other languages and determining a universal set of probing tasks that suits multiple languages, e.g., with richer morphology and freer word order.…”