Hierarchical Document Encoder for Parallel Corpus Mining

LAWDR: Language-Agnostic Weighted Document Representations from Pre-trained Models

“…To understand how its representation quality changes with input length, we report its alignment performance of encoding whole documents and encoding only the first 512 tokens respectively. The average of sentence embedding has been shown to be a strong approach to derive document representation (Guo et al, 2019). We include it as another strong baseline in our experiments, with which we can analyze the gains brought by sentence weighting.…”

“…Averaging is a simple yet strong approach to represent compositional semantics. Strong performance comparable to supervised neural models has been achieved by sentence embeddings derived from average word embeddings (Arora et al, 2017) and document embeddings from average sentence embeddings (Guo et al, 2019) respectively.…”

“…Despite a large body of research on monolingual document representations, there have been very few works in cross-lingual setting. The most recent work designs hierarchical multilingual document encoder (HiDE), which is trained on a large corpus of parallel documents (Guo et al, 2019). It only experiments with English-French and English-Spanish language pairs.…”

“…Transformer based models such as BERT, XLM and XLM-RoBERTa have strict length limits on inputs and therefore cannot process long documents. While complex, neural-based models have been proposed, averaging sentence embeddings has remained a hard-to-beat baseline (Guo et al, 2019). Here, we propose a document-level model based on weighted sentence embeddings, and the sentence weights are inversely related with its density in the corpus, akin to an inverse corpus frequency.…”