No Longer Lost in Translation: Evidence that Google Translate Works for Comparative Bag-of-Words Text Applications

Abstract

Automated text analysis allows researchers to analyze large quantities of text. Yet, comparative researchers are presented with a big challenge: across countries people speak different languages. To address this issue, some analysts have suggested using Google Translate to convert all texts into English before starting the analysis (Lucas et al., 2015). But in doing so, do we get lost in translation? This paper evaluates the usefulness of machine translation for bag-of-words models – such as topic models. We use the europarl dataset and compare term-document matrices as well as topic model results from gold standard translated text and machine-translated text. We evaluate results at both the document and the corpus level. We first find term-document matrices for both text corpora to be highly similar, with significant but minor differences across languages. What is more, we find considerable overlap in the set of features generated from human-translated and machine-translated texts. With regards to LDA topic models, we find topical prevalence and topical content to be highly similar with only small differences across languages. We conclude that Google Translate is a useful tool for comparative researchers when using bag-of-words text models.

Undermining, defusing or defending European integration? Assessing public communication of European executives in times of EU politicisation

“…The speeches were originally sourced from active or archived public websites and are partially auto‐translated to English (De Vries et al. ; Schumacher et al. ).…”

Errors have been made, others will be blamed: Issue engagement and blame shifting in prime minister speeches during the economic crisis in Europe

“…To compare speeches in different languages, we used Google Translate to translate all non-English texts to English, as this was the language of a large majority of speeches. De Vries et al (2018) demonstrate that Google Translate can be used for our purposes. In particular, they compared the output of topic models -an automated text analysis technique -of a text corpus of European Parliament proceedings translated to English by professional translators to a text corpus of the same proceedings translated to English by Google Translate.…”

Selective Attention and the Information Environment: Citizens’ Perceptions of Political Problems in the 2015 Swiss Federal Election Campaign

“…However, because the French, Italian and German answers were translated into a fourth language, the loss of information affects all the answers similarly, which likely increases the answers’ across‐language comparability (de Vries et al. ).…”