The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation

Abstract

Does neural machine translation yield translations that are congenial with common sense? In this paper, we present a test suite to evaluate the commonsense reasoning capability of neural machine translation. The test suite consists of three test sets, covering lexical and contextless/contextual syntactic ambiguity that requires commonsense knowledge to resolve. We manually create 1,200 triples, each of which contain a source sentence and two contrastive translations, involving 7 different common sense types. Language models pretrained on large-scale corpora, such as BERT, GPT-2, achieve a commonsense reasoning accuracy of lower than 72% on target translations of this test suite. We conduct extensive experiments on the test suite to evaluate commonsense reasoning in neural machine translation and investigate factors that have impact on this capability. Our experiments and analyses demonstrate that neural machine translation performs poorly on commonsense reasoning of the three ambiguity types in terms of both reasoning accuracy ( 60.1%) and reasoning consistency ( 31%). We will release our test suite as a machine translation commonsense reasoning testbed to promote future work in this direction.

Exploring Human-Like Translation Strategy with Large Language Models

“…Ambiguity resolution has long been one of the most challenging problems in machine translation. To evaluate the ambiguity resolution capability of machine translator, He et al (2020) provide a lexical ambiguity test set for Chinese→English. The hard part of this test set involves Chinese sentences which are difficult to translate correctly unless the translator resolves their ambiguities.…”

CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks

“…Commonsense Reasoning in Real-World Contexts A few recent works have explored the role of commonsense knowledge in real-world settings, such as open-ended response generation (Zhou et al, 2021;Ghosal et al, 2021Ghosal et al, , 2022, machine translation (He et al, 2020) and reading comprehension (Zhang et al, 2018;Huang et al, 2019) and have proposed new commonsense reasoning tasks and benchmarks. We build on top of these benchmarks and extend them to several other real-world NLP tasks, along with a general data collection methodology for commonsense knowledge annotation and Winograd-style schema generation that can be applied to other tasks in the future.…”

“…Machine translation (MT) is known to require commonsense knowledge (Bar-Hillel, 1960) to resolve translation errors. We select the test suite constructed by He et al, 2020 for Chinese-English translation and the Wino-X dataset (Emelin and Sennrich, 2021) for English to German, French, and Russian translation. Both datasets consist of Winograd-style examples containing a source sen-tence and two translations that minimally differ from each other, but only one of which is correct due to underlying commonsense knowledge.…”

“…The necessity of commonsense to solve realworld tasks has been extensively argued since the early stages of AI, notably by Bar-Hillel (1960) in the context of machine translation. However, despite these early arguments, only recently was there an attempt to construct a commonsense reasoning dataset for machine translation (He et al, 2020), an effort which concluded that the commonsense reasoning abilities of modern models were still in their infancy when applied in real NLP tasks.…”

Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution

“…Finally, while cross-lingual transfer in MLLMs has received much attention in the past (Conneau et al, 2018(Conneau et al, , 2020Hu et al, 2020;, research on CSR in multiple languages remains limited, with (He et al, 2020) being the only relevant machine translation study known to us. Concurrent to our work, (Lin et al, 2021) examine whether MLLMs can perform multilingual CSR on tasks unrelated to Winograd schemas.…”