Exploring Supervised and Unsupervised Rewards in Machine Translation

Abstract

Reinforcement Learning (RL) is a powerful framework to address the discrepancy between loss functions used during training and the final evaluation metrics to be used at test time. When applied to neural Machine Translation (MT), it minimises the mismatch between the cross-entropy loss and non-differentiable evaluation metrics like BLEU. However, the suitability of these metrics as reward function at training time is questionable: they tend to be sparse and biased towards the specific words used in the reference texts. We propose to address this problem by making models less reliant on such metrics in two ways: (a) with an entropy-regularised RL method that does not only maximise a reward function but also explore the action space to avoid peaky distributions; (b) with a novel RL method that explores a dynamic unsupervised reward function to balance between exploration and exploitation. We base our proposals on the Soft Actor-Critic (SAC) framework, adapting the off-policy maximum entropy model for language generation applications such as MT. We demonstrate that SAC with BLEU reward tends to overfit less to the training data and performs better on out-of-domain data. We also show that our dynamic unsupervised reward can lead to better translation of ambiguous words.

Extracting and classifying exceptional COVID‐19 measures from multilingual legal texts: The merits and limitations of automated approaches

“…Finally, we adopt a multilingual approach to automatically analyze the sources of COVID‐19 legislation, limiting the bias associated with the translation of the original texts into English. This decision is based on previous work that has shown limitations in the use of human‐language technologies when it comes to domain‐specific texts (Duboue et al, 2016; Ive et al, 2020; Vieira et al, 2020). To select which of the NLP techniques is best suited to support human coding, we proceed in two steps.…”

SURF: Semantic-level Unsupervised Reward Function for Machine Translation

“…More advanced AC models with Q-Learning are rarely applied to language generation problems. However, there are exceptions (e.g., entropy-regularised AC models that promote exploration of actions (Dai et al, 2018;Ive et al, 2021)). This could be explained by the difficulty of approximating the Q-function for large action space.…”

“…Unsupervised Rewards for Language Generation Tasks Recent work on unsupervised rewards in NLP has explored both dynamic (Ive et al, 2021) and static rewards (Gao et al, 2020;Garg et al, 2021). For example, Ive et al (2021) introduces a dynamic distribution over latent frequency classes as a reward signal. This distribution is shaped to promote more rare words in the policy search space.…”