Color the Word: Leveraging Web Images for Machine Translation of Untranslatable Words

Abstract

Automatic translation allows people around the globe to communicate with one another. However, state-of-the art machine translation is still unable to capture fine-grained meaning. This paper introduces the idea of using Web image selections in text-to-text translation, specifically for lacunae, which are words that do not have a translation in another language. We asked human professional translators to rank Google translate translations of lacunae in German and Dutch. We then compared that ranking with a ranking based on color histograms of Web image data of the words. We found there is viable potential in the idea of using images to address lacunae in the field of machine translation. We publicly release a dataset and our code for others to explore this potential. Finally, we provide an outlook on research directions that would allow this idea to be used in practice.