Leveraging Automated Unit Tests for Unsupervised Code Translation

Abstract

With little to no parallel data available for programming languages, unsupervised methods are well-suited to source code translation. However, the majority of unsupervised machine translation approaches rely on back-translation, a method developed in the context of natural language translation and one that inherently involves training on noisy inputs. Unfortunately, source code is highly sensitive to small changes; a single token can result in compilation failures or erroneous programs, unlike natural languages where small inaccuracies may not change the meaning of a sentence. To address this issue, we propose to leverage an automated unit-testing system to filter out invalid translations, thereby creating a fully tested parallel corpus. We found that fine-tuning an unsupervised model with this filtered data set significantly reduces the noise in the translations so-generated, comfortably outperforming the state-of-the-art for all language pairs studied. In particular, for Java → Python and Python → C++ we outperform the best previous methods by more than 16% and 24% respectively, reducing the error rate by more than 35%.

Human-Machine Cooperative Program Translation Based on Abstract Syntax Tree

“…There are many intolerable errors in the result of machine translation, which leads to compilation failure. Roziere et al [34] proposed using automated unit tests to filter out faulty generated results. Therefore, a parallel corpus can be obtained for fine-tuning the unsupervised training translation model.…”

Mutation analysis for evaluating code translation

“…Their approach achieves outstanding effectiveness. Later on, they presented DOBF (Rozière et al 2021a) and TransCoder-ST (Rozière et al 2021b), the former pretrains a model to revert the code obfuscation function by training a sequence-tosequence model; the latter uses automatic test generation techniques to automatically select high-quality translation pairs to fine-tune the pre-trained model. These works use Computational Accuracy (CA), a measure to evaluate the translated code, which is based on the ratio of test cases that have similar outputs between the input program and its translation.…”

A Stealthy Backdoor Attack for Code Models

“…For example, the neural code translation task employs machine translation technology to automatically translate Java language into C# language, reducing the neural code translation work for programmers [1][2] [3]. Some of the previous approaches concentrate on improving performance by refining code representations, such as the pre-trained model CodeBERT [6].…”

“…In recent years,we have witnessed a dramatic rise in applying deep source code processing models (a.k.a, code models) to source code processing tasks [1][2] [3][4] [5].…”

“…To evaluate the effectiveness of ASTT, we use the deep learning-based models, which have been demonstrated to have state-of-the-art performance [1][2] [3][4] [7][8] [9].…”