Automatic testing and improvement of machine translation

Sun, Zeyu; Zhang, Jie M.; Harman, Mark; Papadakis, Mike; Zhang, Lu

doi:10.1145/3377811.3380420

Cited by 96 publications

(69 citation statements)

References 38 publications

(53 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Purity successfully reports 154 erroneous translation pairs in Google Translate and 177 er roneous translation pairs in Bing Microsoft Translator with high precision (79.3% and 78.3%), revealing 123 and 142 erroneous translations respectively.1 The translation errors found are diverse, including under-translation, over-translation, word/phrase mistranslation, incorrect modification, and un clear logic. Compared with the state-of-the-art [28], [29], Purity can report more erroneous translations with higher precision. Due to its conceptual difference, Purity can reveal many erroneous translations that have not been found by existing approaches (illustrated in Fig.…”

Section: Introductionmentioning

confidence: 92%

“…b) Comparison: We compare Purity with two state-ofthe-art approaches: SIT [28] and TransRepair (ED) [29]. We obtained the source code of SIT from the authors.…”

Section: Ev a L U A T Io Nmentioning

confidence: 99%

“…In our experiment, we use 0.8 as the threshold because we were unable to reproduce the quantity of word pairs that the paper reported using 0.9. In this paper, we evaluate TransRepair-ED because it achieves the highest precision among the four metrics on Google Translate and better overall performance than TransRepair-LCS for Transformers (Table 2 of [29]). In addi tion, we re-tune the parameters of SIT and TransRepair using the strategies introduced in their papers.…”

Section: Ev a L U A T Io Nmentioning

confidence: 99%

“…In contrast, testing the correctness of translations is a more complex task: source text could have multiple correct translations and the output space is magnitudes larger. Last but not least, existing machine translation testing techniques [28], [29] generate test cases (i.e., synthesized sentences) by replacing one word in a sentence via language models. Thus, their performance is limited by the proficiency of existing language models.…”

Section: Introductionmentioning

confidence: 99%

“…If a large difference exists between the translations of the same RTI, we report this pair of texts and their translations as a suspicious issue. The key idea of this paper is conceptually different from existing approaches [28], [29], which replace a word (i.e., the context is fixed) and assume that the translation should have only small changes. In contrast, this paper assumes that the translation of an RTI should be similar across different sentences/phrases (i.e., the context is varied).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Testing Machine Translation via Referential Transparency

Meister

2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

M achine translatio n software has seen ra p id progress in recent years due to the advancement o f deep neural networks. People rou tine ly use machine translation software in th e ir da ily lives fo r tasks such as ord erin g food in a foreign restaurant, receiving medical diagnosis and treatm ent fro m foreign doctors, and reading inte rna tiona l po litica l news online. However, due to the com plexity and in tra c ta b ility of the underlying neural networks, m odern machine translation software is s till fa r fro m robust and can produce poor o r incorrect translations; this can lead to m isunderstanding, financial loss, threats to personal safety and health, and po litica l conflicts. To address this problem , we introduce referentially transparent inputs (RTIs), a simple, w idely applicable methodology fo r validating machine translation software. A referentially transparent in pu t is a piece o f text th a t should have s im ila r translations when used in d iffe re nt contexts. O u r practical im plem entation, P urity, detects when this p ro pe rty is broken by a translation. To evaluate R T I, we use Purity to test Google Translate and B ing M icrosoft T ranslator w ith 200 unlabeled sentences, w hich detected 123 and 142 erroneous translations w ith high precision (79.3% and 78.3%). The translation errors are diverse, in cluding examples of under-translation, over-translation, w ord/phrase m istranslation, incorrect m odification, and unclear logic.

show abstract

Section: Introductionmentioning

confidence: 92%

“…b) Comparison: We compare Purity with two state-ofthe-art approaches: SIT [28] and TransRepair (ED) [29]. We obtained the source code of SIT from the authors.…”

Section: Ev a L U A T Io Nmentioning

confidence: 99%

Section: Ev a L U A T Io Nmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Testing Machine Translation via Referential Transparency

Meister

2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

show abstract

CodeBERT‐Attack: Adversarial attack against source code deep learning models via pre‐trained model

Zhang

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

Over the past few years, the software engineering (SE) community has widely employed deep learning (DL) techniques in many source code processing tasks. Similar to other domains like computer vision and natural language processing (NLP), the state‐of‐the‐art DL techniques for source code processing can still suffer from adversarial vulnerability, where minor code perturbations can mislead a DL model's inference. Efficiently detecting such vulnerability to expose the risks at an early stage is an essential step and of great importance for further enhancement. This paper proposes a novel black‐box effective and high‐quality adversarial attack method, namely CodeBERT‐Attack (CBA), based on the powerful large pre‐trained model (i.e., CodeBERT) for DL models of source code processing. CBA locates the vulnerable positions through masking and leverages the power of CodeBERT to generate textual preserving perturbations. We turn CodeBERT against DL models and further fine‐tuned CodeBERT models for specific downstream tasks, and successfully mislead these victim models to erroneous outputs. In addition, taking the power of CodeBERT, CBA is capable of effectively generating adversarial examples that are less perceptible to programmers. Our in‐depth evaluation on two typical source code classification tasks (i.e., functionality classification and code clone detection) against the most widely adopted LSTM and the powerful fine‐tuned CodeBERT models demonstrate the advantages of our proposed technique in terms of both effectiveness and efficiency. Furthermore, our results also show (1) that pre‐training may help CodeBERT gain resilience against perturbations further, and (2) certain pre‐training tasks may be beneficial for adversarial robustness.

show abstract

MetaLiDAR: Automated metamorphic testing of LiDAR‐based autonomous driving systems

Yang,

Huang,

Zheng

et al. 2023

J Software Evolu Process

View full text Add to dashboard Cite

Recent advances in artificial intelligence technology and perception components have promoted the rapid development of autonomous vehicles. However, as safety‐critical software, autonomous driving systems often make wrong judgments, seriously threatening human and property safety. LiDAR is one of the most critical sensors in autonomous vehicles, capable of accurately perceiving the three‐dimensional information of the environment. Nevertheless, the high cost of manually collecting and labeling point cloud data leads to a dearth of testing methods for LiDAR‐based perception modules. To bridge the critical gap, we introduce MetaLiDAR, a novel automated metamorphic testing methodology for LiDAR‐based autonomous driving systems. First, we propose three object‐level metamorphic relations for the domain characteristics of autonomous driving systems. Next, we design three transformation modules so that MetaLiDAR can generate natural‐looking follow‐up point clouds. Finally, we define corresponding evaluation metrics based on metamorphic relations. MetaLiDAR automatically determines whether source and follow‐up test cases meet the metamorphic relations based on the evaluation metrics. Our empirical research on five state‐of‐the‐art LiDAR‐based object detection models shows that MetaLiDAR can not only generate natural‐looking test point clouds to detect 181,547 inconsistent behaviors of different models but also significantly enhance the robustness of models by retraining with synthetic point clouds.

show abstract

Automatic testing and improvement of machine translation

Cited by 96 publications

References 38 publications

Testing Machine Translation via Referential Transparency

Testing Machine Translation via Referential Transparency

CodeBERT‐Attack: Adversarial attack against source code deep learning models via pre‐trained model

MetaLiDAR: Automated metamorphic testing of LiDAR‐based autonomous driving systems

Contact Info

Product

Resources

About