Large Language Models are Diverse Role-Players for Summarization Evaluation

Wu, Ning; Gong, Ming; Shou, Linjun; Liang, Shining; Jiang, Daxin

doi:10.1007/978-3-031-44693-1_54

Cited by 6 publications

(1 citation statement)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Role Assignment (Peng et al, 2023;AlKhamissi et al, 2023;Wu et al, 2023;Akın, 2023) We configured two distinct roles for the model, each with unique skills. This arrangement empowers the model to simulate more domain-adaptable and specialized expert roles.…”

Section: A6 Prompt Variationsmentioning

confidence: 99%

CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation

Yan,

Tian,

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Recent code translation techniques exploit neural machine translation models to translate source code from one programming language to another to satisfy production compatibility or to improve efficiency of codebase maintenance. Most existing code translation datasets only focus on a single pair of popular programming languages. To advance research on code translation and meet diverse requirements of real-world applications, we construct CodeTransOcean, a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation. CodeTransOcean consists of three novel multilingual datasets, namely, MultilingualTrans supporting translations between multiple popular programming languages, NicheTrans for translating between niche programming languages and popular ones, and LLMTrans for evaluating executability of translated code by large language models (LLMs). CodeTransOcean also includes a novel cross-framework dataset, DLTrans, for translating deep learning code across different frameworks. We develop multilingual modeling approaches for code translation and demonstrate their great potential in improving the translation quality of both low-resource and high-resource language pairs and boosting the training efficiency. We also propose a novel evaluation metric Debugging Success Rate@K for program-level code translation. Last but not least, we evaluate LLM ChatGPT on our datasets and investigate its potential for fuzzy execution predictions. We build baselines for CodeTransOcean and analyze challenges of code translation for guiding future research. The CodeTransOcean datasets and code are publicly available at https://github. com/WeixiangYAN/CodeTransOcean.

show abstract

Section: A6 Prompt Variationsmentioning

confidence: 99%