Lyra: A Benchmark for Turducken-Style Code Generation

Abstract

Writing a survey paper on one research topic usually needs to cover the salient content from numerous related papers, which can be modeled as a multi-document summarization (MDS) task. Existing MDS datasets usually focus on producing the structureless summary covering a few input documents. Meanwhile, previous structured summary generation works focus on summarizing a single document into a multi-section summary. These existing datasets and methods cannot meet the requirements of summarizing numerous academic papers into a structured summary. To deal with the scarcity of available data, we propose BigSurvey, the first large-scale dataset for generating comprehensive summaries of numerous academic papers on each topic. We collect target summaries from more than seven thousand survey papers and utilize their 430 thousand reference papers’ abstracts as input documents. To organize the diverse content from dozens of input documents and ensure the efficiency of processing long text sequences, we propose a summarization method named category-based alignment and sparse transformer (CAST). The experimental results show that our CAST method outperforms various advanced summarization methods.

A Syntax-Guided Multi-Task Learning Approach for Turducken-Style Code Generation

“…Turducken-style code, which was first proposed by (Liang et al 2021), refers to a style of code where declarative programming is embedded within imperative programming. This type of code can be commonly found in real-world software systems.…”

“…The Lyra dataset (Liang et al 2021) which is for mapping functional descriptions to Turduckenstyle code, only considers Python code with embedded SQL. To improve the diversity of Turducken-style code generation, we used a crowd-sourcing approach to translate the Python code in Lyra into its corresponding Java code and modified the functional description.…”

“…However, large software systems are rarely developed exclusively in a declarative language in practical software development. Instead, declarative programs are commonly embedded within imperative programs, which are normally referred to as Turducken-style programs (Liang et al 2021). For example, in information management systems, developers often include SQL statements within imperative programs; in data mining systems, developers often incorporate regular expressions within imperative programs.…”

“…In data science code generation, (Huang et al 2022) found that about 8% of the errors in the generated code are due to syntax issues. Similarly, (Liang et al 2021) have demonstrated the feasibility of using pre-trained language models to generate Turducken-style code but also admitted the less accurate syntactic conformation. The purpose of the automatic code generation model is to improve the efficiency of software development through automation and intelligent techniques.…”

“…To evaluate the effectiveness of TurduckenGen, we conduct experiments on two Turduckenstyle code datasets Lyra and Pisces, where Lyra (Liang et al 2021) is obtained from GitHub repositories and annotated by human annotators, whereas Pisces is obtained by manually translating the python code in Lyra into the corresponding Java code through a crowdsourcing approach. TurduckenGen is compared to six state-of-the-art baselines, including Transformer (Vaswani et al 2017), CodeBERT (Feng et al 2020), GraphCodeBERT (Guo et al 2021), GPT (Radford et al 2019), CodeGPT (Lu et al 2021a), and CodeT5 (Wang et al 2021b), in terms of six automatic performance metrics (i.e., BLEU (Papineni et al 2002), Weighted BLEU (Ren et al 2020), Crystal BLEU (Eghbali and Pradel 2022), Syntax Match (Ren et al 2020), Syntax Exact Match (Liang et al 2021), and Code Executable (Liang et al 2021)). The comparison results show that TurduckenGen outperforms these baselines.…”