On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot

Mastropaolo, Antonio; Pascarella, Luca; Guglielmi, Emanuela; Ciniselli, Matteo; Scalabrino, Simone; Oliveto, Rocco; Bavota, Gabriele

doi:10.1109/icse48619.2023.00181

Cited by 34 publications

(2 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In-context learning is a novel paradigm that conditions the model on task descriptions and demonstrations to generate answers for the same tasks [25]. It has been applied to various domains, including testing [55], code generation [56], and GUI automation [57]. These works use coarse-grained, direct-inquiry style prompt design.…”

Section: Related Workmentioning

confidence: 99%

“…Instead of relying on gradient updates [33]- [35], ICL utilizes zero-or few-shot prompts for task adaptation. This paradigm has been successfully applied in range of software engineering tasks, such as testing [47], code generation [48], and GUI automation [49]. RING [40] used an ICL-based method to fix last-mile syntax errors, avoiding the need to train or fine-tune a LLM.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Prompt-tuned Code Language Model as a Neural Knowledge Base for Type Inference in Statically-Typed Partial Code

Huang

Yuan

Xing

et al. 2022

Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

View full text Add to dashboard Cite

Control Flow Graphs (CFGs) are essential for visualizing, understanding and analyzing program behavior. For statically-typed programming language like Java, developers obtain CFGs by using bytecode-based methods for compilable code and Abstract Syntax Tree (AST)-based methods for partially uncompilable code. However, explicit syntax errors during AST construction and implicit semantic errors caused by bad coding practices can lead to behavioral loss and deviation of CFGs. To address the issue, we propose a novel approach that leverages the error-tolerant and understanding ability of pre-trained Large Language Models (LLMs) to generate CFGs. Our approach involves a Chain of Thought (CoT) with four steps: structure hierarchy extraction, nested code block extraction, CFG generation of nested code blocks, and fusion of all nested code blocks' CFGs. To address the limitations of the original CoT's single-prompt approach (i.e., completing all steps in a single generative pass), which can result in an "epic" prompt with hard-to-control behavior and error accumulation, we break down the CoT into an AI chain with explicit sub-steps. Each sub-step corresponds to a separate AI-unit, with an effective prompt assigned to each unit for interacting with LLMs to accomplish a specific purpose. Our experiments confirmed that our method outperforms existing CFG tools in terms of node and edge coverage, especially for incomplete or erroneous code. We also conducted an ablation experiment and confirmed the effectiveness of AI chain design principles: Hierarchical Task Breakdown, Unit Composition, and Mix of AI Units and Non-AI Units. Our work opens up new possibilities for building foundational software engineering tools based on LLMs, as opposed to traditional program analysis methods.

show abstract