Assessing the quality of GitHub copilot’s code generation

Yetistiren, Burak; Ozsoy, Isik; Tüzün, Eray

doi:10.1145/3558489.3559072

Cited by 60 publications

(34 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The research community explored GLLMs for coding tasks across various languages like Java [251], [252], [255], [260], [263], [264], [266], [267], [269], [270], Python [253], [254], [256]- [258], [260], [262], [263], [265], [267], [268], [271], PHP [260], GO [260], Ruby [260], JavaScript [260], C [261], [268], C++ [259], [268], Julia [268], and MATLAB [268]. Most of the research works focused on Python and Java languages, while a few research works focused on other languages like GO, PHP, GO, Ruby, JavaScript, C, C++, Julia and MATLAB.…”

Section: Research Work Exploring Gllms For Various Coding Tasksmentioning

confidence: 99%

“…Table 9 presents a summary of research works exploring GLLMs for various coding tasks. Some of the research works [253], [257], [259], [268], [269] explored GLLMs for code generation task. Yeticstiren et al [253] compared various AI-assisted code generation tools like ChatGPT, Amazon's Code Whisperer and Github's Copilot on the Human Eval [103] dataset.…”

Section: Research Work Exploring Gllms For Various Coding Tasksmentioning

confidence: 99%

“…Some of the research works [253], [257], [259], [268], [269] explored GLLMs for code generation task. Yeticstiren et al [253] compared various AI-assisted code generation tools like ChatGPT, Amazon's Code Whisperer and Github's Copilot on the Human Eval [103] dataset. ChatGPT outperforms other tools by generating correct code 65.2% of the time, while the other tools generate correct code for a maximum of 46.3% of the time only.…”

Section: Research Work Exploring Gllms For Various Coding Tasksmentioning

confidence: 99%

See 2 more Smart Citations

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

Kalyan

2023

SSRN Journal

View full text Add to dashboard Cite

Section: Research Work Exploring Gllms For Various Coding Tasksmentioning

confidence: 99%

Section: Research Work Exploring Gllms For Various Coding Tasksmentioning

confidence: 99%

Section: Research Work Exploring Gllms For Various Coding Tasksmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

Kalyan

2023

SSRN Journal

View full text Add to dashboard Cite

“…Imai [11] compared the effectiveness of programming with Copilot versus human programming, and found that the generated code by Copilot is inferior than human-written code. Yetistiren et al [8] assessed the quality of generated code by Copilot in terms of validity, correctness, and efficiency. Their empirical analysis shows Copilot is a promising tool.…”

Section: Analyzing the Code Generated Using Copilotmentioning

confidence: 99%

“…Although the emergence of AI-assisted programming tools has empowered practitioners in their software development efforts, there is little evidence and lack of empirically-rooted studies (e.g, [3], [5], [6]) on the role of AI-assisted programming tools in software development. The existing studies such as [7] and [8] primarily focus on the correctness and understanding of the code suggested by Copilot, and little is known about the practices, challenges, and expected features of using Copilot during programming and software development activities for the developers and users of Copilot. To ameliorate this gap, we conducted this study that collects data from Stack Overflow (SO) and GitHub Discussions to get practitioners' perspectives on using Copilot during software development.…”

Section: Introductionmentioning

confidence: 99%

Demystifying Practices, Challenges and Expected Features of Using GitHub Copilot

Zhang,

Liang,

Zhou

et al. 2023

Int. J. Soft. Eng. Knowl. Eng.

View full text Add to dashboard Cite

With the advances in machine learning, there is a growing interest in AI-enabled tools for autocompleting source code. GitHub Copilot, also referred to as the “AI Pair Programmer”, has been trained on billions of lines of open source GitHub code, and is one of such tools that has been increasingly used since its launch in June 2021. However, little effort has been devoted to understanding the practices, challenges, and expected features of using Copilot in programming for auto-completed source code from the point of view of practitioners. To this end, we conducted an empirical study by collecting and analyzing the data from Stack Overflow (SO) and GitHub Discussions. More specifically, we searched and manually collected 303 SO posts and 927 GitHub discussions related to the usage of Copilot. We identified the programming languages, Integrated Development Environments (IDEs), technologies used with Copilot, functions implemented, benefits, limitations, and challenges when using Copilot. The results show that when practitioners use Copilot: (1) The major programming languages used with Copilot are JavaScript and Python, (2) the main IDE used with Copilot is Visual Studio Code, (3) the most common used technology with Copilot is Node.js, (4) the leading function implemented by Copilot is data processing, (5) the main purpose of users using Copilot is to help generate code, (6) the significant benefit of using Copilot is useful code generation, (7) the main limitation encountered by practitioners when using Copilot is difficulty of integration, and (8) the most common expected feature is that Copilot can be integrated with more IDEs. Our results suggest that using Copilot is like a double-edged sword, which requires developers to carefully consider various aspects when deciding whether or not to use it. Our study provides empirically grounded foundations that could inform software developers and practitioners, as well as provide a basis for future investigations on the role of Copilot as an AI pair programmer in software development.

show abstract

Application of Artificial Intelligence in Mental Healthcare: Generative Pre-trained Transformer 3 (GPT-3) and Cognitive Distortions

Nazarova

2023

Lecture Notes in Networks and Systems

View full text Add to dashboard Cite

Assessing the quality of GitHub copilot’s code generation

Cited by 60 publications

References 13 publications

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

Demystifying Practices, Challenges and Expected Features of Using GitHub Copilot

Application of Artificial Intelligence in Mental Healthcare: Generative Pre-trained Transformer 3 (GPT-3) and Cognitive Distortions

Contact Info

Product

Resources

About