Defexts: A Curated Dataset of Reproducible Real-World Bugs for Modern JVM Languages

Benton, Samuel; Ghanbari, Ali; Zhang, Lingming

doi:10.1109/icse-companion.2019.00035

Cited by 18 publications

(5 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Schuster and Paliwal (1997) discovers in-the-wild attacks on code autocompletion tools. Compared to language models, code models may be more vulnerable to adversarial attacks due to synthetic labels (Ferenc et al, 2018;Pradel and Sen, 2018;Benton et al, 2019) and duplication (Allamanis, 2019) that degrade generalization.…”

Section: Code Representation Learningmentioning

confidence: 99%

Contrastive Code Representation Learning

Jain¹,

Jain²,

Zhang³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Recent work learns contextual representations of source code by reconstructing tokens from their context. For downstream semantic understanding tasks like code clone detection, these representations should ideally capture program functionality. However, we show that the popular reconstruction-based RoBERTa model is sensitive to source code edits, even when the edits preserve semantics. We propose Con-traCode: a contrastive pre-training task that learns code functionality, not form. Con-traCode pre-trains a neural network to identify functionally similar variants of a program among many non-equivalent distractors. We scalably generate these variants using an automated source-to-source compiler as a form of data augmentation. Contrastive pretraining outperforms RoBERTa on an adversarial code clone detection benchmark by 39% AUROC. Surprisingly, improved adversarial robustness translates to better accuracy over natural code; ContraCode improves summarization and TypeScript type inference accuracy by 2 to 13 percentage points over competitive baselines. All source is available at https://github.com/parasj/contracode.

show abstract

Section: Code Representation Learningmentioning

confidence: 99%

Contrastive Code Representation Learning

Jain¹,

Jain²,

Zhang³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Like Defects4J, both of them consist of bugs in real-world projects and provide testing infrastructure. Similarly, Defexts [8] is an APR benchmark for Kotlin and Groovy.…”

Section: Apr Benchmarksmentioning

confidence: 99%

Can OpenAI's codex fix bugs?

Prenner

Babii

Robbes

2022

Proceedings of the Third International Workshop on Automated Program Repair

View full text Add to dashboard Cite

Recently, we can notice a transition to data-driven techniques in Automated Program Repair (APR), in particular towards deep neural networks. This entails training on hundreds of thousands or even millions of non-executable code fragments. We would like to bring more attention to an aspect of code often neglected in Neural Program Repair (NPR), namely its execution. Code execution has several significant advantages. It allows for test-based evaluation of candidate fixes and can provide valuable information to aid repair. In this work we present a fully executable dataset of 450,000 small buggy/fixed program pairs originally submitted to programming competition websites written in eight different programming languages. Along with the dataset we provide infrastructure to compile, safely execute and test programs as well as fine-grained bug-type labels. To give a point of reference, we provide basic evaluation results for two baselines, one based on a generate-andvalidate approach and one on deep learning. With this dataset we follow several goals: we want to lift Neural Program Repair beyond fully static code representations, foster the use of executionbased features and, by including several different languages, counterbalance the predominance of Java in the current landscape of APR datasets and benchmarks. Keywords automated program repair, data-driven software engineering, fault localizationRecently, more and more APR research builds on deep learning. So much so that this sub-field has been given its own name: Neural Program Repair (NPR). NPR systems are trained on up to millions of buggy/fixed code fragment pairs. So far, there is a strong focus on static code features, in particular textual features [13,47,32], less commonly, tree or graph representations [43,17].Because NPR systems are data hungry and manually collecting and isolating bugs is infeasible on a large scale, copious amounts of bug data are mined from open source code repositories (e.g., GitHub) [71].

show abstract

“…Similar to our work, BugsJS aims to fill the void of a good benchmark in its target programming language, providing 453 real bugs from 10 JavaScript programs. Defexts [2] was proposed recently for Kotlin and Groovy, providing 225 Kotlin and 301 Groovy bugs.…”

Section: Related Workmentioning

confidence: 99%

“…Python is among one of the most popular programming languages in the world today 1,2 . Understanding the bugs and faults in large software repositories built in Python is therefore important.…”

Section: Introductionmentioning

confidence: 99%

BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies

Widyasari

Sim

Lok

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

The 2019 edition of Stack Overflow developer survey highlights that, for the first time, Python outperformed Java in terms of popularity. The gap between Python and Java further widened in the 2020 edition of the survey. Unfortunately, despite the rapid increase in Python's popularity, there are not many testing and debugging tools that are designed for Python. This is in stark contrast with the abundance of testing and debugging tools for Java. Thus, there is a need to push research on tools that can help Python developers. One factor that contributed to the rapid growth of Java testing and debugging tools is the availability of benchmarks. A popular benchmark is the Defects4J benchmark; its initial version contained 357 real bugs from 5 real-world Java programs. Each bug comes with a test suite that can expose the bug. Defects4J has been used by hundreds of testing and debugging studies and has helped to push the frontier of research in these directions. In this project, inspired by Defects4J, we create another benchmark database and tool that contain 493 real bugs from 17 real-world Python programs. We hope our benchmark can help catalyze future work on testing and debugging tools that work on Python programs. CCS CONCEPTS • Software and its engineering → Software libraries and repositories.

show abstract

Defexts: A Curated Dataset of Reproducible Real-World Bugs for Modern JVM Languages

Cited by 18 publications

References 10 publications

Contrastive Code Representation Learning

Contrastive Code Representation Learning

Can OpenAI's codex fix bugs?

BugsInPy: a database of existing bugs in Python programs to enable controlled testing and debugging studies

Contact Info

Product

Resources

About