Replication Can Improve Prior Results: A GitHub Study of Pull Request Acceptance

Chen, Di; Stolee, Kathryn T.

doi:10.1109/icpc.2019.00037

Cited by 20 publications

(14 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2. Such a 50% "cull rate" is common practice in crowdsourcincg [19] tions, labelling 500 projects of Github issues would require 39,000 hours and $320K of grant money (with nothing left over for graduate student wages or other equipment).…”

Section: The Role Of Labelling In Data Miningmentioning

confidence: 99%

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Standard automatic methods for recognizing problematic development commits can be greatly improved via the incremental application of human+artificial expertise. In this approach, called EMBLEM, an AI tool first explore the software development process to label commits that are most problematic. Humans then apply their expertise to check those labels (perhaps resulting in the AI updating the support vectors within their SVM learner). We recommend this human+AI partnership, for several reasons. When a new domain is encountered, EMBLEM can learn better ways to label which comments refer to real problems. Also, in studies with 9 open source software projects, labelling via EMBLEM's incremental application of human+AI is at least an order of magnitude cheaper than existing methods (≈ eight times). Further, EMBLEM is very effective. For the data sets explored here, EMBLEM better labelling methods significantly improved Popt20 and G-score performance in nearly all the projects studied here. TABLE 1This paper argues against using keywords like these as a method for labelling a commit as "buggy'.

show abstract

Section: The Role Of Labelling In Data Miningmentioning

confidence: 99%

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…We use Amazon's Mechanical Turk ( AMT ; Amazon Mechanical Turk & Inc., 2018) to select participants for our experiment. AMT has been used before to recruit subjects for programming studies (Alqaimi, Thongtanunam, & Treude, 2019; Chen, Stolee, & Menzies, 2019; Prana, Treude, Thung, Atapattu, & Lo, 2019). Since anyone can sign up with AMT, we selectively filter out a sample that can reasonably represent Java programmers.…”

Section: Methodsmentioning

confidence: 99%

“…AMT has been used before to recruit subjects for programming studies (Prana et al, 2019;Chen et al, 2019;Alqaimi et al, 2019). Since anyone can sign up with AMT, we selectively filter out a sample that can reasonably represent Java programmers.…”

Section: Methodsmentioning

confidence: 99%

Do Programmers Prefer Predictable Expressions in Code?

et al. 2020

View full text Add to dashboard Cite

Source code is a form of human communication, albeit one where the information shared between the programmers reading and writing the code is constrained by the requirement that the code executes correctly. Programming languages are more syntactically constrained than natural languages, but they are also very expressive, allowing a great many different ways to express even very simple computations. Still, code written by developers is highly predictable, and many programming tools have taken advantage of this phenomenon, relying on language model surprisal as a guiding mechanism. While surprisal has been validated as a measure of cognitive load in natural language, its relation to human cognitive processes in code is still poorly understood. In this paper, we explore the relationship between surprisal and programmer preference at a small granularity—do programmers prefer more predictable expressions in code? Using meaning‐preserving transformations, we produce equivalent alternatives to developer‐written code expressions and run a corpus study on Java and Python projects. In general, language models rate the code expressions developers choose to write as more predictable than these transformed alternatives. Then, we perform two human subject studies asking participants to choose between two equivalent snippets of Java code with different surprisal scores (one original and transformed). We find that programmers do prefer more predictable variants, and that stronger language models like the transformer align more often and more consistently with these preferences.

show abstract

“…Tu et al [67] offers a cost estimate of what resources would be required to sub-contract that effort to dozens of crowd sourced workers via tools like Mechanical Turk (MT). Applying best practices in crowd sourcing [10], assuming (a) at least USA minimum ages [60]; and (b) our university taking a 50% overhead tax on grants; then crowd sourcing the labeling of the issues from 500 projects would require $320,000 of grant reserve.…”

Section: Crowd Sourcingmentioning

confidence: 99%

DebtFree: Minimizing Labeling Cost in Self-Admitted Technical Debt Identification using Semi-Supervised Learning

Tu¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Keeping track of and managing Self-Admitted Technical Debts (SATDs) is important for maintaining a healthy software project. Current active-learning SATD recognition tool involves manual inspection of 24% of the test comments on average to reach 90% of the recall. Among all the test comments, about 5% are SATDs. The human experts are then required to read almost a quintuple of the SATD comments which indicates the inefficiency of the tool. Plus, human experts are still prone to error: 95% of the false-positive labels from previous work were actually true positives.To solve the above problems, we propose DebtFree, a two-mode framework based on unsupervised learning for identifying SATDs. In mode1, when the existing training data is unlabeled, DebtFree starts with an unsupervised learner to automatically pseudo-label the programming comments in the training data. In contrasts, in mode2 where labels are available with the corresponding training data, DebtFree starts with a pre-processor that identifies the highly prone SATDs from the test dataset. Then, our machine learning model is employed to assist human experts in manually identifying the remaining SATDs. Our experiments on 10 software projects show that both models yield statistically significant improvement in effectiveness over the state-of-the-art automated and semi-automated models. Specifically, DebtFree can reduce the labeling effort by 99% in mode1 (unlabeled training data), and up to 63% in mode2 (labeled training data) while improving the current active learner's F1 relatively to almost 100%.

show abstract

Replication Can Improve Prior Results: A GitHub Study of Pull Request Acceptance

Cited by 20 publications

References 57 publications

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

Do Programmers Prefer Predictable Expressions in Code?

DebtFree: Minimizing Labeling Cost in Self-Admitted Technical Debt Identification using Semi-Supervised Learning

Contact Info

Product

Resources

About