Let's talk about it: evaluating contributions through discussion in GitHub

Abstract

Open source software projects often rely on code contributions from a wide variety of developers to extend the capabilities of their software. Project members evaluate these contributions and often engage in extended discussions to decide whether to integrate changes. These discussions have important implications for project management regarding new contributors and evolution of project requirements and direction. We present a study of how developers in open work environments evaluate and discuss pull requests, a primary method of contribution in GitHub, analyzing a sample of extended discussions around pull requests and interviews with GitHub developers. We found that developers raised issues around contributions over both the appropriateness of the problem that the submitter attempted to solve and the correctness of the implemented solution. Both core project members and third-party stakeholders discussed and sometimes implemented alternative solutions to address these issues. Different stakeholders also influenced the outcome of the evaluation by eliciting support from different communities such as dependent projects or even companies. We also found that evaluation outcomes may be more complex than simply acceptance or rejection. In some cases, although a submitter's contribution was rejected, the core team fulfilled the submitter's technical goals by implementing an alternative solution. We found that the level of a submitter's prior interaction on a project changed how politely developers discussed the contribution and the nature of proposed alternative solutions.

Work practices and challenges in pull-based development

“…They experience difficulties prioritizing the contributions to be merged and face challenges identifying factors that will reveal contribution quality. By focusing on how discussions affect contribution evaluation in GitHub, Tsay et al [46] found that stakeholders external to the project may influence the evaluation discussions while power plays are in effect. Social signals also play an important role: Marlow et al [34] found that core members form an impression of the quality of incoming contributions by using social signals such as the developer's coding activity and the developer's social actions (e.g., following other developers).…”

“…Previous research considered the lifetime characteristics of PRs [25], macroscopic factors that lead to contribution acceptance [25,45], the barriers faced by first time contributors [44], how contributions are evaluated through discussions [46], and the working habits and challenges faced by integrators [27]. Here we present the contributor's perspective by investigating contributors' work habits and the challenges they face.…”

Automatic Classification of Review Comments in Pull-based Development Model

“…Previous work has studied the challenges faced by pullrequest reviewers and the issues introduced by pull-request submitters [10], [20]. Inspired by their work, we decide to comprehensively observe what reviewers are talking about in code reviews rather than merely focusing on technical and nontechnical perspectives.…”

“…Code review is a communication channel where integrators, who are core members of a project, can express their concern for the contribution [10], [20], [22]. If they doubt the quality of a submitted pull-request, integrators make comments that ask the contributors to provide several use cases or improve the implementation.…”

Arsonists or Firefighters? Affectiveness in Agile Software Development

“…The authors demonstrated that emotions did have an effect on the issue fixing time. Research has focused on understanding how the human aspects of a technical discipline can affect final results [3,7,11], and the effect of politeness [14,23,25]. The Manifesto for Agile Development indicates that people and communications are more essential than procedures and tools [2].…”