Identifying Experts in Software Libraries and Frameworks Among GitHub Users

Abstract

Software development increasingly depends on libraries and frameworks to increase productivity and reduce time-to-market. Despite this fact, we still lack techniques to assess developers expertise in widely popular libraries and frameworks. In this paper, we evaluate the performance of unsupervised (based on clustering) and supervised machine learning classifiers (Random Forest and SVM) to identify experts in three popular JavaScript libraries: facebook/react, mongodb/node-mongodb, and socketio/socket.io. First, we collect 13 features about developers activity on GitHub projects, including commits on source code files that depend on these libraries. We also build a ground truth including the expertise of 575 developers on the studied libraries, as self-reported by them in a survey. Based on our findings, we document the challenges of using machine learning classifiers to predict expertise in software libraries, using features extracted from GitHub. Then, we propose a method to identify library experts based on clustering feature data from GitHub; by triangulating the results of this method with information available on Linkedin profiles, we show that it is able to recommend dozens of GitHub users with evidences of being experts in the studied JavaScript libraries. We also provide a public dataset with the expertise of 575 developers on the studied libraries.

Representation of Developer Expertise in Open Source Software

“…al. [15] present an approach to determine experts for three JavaScript libraries. The authors identify developers who have made changes to projects that depend on these libraries and conduct a survey with 575 developers to obtain their self-reported expertise.…”

“…Finally, we use a previously reported survey [15] of JavaScript developers to compare how aligned each surveyed developer is to the the API in which developers were reported to be proficient. Since the survey did not include APIs where developers reported being not proficient, we randomly chose ten other APIs under the assumption that they might not be equally proficient in these 10 randomly chosen APIs.…”

“…However, previous attempts at measuring developer expertise either focus on very detailed views, e.g. counting "experience atoms" associated with changes made by a developer on a specific source code file [14], or, at the least granular level, counting the volume, frequency, and breadth [15], [16] of a developer's overall activities. Unfortunately, the former approach can not be applied for developers who have never participated in a specific project, while the latter does not account for the specific experience of a developer beyond the aggregated activity traces and projects they've worked on.…”

“…Aggregates of developer's contributions by programming language was previously proposed by Amreen et al [16], however, experience in a particular language does not immediately confer experience in the variety of libraries or frameworks in that language which specific applications might rely on. However, this measure of domain expertise, or expertise measured by the fluency of using specific APIs, is something that may be of greater concern to projects [15] than a potential contributor's overall skill in a language.…”

Studying the Health of Bitcoin Ecosystem in GitHub

“…In addition to studying GitHub itself, there are diverse studies on GitHub mining, such as identifying experts [104], analysing the discussions to explore the social norms among GitHub developers [105]. Tsay et al [105] claim that there is a lack of detailed understanding of the form and content of the discussions around pull requests.…”