PyDriller: Python framework for mining software repositories

Spadini, Davide; Aniche, Maurício; Bacchelli, Alberto

doi:10.1145/3236024.3264598

Cited by 231 publications

(131 citation statements)

References 24 publications

Supporting

Mentioning

118

Contrasting

Unclassified

Order By: Relevance

“…PyDriller [39] is a Python framework meant for mining Git repositories. It provides easy extraction of information from a Git repository.…”

Section: Pydrillermentioning

confidence: 99%

“…We extracted the commit information from the Git repositories using PyDriller [39]. This tool was used instead of connecting directly to the GitHub API for maintainability reasons.…”

Section: Commit and Changes Informationmentioning

confidence: 99%

“…The dataset was built by extracting the projects' data and analyzing it using several tools. To get the data, the projects' GitHub repositories were cloned, commit information was collected from the git log using PyDriller [39], refactorings were classified using Refactoring Miner [41], and fault information was obtained by extracting issues from the Jira issue tracker. After that, code quality was inspected using two tools: Technical Debt items were analyzed with SonarQube, and code smells [10] and anti-patterns [1] with Ptidej [13].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Technical Debt Dataset

Lenarduzzi

Saarimäki

Taibi

2019

Proceedings of the Fifteenth International Conference on Predictive Models and Data Analytics in Software Engineering

View full text Add to dashboard Cite

Technical Debt analysis is increasing in popularity as nowadays researchers and industry are adopting various tools for static code analysis to evaluate the quality of their code. Despite this, empirical studies on software projects are expensive because of the time needed to analyze the projects. In addition, the results are difficult to compare as studies commonly consider different projects. In this work, we propose the Technical Debt Dataset, a curated set of project measurement data from 33 Java projects from the Apache Software Foundation. In the Technical Debt Dataset, we analyzed all commits from separately defined time frames with SonarQube to collect Technical Debt information and with Ptidej to detect code smells. Moreover, we extracted all available commit information from the git logs, the refactoring applied with Refactoring Miner, and fault information reported in the issue trackers (Jira). Using this information, we executed the SZZ algorithm to identify the fault-inducing and -fixing commits. We analyzed 78K commits from the selected 33 projects, detecting 1.8M SonarQube issues, 38K code smells, 28K faults and 57K refactorings. The project analysis took more than 200 days. In this paper, we describe the data retrieval pipeline together with the tools used for the analysis. The dataset is made available through CSV files and an SQLite database to facilitate queries on the data. The Technical Debt Dataset aims to open up diverse opportunities for Technical Debt research, enabling researchers to compare results on common projects.

show abstract

“…PyDriller [39] is a Python framework meant for mining Git repositories. It provides easy extraction of information from a Git repository.…”

Section: Pydrillermentioning

confidence: 99%

“…We extracted the commit information from the Git repositories using PyDriller [39]. This tool was used instead of connecting directly to the GitHub API for maintainability reasons.…”

Section: Commit and Changes Informationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Technical Debt Dataset

Lenarduzzi

Saarimäki

Taibi

2019

Proceedings of the Fifteenth International Conference on Predictive Models and Data Analytics in Software Engineering

View full text Add to dashboard Cite

show abstract

“…Sets of changes made by a developer to potentially multiple files are recorded as commits, where each commit is identified by a unique hash. Building on the package pydriller [7], we first extract the history of all commits in a repository and record the meta-data (author, time of commit, branch, etc.) for each commit.…”

Section: Mining Co-editing Relations From Git Repositories a Frmentioning

confidence: 99%

“…We introduce git2net, a python tool that can be used to mine time-stamped co-editing relations between developers from the sequence of file modifications contained in git repositories. Building on the repository mining framework pyDriller [7], git2net can operate both on local and remote repositories. Providing a command-line interface as well as an API, git2net can be used as stand-alone tool for standard analysis tasks as well as a framework for the implementation of advanced data mining scripts.…”

Section: Introductionmentioning

confidence: 99%

git2net - Mining Time-Stamped Co-Editing Networks from Large git Repositories

Gote

Scholtes

Schweitzer

2019

2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)

View full text Add to dashboard Cite

Data from software repositories have become an important foundation for the empirical study of software engineering processes. A recurring theme in the repository mining literature is the inference of developer networks capturing e.g. collaboration, coordination, or communication from the commit history of projects. Most of the studied networks are based on the co-authorship of software artefacts defined at the level of files, modules, or packages. While this approach has led to insights into the social aspects of software development, it neglects detailed information on code changes and code ownership, e.g. which exact lines of code have been authored by which developers, that is contained in the commit log of software projects.Addressing this issue, we introduce git2net, a scalable python software that facilitates the extraction of fine-grained co-editing networks in large git repositories. It uses text mining techniques to analyse the detailed history of textual modifications within files. This information allows us to construct directed, weighted, and time-stamped networks, where a link signifies that one developer has edited a block of source code originally written by another developer. Our tool is applied in case studies of an Open Source and a commercial software project. We argue that it opens up a massive new source of high-resolution data on human collaboration patterns. 1 https://github.com/gotec/git2net © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

show abstract

Exploring the relationship between refactoring and code debt indicators

Halepmollasi

Tosun

2022

J Software Evolu Process

View full text Add to dashboard Cite

Refactoring, which aims to improve the internal structure of the software systems preserving their behavior, is the most common payment strategy for technical debt (TD) by removing the code smells. There exist many studies presenting code smell detection approaches/tools or investigating their impact on quality attributes. There are also studies that focus on refactoring techniques, their relation with quality attributes, tool supports, and opportunities for them. Although there are several studies addressing the gap between refactoring and TD indicators, the empirical evidence provided is still limited. In this study, we examine the distribution of 29 refactoring types among the different projects and their relation with code smells or faults. We explore the refactoring types that are most commonly performed together and other activities performed with refactorings. We conduct a large exploratory study with automatically detected 57,528 refactorings, 37,553 smells, 27,340 faults, and 134,812 commits of 33 Java projects. Results show that some refactoring types are more commonly applied by developers. Our analysis indicates that refactorings usually remove or do not affect the code smells, and this contradicts with the previous studies. Also, the commits in which refactoring(s) is performed are three times more fault inducing than those without refactoring.

show abstract

PyDriller: Python framework for mining software repositories

Cited by 231 publications

References 24 publications

The Technical Debt Dataset

The Technical Debt Dataset

git2net - Mining Time-Stamped Co-Editing Networks from Large git Repositories

Exploring the relationship between refactoring and code debt indicators

Contact Info

Product

Resources

About