The Debsources Dataset: two decades of free and open source software

Caneill, Matthieu; Germán, Daniel M.; Zacchiroli, Stefano

doi:10.1007/s10664-016-9461-5

Cited by 22 publications

(12 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Boa was designed to let users create simple programs that allowed for a quick and comprehensive exploration of all the data. Debsources ( Caneill, Germán & Zacchiroli, 2017 ), Software Heritage ( Pietri, Spinellis & Zacchiroli, 2019 ) and World of Code ( Ma et al, 2019 ) are aimed to retrieve and archive source code, for different purposes. Debsources maintains a dataset with all the source code from Debian packages, including some metadata about it.…”

Section: Related Workmentioning

confidence: 99%

GrimoireLab: A toolset for software development analytics

Dueñas¹,

Cosentino²,

González-Barahona

et al. 2021

PeerJ Computer Science

View full text Add to dashboard Cite

Background After many years of research on software repositories, the knowledge for building mature, reusable tools that perform data retrieval, storage and basic analytics is readily available. However, there is still room to improvement in the area of reusable tools implementing this knowledge. Goal To produce a reusable toolset supporting the most common tasks when retrieving, curating and visualizing data from software repositories, allowing for the easy reproduction of data sets ready for more complex analytics, and sparing the researcher or the analyst of most of the tasks that can be automated. Method Use our experience in building tools in this domain to identify a collection of scenarios where a reusable toolset would be convenient, and the main components of such a toolset. Then build those components, and refine them incrementally using the feedback from their use in both commercial, community-based, and academic environments. Results GrimoireLab, an efficient toolset composed of five main components, supporting about 30 different kinds of data sources related to software development. It has been tested in many environments, for performing different kinds of studies, and providing different kinds of services. It features a common API for accessing the retrieved data, facilities for relating items from different data sources, semi-structured storage for easing later analysis and reproduction, and basic facilities for visualization, preliminary analysis and drill-down in the data. It is also modular, making it easy to support new kinds of data sources and analysis. Conclusions We present a mature toolset, widely tested in the field, that can help to improve the situation in the area of reusable tools for mining software repositories. We show some scenarios where it has already been used. We expect it will help to reduce the effort for doing studies or providing services in this area, leading to advances in reproducibility and comparison of results.

show abstract

Section: Related Workmentioning

confidence: 99%

GrimoireLab: A toolset for software development analytics

Dueñas¹,

Cosentino²,

González-Barahona

et al. 2021

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Hence, up to now, most studies have resorted to selecting relatively small subsets 4 of the full corpus, using different criteria, and introducing biases that are difficult to estimate. For instance, an analysis of the growth of the Debian distribution spanning two decades has been performed in [10], observing initial superlinear growth of both the number of packages and their size. But Debian is a collection maintained by humans, so the number of packages in it depends on the effort that the Debian community can consent.…”

Section: Growth Of Public Source Code (Rq3)mentioning

confidence: 99%

“…To answer this question we perform an extensive study of the Software Heritage archive, continuing a long tradition of software evolution studies [10,12,25,26,35], which we extend here by several orders of magnitude and perform over a period of more than 40 years. We show evidence of a remarkably stable exponential growth rate of original commits and files over time.…”

Section: Introductionmentioning

confidence: 99%

Software provenance tracking at the scale of public source code

2020

Self Cite

View full text Add to dashboard Cite

We study the possibilities to track provenance of software source code artifacts within the largest publicly accessible corpus of publicly available source code, the Software Heritage archive, with over 4 billions unique source code files and 1 billion commits capturing their development histories across 50 million software projects. We perform a systematic and generic estimate of the replication factor across the different layers of this corpus, analysing how much the same artifacts (e.g., SLOC, files or commits) appear in different contexts (e.g., files, commits or source code repositories). We observe a combinatorial explosion in the number of identical source code files across different commits. To discuss the implication of these findings, we benchmark different data models for capturing software provenance information at this scale, and we identify a viable solution, based on the properties of isochrone subgraphs, that is deployable on commodity hardware, is incremental and appears to be maintainable for the foreseeable future. Using these properties, we quantify, at a scale never achieved previously, the growth rate of original, i.e. neverseen-before, source code files and commits, and find it to be exponential over a period of more than 40 years.

show abstract

“…Three software engineering datasets [37][38][39] are also presented in the special section of Empirical Software Engineering on mining software repositories [33]. These datasets are given a thorough description in order that they might be of use to future researchers and practitioners in mining software repositories and in putting these valuable resources to the test.…”

Section: Data Mining Software Repositoriesmentioning

confidence: 99%

Big Data and software engineering: prospects for mutual enrichment

Arndt

2017

Iran J Comput Sci

View full text Add to dashboard Cite

Software engineering has evolved over the last 50 years, initially as a response to the so-called software crisis (the problems that organizations had producing quality software systems on time and on budget) of the 1960s and 1970s. Software engineering (SE) has been defined as "the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software". Software engineering has developed a number of approaches to areas such as software requirements, software design, software testing, and software maintenance. Software development processes such as the waterfall model, incremental development, and the spiral model have been successfully applied to produce high-quality software on time and under budget. More recently, agile software development has gained popularity as an alternative to the more traditional development methods for development of complex systems. Within the last decade or so, advances in technologies such as mobile computing, social networks, cloud computing, and the Internet of things have given rise to massive datasets which have been given the name Big Data (BD). Big Data has been defined as data with 3Vs-high volume, velocity, and variety. Big Data contains so much data that low probability events are captured in the data. These events can be discovered using analytics methods and turned into actionable intelligence which can be used by businesses to gain a competitive advantage. Unfortunately, the very scale of BD often renders inadequate SQL-based relational database systems which have formed the backbone of data intensive systems for the last 30 years, requiring new NoSQL technologies to be effective. In this paper, we will explore how well-established SE technology can be adapted to support successful development of BD projects, as well as how BD techniques can be used to increase the utility of SE processes and techniques. Thus, BD and SE may mutually support and enrich each other.

show abstract

The Debsources Dataset: two decades of free and open source software

Cited by 22 publications

References 16 publications

GrimoireLab: A toolset for software development analytics

GrimoireLab: A toolset for software development analytics

Software provenance tracking at the scale of public source code

Big Data and software engineering: prospects for mutual enrichment

Contact Info

Product

Resources

About