Source code analysis dataset

Abstract

The data in this article pair source code with three artifacts from 108,568 projects downloaded from Github that have a redistributable license and at least 10 stars. The first set of pairs connects snippets of source code in C, C++, Java, and Python with their corresponding comments, which are extracted using Doxygen. The second set of pairs connects raw C and C++ source code repositories with the build artifacts of that code, which are obtained by running the make command. The last set of pairs connects raw C and C++ source code repositories with potential code vulnerabilities, which are determined by running the Infer static analyzer. The code and comment pairs can be used for tasks such as predicting comments or creating natural language descriptions of code. The code and build artifact pairs can be used for tasks such as reverse engineering or improving intermediate representations of code from decompiled binaries. The code and static analyzer pairs can be used for tasks such as machine learning approaches to vulnerability discovery.

An open-source natural language processing toolkit to support software development: addressing automatic bug detection, code summarisation and code search

“…The augmented C/C++ dataset, built by extracting 250k observations in C and 250k observations in C++ from a SQL database provided by 29 . They collected source code files from GitHub repositories written in C, C++, Java and Python; extracted comments using Doxygen and condensed such pairs into the database.…”

“…For the generation of JAVA, we based our experiments on the Concode corpus 30 . For C++, we used the DECODER OpenCV use case corpus presented above and a larger but less clean corpus, from the “Code and comments dataset” 29 , of substantial size, named “C&C” below. This dataset contains a total of 16,115,540 pairs of comment and code, mined from 106,304 GitHub projects coded in Python, Java, C and C++ 31 .…”

PENTACET data - 23 Million Contextual Code Comments and 250,000 SATD comments

“…Context in prior work [25] Code Context in SoCCMiner [20] public void NoEvent(int ix, V val){ // TODO elastic? // TODO elastic?…”

The Integrity of Source Code Commenting : Benchmark Dataset and Empirical Analysis

“…To collect a set of naturally written comments by developers, we used the extracted code-comment pair by Gelman et al [22]. This datapool of codecomment pairs holds extracted pairs from GitHub projects for five different programming languages.…”

“…C# is only considered in the scope of this work. To pull the code-comment pair, the author of [22] chose the GitHub repository based on having a redistributable license and at least 10 stars. Table 1 represents the list of the C# projects from the datapool, selected for this research.…”