GitHub repositories with links to academic papers: Public access, traceability, and evolution

Wattanakriengkrai, Supatsara; Chinthanet, Bodin; Hata, Hideaki; Kula, Raula Gaikovina; Guo, Jin; Matsumoto, Kenichi

doi:10.1016/j.jss.2021.111117

Cited by 20 publications

(11 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study also suggested that transparency of such the networked activity information can support knowledge sharing, new ideas, and community. Wattanakriengkrai et al (2022) investigated the referencing of academic papers in README files of GitHub repositories. They found that this knowledge sharing is rarely occurring (0.4%), however, the majority of these referenced academic papers are open access (98.5%).…”

Section: Knowledge Sharingmentioning

confidence: 99%

18 million links in commit messages: purpose, evolution, and decay

et al. 2023

Self Cite

View full text Add to dashboard Cite

Commit messages contain diverse and valuable types of knowledge in all aspects of software maintenance and evolution. Links are an example of such knowledge. Previous work on "9.6 million links in source code comments" showed that links are prone to decay, become outdated, and lack bidirectional traceability. We conducted a large-scale study of 18,201,165 links from commits in 23,110 GitHub repositories to investigate whether they suffer the same fate. Results show that referencing external resources is prevalent and that the most frequent domains other than github.com are the external domains of Stack Overflow and Google Code. Similarly, links serve as source code context to commit messages, with inaccessible links being frequent. Although repeatedly referencing links is rare (4%), 14% of links that are prone to evolve become unavailable over time; e.g., tutorials or articles and software homepages become unavailable over time. Furthermore, we find that 70% of the distinct links suffer from decay; the domains that occur the most frequently are related to

show abstract

Section: Knowledge Sharingmentioning

confidence: 99%

18 million links in commit messages: purpose, evolution, and decay

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…There are efforts to share such code as open-source GitHub repositories, like Papers With Code [28]. However, it has been observed that these repositories are often incomplete, lacking key information, and not maintained, making the results difficult to reproduce [14,29,30]. This has led to the establishment of AI reproducibility challenges [30,31].…”

Section: Fair Principles For Ai Models In Hepmentioning

confidence: 99%

FAIR AI models in high energy physics

Duarte,

Li,

Roy

et al. 2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

The findable, accessible, interoperable, and reusable (FAIR) data principles provide a framework for examining, evaluating, and improving how data is shared to facilitate scientific discovery. Generalizing these principles to research software and other digital products is an active area of research. Machine learning models—algorithms that have been trained on data without being explicitly programmed—and more generally, artificial intelligence (AI) models, are an important target for this because of the ever-increasing pace with which AI is transforming scientific domains, such as experimental high energy physics (HEP). In this paper, we propose a practical definition of FAIR principles for AI models in HEP and describe a template for the application of these principles. We demonstrate the template’s use with an example AI model applied to HEP, in which a graph neural network is used to identify Higgs bosons decaying to two bottom quarks. We report on the robustness of this FAIR AI model, its portability across hardware architectures and software frameworks, and its interpretability.

show abstract

“…Previous works have investigated the opposite: representation of links to scholarly literature from scholarly source code repositories. Wattanakriengkrai et al [28] studied the extent to which scholarly papers are cited in public GitHub repositories to gain key insights into the landscape of scholarly source code production, and uncovered potential problems with long-term access, tracing, and evolution of these repositories. Färber [9] analyzed data from Microsoft Academic Graph, which attempts to map publications to their source code repositories, in order to look at the content and popularity of academic source code related to published work.…”

Section: Related Workmentioning

confidence: 99%

The Rise of GitHub in Scholarly Publications

Escamilla

Klein

Cooper

et al. 2022

Linking Theory and Practice of Digital Libraries

View full text Add to dashboard Cite

The definition of scholarly content has expanded to include the data and source code that contribute to a publication. While major archiving efforts to preserve conventional scholarly content, typically in PDFs (e.g., LOCKSS, CLOCKSS, Portico), are underway, no analogous effort has yet emerged to preserve the data and code referenced in those PDFs, particularly the scholarly code hosted online on Git Hosting Platforms (GHPs). Similarly, the Software Heritage Foundation is working to archive public source code, but there is value in archiving the issue threads, pull requests, and wikis that provide important context to the code while maintaining their original URLs. In current implementations, source code and its ephemera are not preserved, which presents a problem for scholarly projects where reproducibility matters. To understand and quantify the scope of this issue, we analyzed the use of GHP URIs in the arXiv and PMC corpora from January 2007 to December 2021. In total, there were 253,590 URIs to GitHub, SourceForge, Bitbucket, and GitLab repositories across the 2.66 million publications in the corpora. We found that GitHub, GitLab, SourceForge, and Bitbucket were collectively linked to 160 times in 2007 and 76,746 times in 2021. In 2021, one out of five publications in the arXiv corpus included a URI to GitHub. The complexity of GHPs like GitHub is not amenable to conventional Web archiving techniques. Therefore, the growing use of GHPs in scholarly publications points to an urgent and growing need for dedicated efforts to archive their holdings in order to preserve research code and its scholarly ephemera.

show abstract

GitHub repositories with links to academic papers: Public access, traceability, and evolution

Cited by 20 publications

References 15 publications

18 million links in commit messages: purpose, evolution, and decay

18 million links in commit messages: purpose, evolution, and decay

FAIR AI models in high energy physics

The Rise of GitHub in Scholarly Publications

Contact Info

Product

Resources

About