A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks

Pimentel, João Felipe; Murta, Leonardo; Braganholo, Vanessa; Freire, Juliana

doi:10.1109/msr.2019.00077

Cited by 154 publications

(122 citation statements)

References 15 publications

Supporting

Mentioning

110

Contrasting

Unclassified

Order By: Relevance

“…Unfortunately, many code snippets require a non-trivial environment configuration in order to execute successfully [7], [8], and are not accompanied by sufficient information for developers to easily recreate that configuration [3], [9], [10]. This leads to the problem of configuration drift, where a code snippet goes out-of-date because the APIs that it depends on experience breaking changes over time.…”

Section: Introductionmentioning

confidence: 99%

V2: Fast Detection of Configuration Drift in Python

Horton

Parnin

2019

2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)

View full text Add to dashboard Cite

Code snippets are prevalent, but are hard to reuse because they often lack an accompanying environment configuration. Most are not actively maintained, allowing for drift between the most recent possible configuration and the code snippet as the snippet becomes out-of-date over time. Recent work has identified the problem of validating and detecting out-of-date code snippets as the most important consideration for code reuse. However, determining if a snippet is correct, but simply out-of-date, is a non-trivial task. In the best case, breaking changes are well documented, allowing developers to manually determine when a code snippet contains an out-of-date API usage. In the worst case, determining if and when a breaking change was made requires an exhaustive search through previous dependency versions.We present V2, a strategy for determining if a code snippet is out-of-date by detecting discrete instances of configuration drift, where the snippet uses an API which has since undergone a breaking change. Each instance of configuration drift is classified by a failure encountered during validation and a configuration patch, consisting of dependency version changes, which fixes the underlying fault. V2 uses feedback-directed search to explore the possible configuration space for a code snippet, reducing the number of potential environment configurations that need to be validated. When run on a corpus of public Python snippets from prior research, V2 identifies 248 instances of configuration drift.

show abstract

Section: Introductionmentioning

confidence: 99%

V2: Fast Detection of Configuration Drift in Python

Horton

Parnin

2019

2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)

View full text Add to dashboard Cite

show abstract

“…Notebooks have been applauded by many, yet have also seen push back from an engineering point of view (see e.g. Grus, 2018;Pimentel et al, 2019). We argue that they work well for freeform documentation, and demonstrating selected parts of research.…”

Section: Sub-challenge 2: Interpretability Of Reproducible Researchmentioning

confidence: 76%

Towards Replication in Computational Cognitive Modeling: A Machine Learning Perspective

Emmery¹,

Kádár²,

Wiltshire³

et al. 2019

Preprint

View full text Add to dashboard Cite

The suggestions proposed by Lee et al. to improve cognitive modeling practices have significant parallels to the current best practices for improving reproducibility in the field of Machine Learning. In the current commentary on `Robust modeling in cognitive science', we highlight the practices that overlap and discuss how similar proposals have produced novel ongoing challenges, including cultural change towards open science, the scalability and interpretability of required practices, and the downstream effects of having robust practices that are fully transparent. Through this, we hope to inform future practices in computational modeling work with a broader scope.

show abstract

“…Currently, there are several studies tackling usability, cognitive, and reproducibility aspects of computational notebooks. For instance, in reproducibility [36,47,48,58], education [44], exploratory programming [2,18,29,30], documentation [76], and data journalism [78]. However, no attention has been paid from the language engineering point of view.…”

Section: :21mentioning

confidence: 99%

“…Wolfram Mathematica was one of the first platforms for notebooks [17] [75], yet its appropriation was limited due to their commercial licensing model. Later, in 2014, Project Jupyter [33] developed an open-source notebook platform that has widespread the adoption of the notebook metaphor among different disciplines [29,47,55,56,60].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bacatá: Notebooks for DSLs, Almost for Free

2020

View full text Add to dashboard Cite

Context: Computational notebooks are a contemporary style of literate programming, in which users can communicate and transfer knowledge by interleaving executable code, output, and prose in a single rich document. A Domain-Specific Language (DSL) is a software language tailored for an application domain. Usually, DSL users are domain experts that may not have a software engineering background. Therefore, they might not be familiar with Integrated Development Environments (IDEs). In brief, the development of tools that offer different interfaces for interacting with a DSL is relevant.Inquiry: However, DSL designers' resources are limited. We want to leverage General-purpose Languages (GPLs) tooling in the context of DSLs. Computational notebooks are an example of such tools. Then, our main question is: What is an efficient and effective method of designing and implementing notebook interfaces for DSLs? By addressing this question, we might be able to speed up the development of DSL tools, and ease the interaction between end-users and DSLs.Approach: In this paper, we present Bacatá, a mechanism for generating notebook interfaces for external DSLs in a language-parametric fashion. This mechanism is designed in a way in which language engineers can reuse as many language components as possible (e.g., language processors, type checkers, code generators). In addition, we present a Feature-Oriented Domain Analysis that depicts language dependent and language independent features of computational notebooks.Knowledge: Our results show that notebook interfaces generated by Bacatá can be used with little manual configuration. However, there are a few considerations and caveats that should be addressed by language engineers that rely on language design aspects. The creation of a notebook for a DSL with Bacatá becomes a matter of writing the code that wires existing language components in the Rascal language workbench with the Jupyter platform.Grounding: We evaluate Bacatá by generating functional computational notebook interfaces for three different non-trivial DSLs, namely: a small subset of Halide (a DSL for digital image processing), SweeterJS (an extended version of JavaScript), and QL (a DSL for questionnaires). Additionally, it is relevant to generate notebook implementations rather than implementing them manually. To illustrate this, we measured and compared the number of source lines of code that we reused from existing implementations of those languages.Importance: The adoption of notebooks by novice-programmers and end-users has made them very popular in several domains such as exploratory programming, data science, data journalism, and machine learning. Why are they popular? In (data) science, it is essential to make results reproducible as well as understandable. However, notebooks are only available for GPLs. This paper opens the notebook metaphor for DSLs to improve the end-user experience when interacting with code and to increase DSLs adoption. ACM CCS 2012Software and its engineering → Domain specific lang...

show abstract

A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks

Cited by 154 publications

References 15 publications

V2: Fast Detection of Configuration Drift in Python

V2: Fast Detection of Configuration Drift in Python

Towards Replication in Computational Cognitive Modeling: A Machine Learning Perspective

Bacatá: Notebooks for DSLs, Almost for Free

Contact Info

Product

Resources

About