Towards Understanding Data Analysis Workflows using a Large Notebook Corpus

Rehman, M. Suhail

doi:10.1145/3299869.3300107

Cited by 9 publications

(4 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite synthesizing rich observations, interview studies were limited to dozens of participants. A few studies conducted large-scale analysis of Jupyter notebooks, but were limited to simple summary statistics [6], a single library [7], or code quality [8]. Our model enables the analysis of data science both at scale and in depth, which may validate and complement findings from previous qualitative studies.…”

Section: Studies Of Data Analysis Practicesmentioning

confidence: 78%

“…1 In this layer, we aim to compute p stage -a probability distribution over these six stages -from the topic distribution computed in Eq. (7). We implement this by mapping the topic distribution p topic to a probability distribution p stage over the n stages = 6 stages.…”

Section: A Cell With One Line Of Code That Does Not Create a New Vari...mentioning

confidence: 99%

“…Previous in depth analyses of scientific code heavily rely on expert annotations, limiting the scale of these studies to the order of a hundred examples [5,6]. Large-scale studies across thousands of examples have been limited to simple summaries such as the number or nature of imported libraries, total line counts, or the fraction of lines that are used for comments [6][7][8]. The software engineering community has emphasized the inadequacy of these analyses, noting that "there is a strong need to programmatically analyze Jupyter notebooks" [9], while HCI researchers have observed that studying the data science process through notebooks may play a role in addressing the scientific reproducability crisis [5,10].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

CORAL: COde RepresentAtion learning with weakly-supervised transformers for analyzing data analysis

et al. 2022

View full text Add to dashboard Cite

Large scale analysis of source code, and in particular scientific source code, holds the promise of better understanding the data science process, identifying analytical best practices, and providing insights to the builders of scientific toolkits. However, large corpora have remained unanalyzed in depth, as descriptive labels are absent and require expert domain knowledge to generate. We propose a novel weakly supervised transformer-based architecture for computing joint representations of code from both abstract syntax trees and surrounding natural language comments. We then evaluate the model on a new classification task for labeling computational notebook cells as stages in the data analysis process from data import to wrangling, exploration, modeling, and evaluation. We show that our model, leveraging only easily-available weak supervision, achieves a 38% increase in accuracy over expert-supplied heuristics and outperforms a suite of baselines. Our model enables us to examine a set of 118,000 Jupyter Notebooks to uncover common data analysis patterns. Focusing on notebooks with relationships to academic articles, we conduct the largest study of scientific code to date and find that notebooks which devote an higher fraction of code to the typically labor-intensive process of wrangling data in expectation exhibit decreased citation counts for corresponding papers. We also show significant differences between academic and non-academic notebooks, including that academic notebooks devote substantially more code to wrangling and exploring data, and less on modeling.

show abstract

Section: Studies Of Data Analysis Practicesmentioning

confidence: 78%

Section: A Cell With One Line Of Code That Does Not Create a New Vari...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CORAL: COde RepresentAtion learning with weakly-supervised transformers for analyzing data analysis

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Previous in-depth analyses of scientific code heavily rely on expert annotations, limiting the scale of these studies to the order of a hundred examples [5], [6]. Large-scale studies across thousands of examples have been limited to simple summaries such as the number or nature of imported libraries, total line counts, or the fraction of lines that are used for comments [6], [7], [8]. The software engineering community has emphasized the inadequacy of these analyses, noting that "there is a strong need to programmatically analyze Jupyter notebooks" [9], while HCI researchers have observed that studying the data science § These authors contributed equally to this work.…”

Section: Introductionmentioning

confidence: 99%

CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers for Analyzing Data Analysis

Zhang¹,

Merrill²,

Liu³

et al. 2020

Preprint

View full text Add to dashboard Cite

Large scale analysis of source code, and in particular scientific source code, holds the promise of better understanding the data science process, identifying analytical best practices, and providing insights to the builders of scientific toolkits. However, large corpora have remained unanalyzed in depth, as descriptive labels are absent and require expert domain knowledge to generate. We propose a novel weakly supervised transformer-based architecture for computing joint representations of code from both abstract syntax trees and surrounding natural language comments. We then evaluate the model on a new classification task for labeling computational notebook cells as stages in the data analysis process from data import to wrangling, exploration, modeling, and evaluation. We show that our model, leveraging only easily-available weak supervision, achieves a 38% increase in accuracy over expert-supplied heuristics and outperforms a suite of baselines. Our model enables us to examine a set of 118,000 Jupyter Notebooks to uncover common data analysis patterns. Focusing on notebooks with relationships to academic articles, we conduct the largest ever study of scientific code and find that notebook composition correlates with the citation count of corresponding papers.

show abstract

ReproduceMeGit: A Visualization Tool for Analyzing Reproducibility of Jupyter Notebooks

Samuel

König-Ries

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Towards Understanding Data Analysis Workflows using a Large Notebook Corpus

Cited by 9 publications

References 2 publications

CORAL: COde RepresentAtion learning with weakly-supervised transformers for analyzing data analysis

CORAL: COde RepresentAtion learning with weakly-supervised transformers for analyzing data analysis

CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers for Analyzing Data Analysis

ReproduceMeGit: A Visualization Tool for Analyzing Reproducibility of Jupyter Notebooks

Contact Info

Product

Resources

About