Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code

Abstract

We present in this paper an automated method to assess the quality of Jupyter notebooks. The quality of notebooks is assessed in terms of reproducibility and executability. Specifically, we automatically extract a number of expert-defined features for each notebook, perform a feature selection step, and then trained supervised binary classifiers to predict whether a notebook is reproducible and executable, respectively. We also experimented with semantic code embeddings to capture the notebooks' semantics. We have evaluated these methods on a dataset of 306,539 notebooks and achieved an F1 score of 0.87 for reproducibility and 0.96 for executability (using expert-defined features) and an F1 score of 0.81 for reproducibility and 0.78 for executability (using code embeddings). Our results suggest that semantic code embeddings can be used to determine with good performance the reproducibility and executability of Jupyter notebooks, and since they can be automatically derived, they have the advantage of no need for expert involvement to define features.

Purchasing Patterns Analysis in E-commerce: A Big Data-driven Approach and Methodological

“…Various analytical techniques are used to understand the collected data. Descriptive analysis is used to understand the distribution of data, calculate summary statistics such as mean, median, mode, and standard deviation, and identify patterns and trends in data [42]. Meanwhile, inferential analysis tests hypotheses about relationships between variables, builds statistical models to predict purchasing behavior, and groups customers based on their profiles and preferences [38].…”

An approach to assess the quality of Jupyter projects published by GLAM institutions

“…The quality of notebooks has been addressed in previous works that identify common issues (Oli et al, 2021;Pimentel et al, 2019Pimentel et al, , 2021. Examples of these include: the execution order of cells, and problems such as unnamed notebooks (e.g., Untitled1.ipynb) or the reproducibility of results.…”