Gistable: Evaluating the Executability of Python Code Snippets on GitHub

Abstract

Software developers create and share code online to demonstrate programming language concepts and programming tasks. Code snippets can be a useful way to explain and demonstrate a programming concept, but may not always be directly executable. A code snippet can contain parse errors, or fail to execute if the environment contains unmet dependencies.This paper presents an empirical analysis of the executable status of Python code snippets shared through the GitHub gist system, and the ability of developers familiar with software configuration to correctly configure and run them. We find that 75.6% of gists require non-trivial configuration to overcome missing dependencies, configuration files, reliance on a specific operating system, or some other environment configuration. Our study also suggests the natural assumption developers make about resource names when resolving configuration errors is correct less than half the time.We also present Gistable, a database and extensible framework built on GitHub's gist system, which provides executable code snippets to enable reproducible studies in software engineering. Gistable contains 10,259 code snippets, approximately 5,000 with a Dockerfile to configure and execute them without import error. Gistable is publicly available at https://github.com/gistable/gistable.Another developer had difficult working with a cloud provider package.Couldn't find clouddns module. Couldn't solve dependency. Spent 2 hours on it.

V2: Fast Detection of Configuration Drift in Python

“…McIntosh et al agree that build maintenance imposes a large overhead on developers [22]. Yang et al [7] evaluate Python code snippets, as do Horton and Parnin [3]. They find that Python code snippets are not executable by default due to configuration errors, and Horton and Parnin additionally highlight the difficulty of correct configuration, of which inferring correct dependency versions ranks highly.…”

“…Although these works demonstrate that configuration of dependencies is a frequent cause of build and execution failure, and that developers have need of better tooling support of versions [27], [28], [29], [30], no study attempts to uncover instances of configuration drift automatically. Horton and Parnin do classify 15 code snippets which have drifted and only execute successfully with an older version of a dependency (Table II from Gistable [3]), but this classification was performed by manually generating configuration scripts.…”

DockerizeMe: Automatic Inference of Environment Dependencies for Python Code Snippets

“…Seo et al also found that approximately 1 https://gist.github.com/1cdd9646046ae10c2932 2 https://github.com/getsentry/raven-python/issues/1075 50% of build errors are caused by dependencies [7]. Further, the effort involved in manually constructing an environment specification is non-trivial -developers can spend between 20 minutes and 2 hours creating a Dockerfile for a single code snippet, and often fail to construct a valid specification [6]. Common challenges include mapping a code resource to its originating package and determining the correct order of installation for transitive dependencies.…”

“…To evaluate DockerizeMe, we performed environment configuration for 2,891 gists from the Gistable dataset which still failed due to Python's ImportError after applying Gistable's Environment Inference Algorithm (Section III-C from [6]). DockerizeMe successfully removed import errors for 892 gists.…”

“…• An inference algorithm for direct and transitive dependencies that respects installation order. [3], [6]. Discovering all required dependencies is a time consuming process that even experienced developers have trouble with [6].…”