The Reproducibility of Programming-Related Issues in Stack Overflow Questions

Abstract

Software developers often look for solutions to their code-level problems using the Stack Overflow Q&A website. To receive help for coding problems, developers frequently submit questions that contain sample code segments along with the description of the programming issue. Unfortunately, it is not always possible to reproduce the issues they report from the code segments they provide. Issues that are not easily reproducible may impede questions from receiving prompt and appropriate solutions. We conducted an exploratory study on the reproducibility of issues discussed in 800 Stack Overflow questions (400 Java + 400 Python). We parsed, compiled, executed, and carefully examined the code segments from these questions in order to reproduce the reported programming issues, expending 300 person-hours of effort. The outcomes of our study are three-fold. First, we found that we can reproduce the issues reported by developers for approximately 68% of Java and 71% of Python code segments, whereas we were unable to reproduce the issues for approximately 22% of Java code segments and 19% of Python code segments. Of the issues that were reproducible, approximately 67% of the Java code segments and 20% of the Python code segments required minor or major modifications to reproduce the issues. Second, we also carefully investi-