BugsJS: a Benchmark of JavaScript Bugs

Abstract

JavaScript is a popular programming language that is also error-prone due to its asynchronous, dynamic, and loosely-typed nature. In recent years, numerous techniques have been proposed for analyzing and testing JavaScript applications. However, our survey of the literature in this area revealed that the proposed techniques are often evaluated on different datasets of programs and bugs. The lack of a commonly used benchmark limits the ability to perform fair and unbiased comparisons for assessing the efficacy of new techniques. To fill this gap, we propose BUGSJS, a benchmark of 453 real, manually validated JavaScript bugs from 10 popular JavaScript server-side programs, comprising 444k LOC in total. Each bug is accompanied by its bug report, the test cases that detect it, as well as the patch that fixes it. BUGSJS features a rich interface for accessing the faulty and fixed versions of the programs and executing the corresponding test cases, which facilitates conducting highly-reproducible empirical studies and comparisons of JavaScript analysis and testing tools.

Poster: Supporting JavaScript Experimentation with BugsJS

“…As part of the analysis and classification of BUGSJS bugs [4], we categorized them based on recurring bug-fix patterns proposed by Pan et al [12]. By comparing the results of this categorization and the ranking scores of the fixing methods provided by Tarantula, we expect to gather insights on how different types of bugs can be successfully localized.…”

“…More details about the characteristics of BUGSJS, its architecture, and our design choices can be found in our full paper [4]. The interested reader can find more information on BUGSJS and access the benchmark on our website: https://bugsjs.github.io/…”

“…Comparing the efficacy of these techniques on a common centralized benchmark is imperative to ensure reliability and replicability. Unfortunately, in the JavaScript domain, our survey of the previous studies showed an extensive heterogeneity in the used evaluation subject systems [4].…”

“…To fill this gap, we have proposed BUGSJS, a benchmark of several hundreds manually-validated bugs from 10 popular JavaScript programs, along with comprehensive bug reports and one or more test cases that demonstrate the bugs [4]. These artifacts can be used for devising and evaluating analysis and testing techniques for JavaScript, facilitated through an API that allows accessing and executing the faulty and fixed versions of the programs and the corresponding tests.…”

BUGSJS: a benchmark and taxonomy of JavaScript bugs

“…In our previous work [21], we presented BUGSJS, a benchmark of 453 JS-related bugs from 10 open-source JS projects, based on Node.js and the Mocha testing framework. BUGSJS features an infrastructure containing detailed reports about the bugs, the faulty versions of programs, the test cases exposing them, as well as the patches that fix them.…”