Design of an Empirical Study for Comparing the Usability of Concurrent Programming Languages

Abstract

The recent turn towards multicore processing architectures has made concurrency an important part of mainstream software development. As a result, an increasing number of developers have to learn to write concurrent programs, a task that is known to be hard even for the expert. Language designers are therefore working on languages that promise to make concurrent programming "easier". However, the claim that a new language is more usable than another cannot be supported by purely theoretical considerations, but calls for empirical studies. In this paper, we present the design of a study to compare concurrent programming languages with respect to comprehending and debugging existing programs and writing correct new programs. A critical challenge for such a study is avoiding the bias that might be introduced during the training phase and when interpreting participants' solutions. We address these issues by the use of self-study material and an evaluation scheme that exposes any subjective decisions of the corrector, or eliminates them altogether. We apply our design to a comparison of two object-oriented languages for concurrency, multithreaded Java and SCOOP (Simple Concurrent ObjectOriented Programming), in an academic setting. We obtain results in favor of SCOOP even though the study participants had previous training in writing multithreaded Java programs.

Benchmarking Usability and Performance of Multicore Languages

“…Nanz et al [15] present an empirical study with 67 students to compare the ease of use (program understanding, debugging, and writing) of two concurrency programming approaches (SCOOP and multi-threaded Java). They use self-study to avoid teaching bias and standard evaluation techniques to avoid subjectivity in the evaluation of the answers.…”

An Interference-Free Programming Model for Network Objects

“…What we leave to future work is a study of users themselves to determine whether the D-SCOOP abstractions are easier to read and program with, regardless of their compactness. (An existing SCOOP study is encouraging [19]. )…”

A systematic mapping study of API usability evaluation methods

“…interactive public display applications [20], concurrent programming languages [63], etc. However, in recent years we see increasing interest in evaluating API usability for security APIs, i.e., APIs that provide security features to protect data confidentiality, integrity and availability [15].…”

“…The articles in this category add a new approach to evaluate the usability of an API in the research literature. Nanz et al [63] presented a design of the study that can be used to compare ease of learning of two different concurrent programming languages by its users.…”

“…al [28] identified debuggability as a usability factor. It is also called better debugging in the work of Nanz et al [63]. In our review, we categorize these views under the same category: usability factor debuggability.…”