Evaluating error detection capabilities of UPC run-time systems

Abstract

Abstract-The ability of system software to detect run-time errors and issue messages that help programmers quickly fix these errors is an important productivity criterion for developing and maintaining application programs. To evaluate this capability for Unified Parallel C (UPC), over two thousand run-time error tests and a run-time error detection (RTED) evaluation tool have been developed. For each error message issued, the RTED evaluation tool assigns a score from 0 to 5 based on the usefulness of the information in the message to help a programmer quickly fix the error. The RTED evaluation tool calculates averages over each error category and then prints the results. All tests and the RTED evaluation tool are freely available at the RTED web site http://rted.public.iastate.edu/UPC. The Cray, Berkeley, HP and GNU UPC compilers have been evaluated and results posted on this same web site.

UPC-CHECK: A scalable tool for detecting run-time errors in Unified Parallel C

“…However, extensive studies show that the error detection capability of compilers and runtime environments for the UPC language is poor [25,30]. In the absence of such error detection capabilities in the system software, a programmer needs to debug the program by manually computing the program.…”

“…The message pin points the kind of error, line number and the name of the file where the error occurred and information which might be helpful to the programmer to fix the error quickly. The error detection capability of the tool and the quality of error messages issued were graded using the UPC-RTED [30] tool. The results show a marked improvement in the error detection and reporting capability over existing run-time systems like Cray, Berkeley, HP and GNU.…”

“…The importance of error detection is well documented by Glenn Luecke et al [31]: "the ability of system software to detect run-time errors and issue messages that help programmers quickly correct these errors is an important productivity criterion for developing and maintaining application programs". However, studies show that currently the error detection capability of compilers and run-time systems for Unified Parallel C (UPC) [10,3,17] is poor [25,30].…”