Issues in computer performance evaluation

Abstract

This paper summarizes the results of an ACM/NBS Workshop on Computer Performance Evaluation. Computer Performance Evaluation (CPE) was selected as the subject of an ACM/NBS Workshop because of the significant leverage CPE activities can have on computer usage. This paper describes a number of conclusions abstracted from the discussions as well as presenting recommendations formally adopted by the participants. While several of these conclusions indicate that improvements are needed in performance analysis tools, another suggests that improved application of CPE could be achieved by better documentation of analysis approaches. More integration of data collection and modeling are considered necessary for the performance analysis field to develop its full potential. Participants noted that the common emphasis on data collection or modeling, to the exclusion of considering objectives, often seriously degrades the value of performance analyses; the only savings that really count from a performance analysis are the ones that appear on the bottom line of the balance sheet.

Outcome Based Systems Evaluation to Assess Information Technology Impact Using Arima Methods

“…1977;Shannon and Weaver, 1949;Sprague and Carlson, 1982]. Boehm and Bell [1977] concluded that both the cost of a system and the information needs of the users must be considered in any information systems evaluation. This idea was later discussed and supported by Chandler [1982] who favored a multiple criteria approach.…”

A Multiple Criteria Approach for Evaluating Information Systems

“…In the computer system domain, performance is measured in terms of resource utilization, cost, and efficiency while in the user domain, throughput, reliability, and response time are common measures. A workshop sponsored by ACM and NBS [1] concluded that any performance analyses "should recognize both the costs of a computer installation and the needs of users for service. "…”