Outcome Based Systems Evaluation to Assess Information Technology Impact Using Arima Methods

Abstract

A new method of system evaluation that focuses on the impact the system has on a data series that served as the rationale for systems implementation was designed and modeled by the authors. Called outcome-based evaluation, this method is founded on the concept of intervention analysis and employs interrupted time series designs to determine the impact of an information system on specific organizational goals. Based on a review of the literature on the evolution of systems evaluation methods from focusing on user goals to user satisfaction and system usage, we conclude that user satisfaction and system usage are necessary but not sufficient criteria to establish system effectiveness or success. Thus, we establish a need for the proposed new method of system evaluation. Three business case studies are presented in this article that demonstrate and validate an evaluation method using ARIMA models for the analysis. The value of this tool for managers is its means of assessing IT effectiveness and payoff contextually, thereby enabling businesses to clarify both their IT needs and their outcome expectations a priori.