Research Report—Principles for Examining Predictive Validity: The Case of Information Systems Spending Forecasts

Collopy, Fred; Adya, Monica; Armstrong, J. Scott

doi:10.1287/isre.5.2.170

Cited by 41 publications

(35 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A third example is the study by Collopy et al (1994), who compared diffusion models with simpler linear models for forecasting IS spending, and showed the higher predictive power of linear models.…”

Section: Role 3: Comparing Competing Theoriesmentioning

confidence: 99%

See 1 more Smart Citation

Predictive Analytics in Information Systems Research

Shmueli¹,

Koppius²

2011

MIS Quarterly

736

179

View full text Add to dashboard Cite

This research essay highlights the need to integrate predictive analytics into information systems (IS) research, and shows several concrete ways in which this can be accomplished. Predictive analytics include empirical methods (statistical and other) that generate data predictions as well as methods for assessing predictive power. Predictive analytics not only assist in creating practically useful models, they also play an important role alongside explanatory modeling in theory building and theory testing. We describe six roles for predictive analytics: new theory generation, measurement development, comparison of competing theories, improvement of existing models, relevance assessment, and assessment of the predictability of empirical phenomena. Despite the importance of predictive analytics, we find that they are rare in the empirical IS literature. The latter relies nearly exclusively on explanatory statistical modeling, where statistical inference is used to test and evaluate the explanatory power of underlying causal models. However, explanatory power does not imply predictive power and thus predictive analytics are necessary for assessing predictive power and for building empirical models that predict well. To show the distinction between predictive analytics and explanatory statistical modeling, we present differences that arise in the modeling process of each type. These differences translate into different final models, so that a pure explanatory statistical model is best tuned for testing causal hypotheses and a pure empirical predictive model is best in -known explanatory paper on TAM to a predictive context to illustrate these differences and show how predictive analytics can add theoretical and practical value to IS research.

show abstract

Section: Role 3: Comparing Competing Theoriesmentioning

confidence: 99%

“…For example, Collopy et al (1994) showed that adding a price-adjustment predictor to models for IS spending greatly improves predictive power (i.e., reduces prediction error on out-of-sample data). It is worth re-emphasizing that this predictive assessment is fundamentally different from assessing statistical significance.…”

Section: Role 5: Assessing Relevancementioning

confidence: 99%

Predictive Analytics in Information Systems Research

Shmueli¹,

Koppius²

2011

MIS Quarterly

736

179

View full text Add to dashboard Cite

show abstract

“…The accuracy of expert systems could be compared with the accuracy of the method currently used to make forecasts. Collopy et al [6] presents guidelines for making such an evaluation. Other criteria such as ease of use, ease of interpretation, and cost savings, may also enhance the perception of relative advantage.…”

Section: Discussionmentioning

confidence: 99%

“…We drew upon findings from the diffusion literature to assess the prospects for the diffusion of expert systems in forecasting [1][2][3][4][5][6][7][8][9][10].…”

Section: Introductionmentioning

confidence: 99%

Potential Diffusion of Expert Systems in Forecasting

Armstrong¹,

Yokum²

2001

Technological Forecasting and Social Change

View full text Add to dashboard Cite

We drew upon findings from the diffusion literature to assess the prospects for the diffusion of expert systems in forecasting. Forecasters judged potential adoption of expert systems in relation to two techniques that had been widely adopted in the past, Box-Jenkins and scenarios. They also rated each technique on seven innovation characteristics: relative advantage, compatibility, divisibility, communicability, complexity, product risks, and psychological risks. The respondents were classified by four forecaster roles: researcher, educator, practitioner, and decision maker. In general, the expected probabilities of adoption for expert systems were slightly higher than for the two other techniques. Additionally, the respondents rated expert systems nearly equivalent to Box-Jenkins and scenarios on relative advantage and communicability. In relating the probabilities of adoption to the characteristic ratings, the groups perceived significant negative psychological and product risks with expert systems. However the experts, especially practitioners and decision makers, rated expert systems positive on compatibility, divisibility, and communicability, so it may be desirable to ensure that these positive traits are stressed with potential adopters, especially researchers and educators. Potential Diffusion of Expert Systems in Forecasting J. Scott ArmstrongThe Wharton School, University of Pennsylvania Thomas Yokum Angelo State University AbstractWe drew upon findings from the diffusion literature to assess the prospects for the diffusion of expert systems in forecasting. Forecasters judged potential adoption of expert systems in relation to two techniques that had been widely adopted in the past, Box-Jenkins and scenarios. They also rated each technique on seven innovation characteristics: relative advantage, compatibility, divisibility, communicability, complexity, product risks, and psychological risks. The respondents were classified by four forecaster roles: researcher, educator, practitioner, and decision maker. In general, the expected probabilities of adoption for expert systems were slightly higher than for the two other techniques. Additionally, the respondents rated expert systems nearly equivalent to Box-Jenkins and scenarios on relative advantage and communicability. In relating the probabilities of adoption to the characteristic ratings, the groups perceived significant negative psychological and product risks with expert systems. However the experts, especially practitioners and decision makers, rated expert systems positive on compatibility, divisibility, and communicability, so it may be desirable to ensure that these positive traits are stressed with potential adopters, especially researchers and educators.

show abstract

“…To carry out this task fairly, there are various issues which must be considered. In a study by Adya and Collopy [4], guidelines first described in Collopy et al [12] were used to evaluate the effectiveness of the validation component. These required comparisons with well-accepted models, and the use of ex ante validation data based on a reasonable sample of forecasts.…”

Section: The Evaluation Of Neural Network Forecasting Modelsmentioning

confidence: 99%