Assessing DSS effectiveness using evaluation research methods

Evans, Gerald E.; Riha, James

doi:10.1016/0378-7206(89)90037-2

Cited by 25 publications

(9 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both user evaluations and analysis of method results are necessary to evaluate decision support systems (Evans and Riha, 1989;Gunderson et al, 1995;Hobbs et al, 1992). Therefore, method performance, appropriateness, and ease of use were assessed through participant feedback (questionnaires and structured discussions), while method results and validity were compared by statistical analysis of weights and policy rankings.…”

Section: Introductionmentioning

confidence: 99%

An evaluation of multi‐criteria methods in integrated assessment of climate policy

Bell

Hobbs

Elliott

et al. 2001

Multi Criteria Decision Anal

View full text Add to dashboard Cite

Those who conduct integrated assessments (IAs) are aware of the need to explicitly consider multiple criteria and uncertainties when evaluating policies for preventing global warming. MCDM methods are potentially useful for understanding tradeoffs and evaluating risks associated with climate policy alternatives. A difficulty facing potential MCDM users is the wide range of different techniques that have been proposed, each with distinct advantages. Methods differ in terms of validity, ease of use, and appropriateness to the problem. Alternative methods also can yield strikingly different rankings of alternatives. A workshop was held in which climate change experts and policymakers evaluated the usefulness of MCDM for IA. Participants applied several methods in the context of a hypothetical greenhouse gas policy decision. Methods compared include value and utility functions, goal programming, ELECTRE, fuzzy sets, stochastic dominance, min max regret, and several weight selection methods. Ranges, rather than point estimates, were provided for some questions to incorporate imprecision regarding weights. Additionally, several visualization methods for both deterministic and uncertain cases were used and evaluated. Analysis of method results and participant feedback through questionnaires and discussion provide the basis for conclusions regarding the use of MCDM methods for climate change policy and IA analyses. Hypotheses are examined concerning predictive and convergent validity of methods, existence of splitting bias among experts, perceived ability of methods to aid decision-making, and whether expressing imprecision can change ranking results. Because participants gained from viewing a problem from several perspectives and results from different methods often significantly differed, it appears worthwhile to apply several MCDM methods to increase user confidence and insight. The participants themselves recommended such multimethod approaches for policymaking. Yet they preferred the freedom of unaided decision-making most of all, challenging the MCDM community to create transparent methods that permit maximum user control.

show abstract

Section: Introductionmentioning

confidence: 99%

An evaluation of multi‐criteria methods in integrated assessment of climate policy

Bell

Hobbs

Elliott

et al. 2001

Multi Criteria Decision Anal

View full text Add to dashboard Cite

show abstract

Section: Evaluation Of Mcdamentioning

confidence: 99%

“…Evaluation methods and questions asked were similar to those used in other MCDA and decision support comparisons and evaluations (Evans and Riha, 1989;Hobbs et al, 1992;Gunderson et al, 1994;Zapatero et al, 1997;Qureshi et al, 1999;Bell et al, 2001;Lai et al, 2002). Participants were asked for their opinions on: the suitability of MCDA for the decision; the importance of some of the benefits of MCDA methods; how well the process met what they considered important in evaluating their decision and any other benefits or drawbacks they identified; their perceptions of each stage of the process (e.g.…”

Section: Evaluation Of Mcdamentioning

confidence: 99%

Application of multiple criteria decision analysis in the New Zealand agricultural industry

Dooley

Smeaton

Sheath

et al. 2009

Multi Criteria Decision Anal

View full text Add to dashboard Cite

Agricultural decisions typically involve multiple criteria, some of which are subjective. Business, environmental, and lifestyle criteria are all important criteria in these decisions. These criteria can be difficult to trade-off using traditional methods. Multiple criteria decision analysis (MCDA) provides a formal, quantitative means of evaluating agricultural decisions taking all these factors into account. MCDA for agricultural decisions was evaluated using three applied case studies (use of new processing technology, selection of beef policies, and selection of farm systems in an environmentally sensitive catchment). The case studies all differed in their problem types and decision-maker requirements. A multi-attribute value theory approach was used for all the cases. This approach was selected using a descriptive framework which took the method limitations, problem attributes, and decision-maker requirements into account. The differences between the case studies, the application and the implementation of MCDA, the overall success of the process, and the potential use of MCDA in agriculture going forward are discussed. While MCDA was used to help identify the best decision, the main benefits that the decision makers identified included: learning about the decision, a better understanding of their own and others' perspectives, a structured way to work through the decision, a means to explain the decision, and stimulation of discussion and sharing of ideas. These benefits were particularly important for the group decisions. Participants were not overly concerned with the ranking accuracy. Problems in implementation included an initial lack of commitment to the process, understanding of the process and decision, and ownership of the decision. The limited time decision makers had available contributed to this. The majority of the decision makers liked the MCDA process for these strategic decisions. The quantitative approach and the graphic presentation appealed to them. Hence, simple MCDA approaches may be as effective as more complex ones, and can deliver many of the benefits particularly where the time is limited. There has been interest in the use of MCDA going forward.

show abstract

“…Thus, the use of computer-based support systems should make the strategy development process more effective. In this study, the effectiveness is measured in terms of performance of decision activity (Keen and Scott Morton, 1978), decision confidence (Turban, 1995;Oz et al, 1993;Davey and Olson, 1998;Van Bruggen et al, 1996), level of consensus (DeSanctis and Gallupe, 1985;Turban, 1995;Sharda et al, 1988), quality of outputs of a system (Evans and Riha, 1989), quality of decisions (Keen and Scott Morton, 1978;Coll et al, 1991), etc. With regard to the provision of computerised support, the measurements for the effectiveness should also include helping strategic thinking (Porter, 1987;Mintzberg, 1994aMintzberg, , 1994b and coupling strategic analysis with managerial judgement (Mintzberg, 1994a(Mintzberg, , 1994b(Mintzberg, , 1994c, etc.…”

Section: Evaluation Questionnairesmentioning

confidence: 99%

Integrating group Delphi, fuzzy logic and expert systems for marketing strategy development: the hybridisation and its effectiveness

Davies²,

Edwards

et al. 2002

Marketing Intelligence &Amp; Planning

View full text Add to dashboard Cite

A hybrid approach for integrating group Delphi, fuzzy logic and expert systems for developing marketing strategies is proposed in this paper. Within this approach, the group Delphi method is employed to help groups of managers undertake SWOT analysis. Fuzzy logic is applied to fuzzify the results of SWOT analysis. Expert systems are utilised to formulate marketing strategies based upon the fuzzified strategic inputs. In addition, guidelines are also provided to help users link the hybrid approach with managerial judgement and intuition. The effectiveness of the hybrid approach has been validated with MBA & MA marketing students. It is concluded that the hybrid approach is more effective in terms of decision confidence, decision quality, group consensus, coupling analysis with judgement, etc.

show abstract

Assessing DSS effectiveness using evaluation research methods

Cited by 25 publications

References 7 publications

An evaluation of multi‐criteria methods in integrated assessment of climate policy

An evaluation of multi‐criteria methods in integrated assessment of climate policy

Application of multiple criteria decision analysis in the New Zealand agricultural industry

Integrating group Delphi, fuzzy logic and expert systems for marketing strategy development: the hybridisation and its effectiveness

Contact Info

Product

Resources

About