Does choice of multicriteria method matter? An experiment in water resources planning

Abstract

Many multiple criteria decision making methods have been proposed and applied to water planning. Their purpose is to provide information on tradeoffs among objectives and to help users articulate value judgments in a systematic, coherent, and documentable manner. The wide variety of available techniques confuses potential users, causing inappropriate matching of methods with problems. Experiments in which water planners apply more than one multicriteria procedure to realistic problems can help dispel this confusion by testing method appropriateness, ease of use, and validity. We summarize one such experiment where U.S. Army Corps of Engineers personnel used several methods to screen urban water supply plans. The methods evaluated include goal programming, ELECTRE I, additive value functions, multiplicative utility functions, and three techniques for choosing weights (direct rating, indifference tradeoff, and the analytical hierarchy process). Among the conclusions we reach are the following. First, experienced planners generally prefer simpler, more transparent methods. Additive value functions are favored. Yet none of the methods are endorsed by a majority of the participants; many preferred to use no formal method at all. Second, there is strong evidence that rating, the most commonly applied weight selection method, is likely to lead to weights that fail to represent the trade‐offs that users are willing to make among criteria. Finally, we show that decisions can be as or more sensitive to the method used as to which person applies it. Therefore, if who chooses is important, then so too is how a choice is made.

Multi-criteria decision analysis with goal programming in engineering, management and social sciences: a state-of-the art review

“…Studies in the assessment of GP techniques area are to a largest extent methodological and try to identify the pros and cons of different GP approaches to assess environmental policies. For example, Hobbs et al (1992) consider different WGP models to evaluate the level of understanding of multicriteria methods among water planners from the US Army Corps of Engineers. A similar experiment based on the same case study can be found in Goicoechea et al (1992).…”

Multicriterion decision analysis approach to assess the utility of watershed modeling for management decisions

“…The AVF, which simply can be perceived as the normalized weighted summation of all criteria scores with respect to each alternative, is an easy to understand method. It is well perceived by water resources planners [Hobbs et al, 1992] and has been used by other researchers in water resources literature [Qureshi and Harrison, 2001;Huth et al, 2004;Fassio et al, 2005]. ELECTRE II was selected because it is conceptually different from the AVF in the way it solves the payoff matrix and ranks different alternatives.…”

“…Variants of ELECTRE have been successfully used in water resources literature [Tecle et al, 1988;Hobbs et al, 1992;Roy et al, 1992;Raju and Duckstein, 2004]. ELECTRE II is a variant of ELECTRE family that produces a ranking of alternatives rather than indicating the most preferred.…”

“…[19] Hobbs et al [1992] posed and addressed the question of how various MCDA methods differ from one another. On the basis of their experiment, it was concluded that it is desirable to apply more than one method as a check.…”

Strategic decision-making for water resource management in semi-arid metropolitan and rural areas

“…• MCDA improves the justification of decisions It must be noted that MCDA does not claim to provide a "correct" or "true" system of weights or scores, as these are determined by the inputs of the stakeholders of the decision-making process (Hobbs et al, [18]; Stewart et al, [34,35]). The "correct" system reflects the trade-offs society is willing to make in any specific situation.…”