Rank disagreement: A comparison of multi‐criteria methodologies

Abstract

A number of multi‐criteria decision support techniques have emerged in recent years that use varying computational approaches to arrive at the most desirable solution and thereby ‘recommend’ a course of action. Decision makers who use the results of this analytic work should be assured that the computational schemes used by their supporting analysts or decision support software produce the appropriate solutions. We conducted a series of simulation experiments that compared the top‐ranked options resulting from the computational algorithms that support Multi‐Attribute Value Theory (MAVT) and three methods that are reported in the literature that allow rank reversals, the change in rank order of two options when an unrelated option is added or deleted from the analysis: the Analytical Hierarchy Process (AHP), Percentaging and the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). We also included a Fuzzy algorithm proposed by Yager to gauge its consistency with the other algorithms, even though it is not subject to rank reversals. These experiments demonstrated that the MAVT and AHP techniques, when provided with the same decision outcome data, very often identify the same alternative as ‘best’. The other techniques are noticeably less consistent with MAVT, the Fuzzy algorithm being the least consistent. The situations under which the most frequent and significant differences occurred were dependent upon the method. The results of our experiments indicate that other issues (e.g. the processes used for problem structuring and the elicitation of value weights) are likely to be of greater significance to problem outcome (based on our experience) than the choice between the computational algorithms of MAVT and AHP. The results cause us to be concerned about the use of the other methods.

How to Build up the Loan - Evaluation System toward Small and Medium Enterprises between Taiwan and China’s Commercial Banks？ The Application for Multi Criteria Decision Making

“…The decision makers can decide which replaceable method has possibility or the priority to operate cases. Buede and Maxwell (1995) indicate that original assessment criteria are probable quality analysis or quantitative analysis, but using MCDM still needs to transform non-quantitative criteria before proceeding analysis. In conclusion, the final case decision is from assessing individual criterion between each other or comparing same assessment criterion between individual cases.…”

The Pugh Controlled Convergence method: model-based evaluation and implications for design theory

“…Procedures such as the Analytical Hierarchy Process are meant to take the raw data of pairwise comparison and to create interval scale measurements. In the process, inconsistencies or rank disagreements may be discovered (Buede and Maxwell 1995) and procedures have been suggested for correcting those (Limayem and Yannou 2007). Even if such inconsistencies are not present, there is still substantial uncertainty in rankings due to uncertainties in the pairwise comparisons and there exist methods for quantifying these uncertainties (Scott 2007).…”

Considerations Regarding the Choice of Ranking Multiple Criteria Decision Making Methods

“…Buede & Maxwell (1995) have already identified problems related to final ranking inconsistency in some well-known multiple criteria methods. According to the authors, although these methods have been developed based on a different number of theories and algorithms, the decision is always made considering the preferences on a set of weighting criteria (Buede & Maxwell, 1995).…”

“…The problem is further increased by the knowledge that solution of MCDM ranking methods may be sensitive to slight variations in entrance data, e.g. small changes in the weighting vector based on the decision maker's preferences, or to the computational algorithm employed (Yoon & Hwang, 1995; Buede & Maxwell, 1995;Zanakis et al, 1998;Yeh, 2002;Maxwell, 1995). There is also the fact that MCDM ranking methods, in some cases, might replace the best alternative for the worst when the weightings for the criteria are changed (Tallarico, 1990;Wang & Triantaphyllou, 2008;Brunner & Starkl, 2004).…”

“…For example, Mareschal (1988) defines stability intervals for the weightings of the different criteria in order to study the stability of the results generated by the PROMETHEE method. Buede & Maxwell (1995) used the Monte Carlo approach to study MCDM inconsistency in the Analytic Hierarchy Process (AHP) and TOPSIS methods. Zanakis et al (1998) verified the ranking inconsistency of ELEC-TRE, TOPSIS, SAW, and four versions of AHP, also using the Monte Carlo approach based on twelve consistency measures.…”