The Evaluation of Research Papers (Or How to Get an Academic Committee to Agree on Something)

Abstract

The College of Commerce and Finance at Villanova University sponsors an annual research aw^ards program. Its objectives are to encourage and reward faculty research efforts. The process of selecting papers to win the award turned out to be far more complex than we initially expected. After many unsuccessful attempts at developing a manageable procedure, we used the analytic hierarchy process to structure the selection process. We have used this approach for the last three years without any serious problems. The faculty and external advisors using the system believe that it can better capture the subjective judgments of the evaluators than any of the previous approaches.

Multi-criteria evaluation approach to GIS-based land-suitability classification for tilapia farming in Bangladesh

“…Saaty (1980) suggests that if the CI/RI is smaller than 0.10, the degree of consistency is satisfactory; if, however, the CI/RI is larger than 0.10, there are inconsistencies and the AHP method may not yield meaningful results. Liberatore et al (1992) suggested the structured approach of AHP could better capture the subjective judgments. Millet (1998) found that the process could enhance shared understanding of decisions, save time, and facilitate consensus.…”

Supplier selection in electronic marketplaces using satisficing and fuzzy AHP

“…The FPP method is proposed in the literature to face the inconsistency problems inherent in the Analytic Hierarchy Process. Following Liberatore's work (Liberatore, 1987;Liberatore, Nydick, & Sanchez, 1992), we introduce a modified version of the FPP method by employing the rating scale used in the pre-qualification stage in order to overcome the explosion in the number of pairwise comparisons when the number of alternatives and/or the number of sub-criteria is large. Without loss of generality, our proposal utilizes a rating scale with five levels.…”

“…The use of a five-grade rating scale to map different evaluation standards for qualitative criteria and absolute values for quantitative criteria has been employed in various problems with success and can be found in various studies (Lee, Lee, & Kang, 2005;Liberatore, 1987;Liberatore et al, 1992;Singh, Murty, Gupta, & Dikshit, 2007;Tam & Rao Tummala, 2001), albeit in the context of the AHP method. An example is shown in Table 1 for one qualitative and one quantitative criterion.…”

“…Thus, the comparison matrices are constructed through pairwise comparisons among the rating levels for each sub-criterion. The use of a rating scale instead of direct comparisons among the alternatives was introduced by Liberatore and can be found in various studies (Lee et al, 2005;Liberatore, 1987;Liberatore et al, 1992;Singh et al, 2007;Tam & Rao Tummala, 2001).…”

“…For example, the relative rating of an ''outstanding" versus a ''good" rating may differ for different criteria. As stated by Liberatore et al (1992), making such fine discriminations in judgment would be very difficult. Furthermore, we want to keep the assessment process as simple as possible.…”