Ratings and Rankings: Voodoo or Science?

Paruolo, Paolo; Saisana, Michaela; Saltelli, Andrea

doi:10.1111/j.1467-985x.2012.01059.x

Cited by 246 publications

(177 citation statements)

References 34 publications

Supporting

Mentioning

148

Contrasting

Unclassified

Order By: Relevance

“…Our aim was to select only methods which are simple, easily understandable and only data driven. Used methods cover two main issues during constructing a composite indicator -correlation and compensability between various indicators (Paruolo et al, 2013). The differences in results as well as suitability of each CI are discussed in the next section.…”

Section: Methodsology Of Composite Indicatorsmentioning

confidence: 99%

Measuring Sustainable Development at the Lower Regional Level in the Czech Republic based on Composite Indicators

Hudrlíková¹,

Kramulová²,

Zeman³

2013

Reg. Stat.

View full text Add to dashboard Cite

Measuring of sustainable development seems to be a great issue as there is neither unified set of indicators nor any preferred methodology how to do it. However, attempts to evaluate entities from the point of view of sustainable development regularly occur. The most problematic level according to sustainable development assessment seems to be "lower" regional level, such as LAU 1 (former NUTS 4) level. On one hand there are usually at this level already serious problems with data availability, on the other hand it is almost impossible to regularly perform detailed questionnaire surveys in all LAU 1 regions (77 districts in case of the Czech Republic), as it is done e.g. in cities. The aim of the paper is to decide, how to assess sustainability at this level. Finally we chose useful indicators, though different from indicators used at the national or NUTS 3 level, with data available for all LAU 1 regions. We succeeded in filling all three pillars of sustainable development (economic, social and environmental) with sufficient number of suitable indicators. In the first step we applied cluster analysis to find coherences among regions that are e.g. affected by similar problems. After that we constructed composite indicators in order to create ranking of all 77 districts. Ranking was derived from composite indicator approach. We constructed 10 composite indicators to test different methods of normalisation, weighting and aggregation. The results show the ranking of LAU 1 regions in the Czech Republic from the sustainability point of view, both including and excluding capital city of Prague as an outlying district. A good interconnection between cluster analysis and constructed composite indicators can be seen, which is supported by the discussion of the results.

show abstract

Section: Methodsology Of Composite Indicatorsmentioning

confidence: 99%

Measuring Sustainable Development at the Lower Regional Level in the Czech Republic based on Composite Indicators

Hudrlíková¹,

Kramulová²,

Zeman³

2013

Reg. Stat.

View full text Add to dashboard Cite

show abstract

“…University rankings are basically composite indicators, where weighted arithmetic averages are linearly aggregated, with the purpose of measuring university performance (Paruolo et al, 2013). As for all composite indicators, university rankings are subjected to normative assumptions about the type of variables and associated weightings.…”

Section: Methodsological Criticismmentioning

confidence: 99%

“…Nonetheless, there is a body of literature highlighting the methodological problems of rankings (Marginson & van der Wende, 2007;Paruolo, Saisana, & Saltelli, 2013;Saisana & d'Hombres, 2008;Saisana, d'Hombres, & Saltelli, 2011;Soh, 2013aSoh, , 2015van Raan, 2005) and the concerns related to the implications in terms of institutional design and unintended consequences for national higher education systems (Aust & Musselin, 2014;Hazelkorn, 2014;Kehm, 2014;Sauder & Espeland, 2009).…”

Section: Introductionmentioning

confidence: 99%

One size fits all? A different perspective on university rankings

Goglio¹

2016

Journal of Higher Education Policy and Management

View full text Add to dashboard Cite

In roughly a decade, university rankings gained the foreground in the policy arena for higher education and their influence is not going to decrease. However, several methodological shortcomings and warnings about the unintended consequences for national higher education systems have been raised. Against this background, this paper stresses that the individual recipients of information contained in university rankings are currently overlooked. Indeed, university rankings are addressed to a generic recipient, but actually, there are multiple audiences for rankings, and each of these audiences has different needs and each one attributes a different value to information attached to rankings. Referring to a theoretical tool borrowed from bioethics, this paper highlights that the ranking game involves a variety of recipients and that the current setting of the ranking panorama leaves room for gaps to emerge.

show abstract

“…Furthermore, PCA is sensitive to outlier presence and to modifications in the basic data. PCA derived weights are difficult to interpret and to communicate, and as a result the use of PCA in this context is not widespread [16].…”

Section: Composite Indicatorsmentioning

confidence: 99%

“…This perception can be grossly off the mark. Nominal weights are not a measure of variable importance and the relative importance of variables depends on the characteristics of their distribution as well as their correlation structure [16]. An overview of different weighting methods (and details viz.…”

Section: Composite Indicatorsmentioning

confidence: 99%

Data-driven weights and restrictions in the construction of composite indicators

Perišić

2015

Croat. Oper. Res. Rev.

View full text Add to dashboard Cite

Abstract. Composite indicators are increasingly recognized as a useful tool in policy analysis and public communication. However, if poorly constructed, they can send misleading policy messages. Perhaps the most difficult aspect of constructing a composite indicator is choosing weights for the components. The categorization of Croatian territorial units for development policy is based on the value of the composite indicator called the development index. The main goal of this paper is to propose an empirical approach for weight selection. In order to generate the set of non-subjective weights, principal component analysis and linear programming methods have been applied. An application of data envelopment analysis to the field of composite indicators, known as the Benefit-of-the-Doubt approach, has been demonstrated subject to proportional subindicator share restrictions. Additionally, the Monte Carlo simulation of weights was conducted, and confidence intervals for the values of the development index were estimated. Owing to the fact that the examined weighting schemes have resulted in the different categorization of territorial units, use of unit-specific weights and incorporating uncertainty in the construction of a composite indicator looks promising for further work.

show abstract

Ratings and Rankings: Voodoo or Science?

Cited by 246 publications

References 34 publications

Measuring Sustainable Development at the Lower Regional Level in the Czech Republic based on Composite Indicators

Measuring Sustainable Development at the Lower Regional Level in the Czech Republic based on Composite Indicators

One size fits all? A different perspective on university rankings

Data-driven weights and restrictions in the construction of composite indicators

Contact Info

Product

Resources

About