Ratings and rankings: reconsidering the structure of values and their measurement

Abstract

His other recent publications on values have appeared in Social Science Quarterly ('Race, class and gender differences in high school seniors' values ', Vol. 82) and Sociological Perspectives ('Suggestions of the postmodern self ', Vol. 46). He is also active in research on urban racial inequality in the United States.Despite extensive discussion and analysis over the past three decades, researchers are still divided over the preferability of rating or ranking systems for measuring values. However, the debate originates from the false premise that one method must be used to the exclusion of the other. A conceptualization of the value structure that uses characteristics of both rating and ranking systems opens up theory and research to a more complex understanding of values. Data from the 1995-1997 World Values Survey demonstrate how using both rating and ranking systems can lead to distinct and equally valid conclusions about the differences in value importance among the nations of the world. This suggests that either method by itself provides an incomplete understanding of the value structure. While the potential methodological complications of using both ratings and rankings must be explored, the additional cost would be a productive investment for moving beyond the rating-ranking impasse.One way that research on values differs substantially from research on many other social science topics is the inexact definition of the object of analysis. While quantitative items such as wages and fertility are (largely) agreed upon in their definition and how they are to be measured, values do not have a single, unified definition nor a universal model of measurement. The choices that researchers make in defining what a value is not only determine the nature of the item of analysis (the value structure) but also the method of measurement (ratings or rankings). Therefore, a meta-analysis of values research must address two related questions. First, how are values structured in an individual's mind? Second, how does one measure a value?Historically, values research and theory have focused on the measurement question much more extensively than-and perhaps even to the neglect of-the structure question. The debate over the nature of values has most often been engaged on methodological grounds, specifically whether values are best measured with ranking or rating systems. This debate remains unresolved 30 years after it first became prominent in social science research. While there have been intermittent contributions to the literature

Insight into the relative merits of rating and ranking in a cross-national context using three-way correspondence analysis

“…This study provides insight into the cross-national applicability of rating and ranking procedures. Moreover, with this study we respond to the call for further research on this subject made by Ovadia (2004), who suggested comparing a full ranking of values with a rating of the same values.…”

“…In addition, some studies report higher test-retest reliabilities for ranking (Johnson, Sallis, & Hovell, 1999;Rankin & Grube, 1980), and others for rating (Barylko-Pikielna et al, 2004). The ongoing debate concerning the choice between rating and ranking was characterized by Ovadia (2004) as the rating-ranking impasse. To escape this impasse, we shall approach the problem from a different angle.…”

“…Schwartz, 1992;Villanueva, Petenate, & Da Silva, 2005). However, respondent discrimination between products (Park, O'Mahony, & Kim, 2007;Rodrigue et al, 2000) or values (Kamakura & Mazzon, 1991;Ovadia, 2004) is the common aim of all these studies.…”

“…In particular, when dealing with values there is an ongoing debate about the choice for either rating or ranking for its measurement. Ovadia (2004) refers to the current situation as a rating-ranking impasse.…”

Multimethod approach to measuring values in a school context: exploring the association between Congruence — Discrepancy Index (CO DI) and task commitment

“…Some studies were aimed at assessing whether ranking or rating scales produce similar results and at determining which of the scales is preferable (Maio, Roese, Seligman & Katz, 1996), while others pointed out alternatives (McCarty & Shrum, 2000). Discussion of the studies resulted in the argument that each method might represent personal value systems in a specific way (Ovadia, 2004).…”