“…Digraph III Digraph IV In ranking applications, the data is often assumed to be totally rankable, i.e., observations are some noisy evaluations of an objective function that reflects the true unique ranking. Recently, however, some in the ranking community have challenged this assumption; specifically, in [1,4] quantitative metrics are proposed for measuring the rankability of data, and in [8] general preference modeling methods are developed without assuming total rankability. Since the linear ordering problem results in a ranking that is optimal with respect to the objective function in (1a), it is natural to consider what properties of the LOP can be used to analyze the rankability of the underlying data.…”
Section: Lomentioning
confidence: 99%
“…In many ranking applications, the data is often assumed to be totally rankable, i.e., observations are some noisy evaluations of an objective function that reflects the true unique ranking [8]. In [1], the authors questioned this assumption by proposing the concept of rankability, which refers to a dataset's inherent ability to be meaningfully ranked.…”
In 2019, Anderson et al. proposed the concept of rankability, which refers to a dataset's inherent ability to be meaningfully ranked. In this article, we give an expository review of the linear ordering problem (LOP) and then use it to analyze the rankability of data. Specifically, the degree of linearity is used to quantify what percentage of the data aligns with an optimal ranking. In a sports context, this is analogous to the number of games that a ranking can correctly predict in hindsight. In fact, under the appropriate objective function, we show that the optimal rankings computed via the LOP maximize the hindsight accuracy of a ranking. Moreover, we develop a binary program to compute the maximal Kendall tau ranking distance between two optimal rankings, which can be used to measure the diversity among optimal rankings without having to enumerate all optima. Finally, we provide several examples from the world of sports and college rankings to illustrate these concepts and demonstrate our results.
“…Digraph III Digraph IV In ranking applications, the data is often assumed to be totally rankable, i.e., observations are some noisy evaluations of an objective function that reflects the true unique ranking. Recently, however, some in the ranking community have challenged this assumption; specifically, in [1,4] quantitative metrics are proposed for measuring the rankability of data, and in [8] general preference modeling methods are developed without assuming total rankability. Since the linear ordering problem results in a ranking that is optimal with respect to the objective function in (1a), it is natural to consider what properties of the LOP can be used to analyze the rankability of the underlying data.…”
Section: Lomentioning
confidence: 99%
“…In many ranking applications, the data is often assumed to be totally rankable, i.e., observations are some noisy evaluations of an objective function that reflects the true unique ranking [8]. In [1], the authors questioned this assumption by proposing the concept of rankability, which refers to a dataset's inherent ability to be meaningfully ranked.…”
In 2019, Anderson et al. proposed the concept of rankability, which refers to a dataset's inherent ability to be meaningfully ranked. In this article, we give an expository review of the linear ordering problem (LOP) and then use it to analyze the rankability of data. Specifically, the degree of linearity is used to quantify what percentage of the data aligns with an optimal ranking. In a sports context, this is analogous to the number of games that a ranking can correctly predict in hindsight. In fact, under the appropriate objective function, we show that the optimal rankings computed via the LOP maximize the hindsight accuracy of a ranking. Moreover, we develop a binary program to compute the maximal Kendall tau ranking distance between two optimal rankings, which can be used to measure the diversity among optimal rankings without having to enumerate all optima. Finally, we provide several examples from the world of sports and college rankings to illustrate these concepts and demonstrate our results.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.