Weighted Distance-Based Models for Ranking Data Using the <i>R</i> Package <b>rankdist</b>

Qian, Zhaozhi; Yu, Philip L. H.

doi:10.18637/jss.v090.i05

Cited by 9 publications

(7 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Spearman’s footrule is a distance measure that sums up the absolute differences between the ranks of the two lists, taking the value 0 for complete concordance and increasing values for larger discrepancies. Spearman’s distance , in turn, sums up the squared differences between the ranks of the two lists 20 , which in this context is closely related to the Euclidean distance.…”

Section: Data and Data Processingmentioning

confidence: 99%

Methods and open-source toolkit for analyzing and visualizing challenge results

Wiesenfarth

Reinke

Landman

et al. 2021

Sci Rep

View full text Add to dashboard Cite

Grand challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) we present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) we release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed in the specific context of biomedical image analysis challenges. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.

show abstract

Section: Data and Data Processingmentioning

confidence: 99%

Methods and open-source toolkit for analyzing and visualizing challenge results

Wiesenfarth

Reinke

Landman

et al. 2021

Sci Rep

View full text Add to dashboard Cite

show abstract

“…Spearman's footrule is a distance measure that sums up the absolute differences between the ranks of the two lists, taking the value 0 for complete concordance and increasing values for larger discrepancies. Spearman's distance, in turn, sums up the squared differences between the ranks of the two lists [8], which in this context is closely related to the Euclidean distance.…”

Section: B Ranking Methodsmentioning

confidence: 99%

Methods and open-source toolkit for analyzing and visualizing challenge results

Wiesenfarth¹,

Reinke²,

Landman³

et al. 2019

Preprint

View full text Add to dashboard Cite

Biomedical challenges have become the de facto standard for benchmarking biomedical image analysis algorithms. While the number of challenges is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) We present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) We release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed within this work. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.

show abstract

“…The rankdist R package implements mixtures of the standard Mallows model for complete or subset rankings via maximum likelihood (Qian and Yu, 2019). The BayesMallow R package (Vitelli et al, 2018) implements mixtures of This has the advantage of facilitating inference on the global ranking (in each sub-population), e.g.…”

Section: Mallows Distance-based Modelmentioning

confidence: 99%

Modelling rankings in R: the PlackettLuce package

et al. 2020

View full text Add to dashboard Cite

This paper presents the R package PlackettLuce, which implements a generalization of the Plackett-Luce model for rankings data. The generalization accommodates both ties (of arbitrary order) and partial rankings (complete rankings of subsets of items). By default, the implementation adds a set of pseudo-comparisons with a hypothetical item, ensuring that the underlying network of wins and losses between items is always strongly connected. In this way, the worth of each item always has a finite maximum likelihood estimate, with finite standard error. The use of pseudo-comparisons also has a regularization effect, shrinking the estimated parameters towards equal item worth. In addition to standard methods for model summary, PlackettLuce provides a method to compute quasi standard errors for the item parameters. This provides the basis for comparison intervals that do not change with the choice of identifiability constraint placed on the item parameters. Finally, the package provides a method for model-based partitioning using covariates whose values vary between rankings, enabling the identification of subgroups of judges or settings that have different item worths. The features of the package are demonstrated through application to classic and novel data sets.where A j is the set of alternatives {i j , i j+1 , . . . , i J } from which item i j is chosen. The above model is also derived in Plackett (1975), hence the name Plackett-Luce model.In this paper, we present the R package PlackettLuce . This package implements an extension of the Plackett-Luce model that allows for ties in the rankings. The model can be applied to either complete or partial rankings (complete rankings of subsets of items). PlackettLuce offers a choice of algorithms to fit the model via maximum likelihood. Pseudo-rankings, i.e. pairwise comparisons with a hypothetical item, are used to ensure that the item worths always have finite maximum likelihood estimates (MLEs) with finite standard error. Methods are provided to obtain different parameterizations with corresponding standard errors or quasi-standard errors (that do not change with the identifiability constraint). There is also a method to fit Plackett-Luce trees, which partition the rankings by covariate values to identify subgroups with distinct Plackett-Luce models.In the next section, we review the available software for modelling rankings and make comparisons with Placket-tLuce. Then in Section 3 we describe the Plackett-Luce model with ties and the methods implemented in the package for model-fitting and inference. Plackett-Luce trees are introduced in Section 4, before we conclude the paper with a discussion in Section 5.2 Software for modelling rankings First, in Section 2.1, we consider software to fit the standard Plackett-Luce model. Then in Section 2.2, we review the key packages in terms of their support for features beyond fitting the standard model. Finally in Section 2.3 we review software implementing alternative models for rankings data and discuss how these approaches ...

show abstract

Weighted Distance-Based Models for Ranking Data Using the R Package rankdist

Cited by 9 publications

References 17 publications

Methods and open-source toolkit for analyzing and visualizing challenge results

Methods and open-source toolkit for analyzing and visualizing challenge results

Methods and open-source toolkit for analyzing and visualizing challenge results

Modelling rankings in R: the PlackettLuce package

Contact Info

Product

Resources

About