Tracking with (Un)Certainty

Hofman, Abe D.; Brinkhuis, Matthieu; Bolsinova, Maria; Klaiber, Jonathan; Maris, Gunter; Maas, Han L. J. van der

doi:10.3390/jintelligence8010010

Cited by 4 publications

(7 citation statements)

References 33 publications

(39 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our third application is to the data of an online learning system Math Garden (Hofman et al., 2020; Klinkenberg et al., 2011), which is a platform for practicing primary school mathematics used widely in the Netherlands. Learners can practice various mathematics skills such as addition, fractions and multiplication using series of exercises that are tailored to the level of their skill.…”

Section: Resultsmentioning

confidence: 99%

“…This is not an innocent activity, as it can potentially change the invariant distribution of the ratings. This is a general phenomenon which seems to have received little attention in other works on rating systems (see Hofman et al., 2020, p. 12). In the context of the urnings, we have that if the choice of players is dependent on the current urnings, then the distribution of the updated urnings is no longer equal to the desired invariant distribution:

p false(r^{*} false) = \underset{boldr}{false\sum} \underset{\overset{r}{true}}{false\sum} \underset{i}{false\sum} \underset{j > i}{false\sum} p_{Z} false(r^{*} false| \overset{r}{true}, i, j, boldr false) p_{Y} false(\overset{r}{true} false| i, j, boldr false) p_{X} false(i, j false| boldr false) p_{bold-italicπ} false(boldr false) \neq p_{bold-italicπ} false(r^{*} false),

where the probability of selecting players i and j depends on the current urnings.…”

Section: Methodsmentioning

confidence: 95%

See 1 more Smart Citation

Urnings: A New Method for Tracking Dynamically Changing Parameters in Paired Comparison Systems

Bolsinova

Maris

Hofman

et al. 2022

Journal of the Royal Statistical Society Series C: Applied Statistics

Self Cite

View full text Add to dashboard Cite

We introduce a new rating system for tracking the development of parameters based on a stream of observations that can be viewed as paired comparisons. Rating systems are applied in competitive games, adaptive learning systems and platforms for product and service reviews. We model each observation as an outcome of a game of chance that depends on the parameters of interest (e.g. the outcome of a chess game depends on the abilities of the two players). Determining the probabilities of the different game outcomes is conceptualized as an urn problem, where a rating is represented by a probability (i.e. proportion of balls in the urn). This setup allows for evaluating the standard errors of the ratings and performing statistical inferences about the development of, and relations between, parameters. Theoretical properties of the system in terms of the invariant distributions of the ratings and their convergence are derived. The properties of the rating system are illustrated with simulated examples and its potential for answering research questions is illustrated using data from competitive chess, a movie review system, and an adaptive learning system for math.

show abstract

Section: Resultsmentioning

confidence: 99%

p false(r^{*} false) = \underset{boldr}{false\sum} \underset{\overset{r}{true}}{false\sum} \underset{i}{false\sum} \underset{j > i}{false\sum} p_{Z} false(r^{*} false| \overset{r}{true}, i, j, boldr false) p_{Y} false(\overset{r}{true} false| i, j, boldr false) p_{X} false(i, j false| boldr false) p_{bold-italicπ} false(boldr false) \neq p_{bold-italicπ} false(r^{*} false),

where the probability of selecting players i and j depends on the current urnings.…”

Section: Methodsmentioning

confidence: 95%

Urnings: A New Method for Tracking Dynamically Changing Parameters in Paired Comparison Systems

Bolsinova

Maris

Hofman

et al. 2022

Journal of the Royal Statistical Society Series C: Applied Statistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Typically, the items are selected based on the current ratings of the learner and the items in the system. Bolsinova et al (2022) and Hofman et al (2020) demonstrated that not correcting for the adaptive item selection can have detrimental consequences for the ratings. If the difficulty of selected items is matched to the learner's ability, then the variance of the ratings will artificially increase.…”

Section: Adaptive Item Selectionmentioning

confidence: 99%

“…Another issue with these rating systems is that the adaptive item selection potentially influences the invariant distribution and has to be corrected for, as described by Hofman et al . (2020, p. 13), which to our knowledge has not been implemented in the rating systems presented above.…”

Section: Introductionmentioning

confidence: 99%

Tracking a multitude of abilities as they develop

Bolsinova

Brinkhuis

Hofman

et al. 2022

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

Recently, the Urnings algorithm (Bolsinova et al., 2022, J. R. Stat. Soc. Ser. C Appl. Statistics, 71, 91) has been proposed that allows for tracking the development of abilities of the learners and the difficulties of the items in adaptive learning systems. It is a simple and scalable algorithm which is suited for large‐scale applications in which large streams of data are coming into the system and on‐the‐fly updating is needed. Compared to alternatives like the Elo rating system and its extensions, the Urnings rating system allows the uncertainty of the ratings to be evaluated and accounts for adaptive item selection which, if not corrected for, may distort the ratings. In this paper we extend the Urnings algorithm to allow for both between‐item and within‐item multidimensionality. This allows for tracking the development of interrelated abilities both at the individual and the population level. We present formal derivations of the multidimensional Urnings algorithm, illustrate its properties in simulations, and present an application to data from an adaptive learning system for primary school mathematics called Math Garden.

show abstract

“…Our third application is to the data of an online learning system Math Garden (Klinkenberg et al, 2011;Hofman et al, 2020), which is a platform for practicing primary school mathematics used widely in the Netherlands. Learners can practice various mathematics skills such as addition, fractions, and multiplication using series of exercises that are tailored to the level of their skill.…”

Section: Math Gardenmentioning

confidence: 99%

Urnings: A new method for tracking dynamically changing parameters in paired comparison systems

Bolsinova¹,

Maris²,

Hofman³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce a new rating system for tracking the development of parameters based on a stream of observations. Rating systems are applied in competitive games, adaptive learning systems, and platforms for product and service ratings. We model each observation as an outcome of a game of chance that depends on the parameters of interest (e.g., the outcome of a chess game depends on the abilities of the two players). Determining the probabilities of the different game outcomes is conceptualized as an urn problem, where a rating is represented by a proportion of colored balls in an urn. This setup allows for evaluating the standard errors of the ratings and performing statistical inferences about the development of and relations between parameters. Theoretical properties of the system in terms of the invariant distributions of the ratings and their convergence are derived. The properties of the rating system are illustrated with simulated examples and its potential for answering research questions is illustrated using data from competitive chess.

show abstract

Tracking with (Un)Certainty

Cited by 4 publications

References 33 publications

Urnings: A New Method for Tracking Dynamically Changing Parameters in Paired Comparison Systems

Urnings: A New Method for Tracking Dynamically Changing Parameters in Paired Comparison Systems

Tracking a multitude of abilities as they develop

Urnings: A new method for tracking dynamically changing parameters in paired comparison systems

Contact Info

Product

Resources

About