PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments

Lev, Omer; Mattei, Nicholas; Turrini, Paolo

doi:10.1016/j.artint.2022.103843

Cited by 5 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The second problem that is related to ours is peer nominationsee, for example, [2], [3], [15]. In peer nomination, the real-life motivation is a scenario where a number of researchers are competing for grants and they themselves select proposals that are worthy of funding.…”

Section: Related Workmentioning

confidence: 99%

Learning in Teams: Peer Evaluation for Fair Assessment of Individual Contributions

Duzhin

2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

We develop a game-theoretical model of a classroom scenario, where n students collaborate on a common task. We assume that there exists an objective truth known to the students but not to the course instructor. Each of the students estimates the contributions of all team members and reports her estimates to the instructor. Thus, a matrix A of peer evaluations arises and the instructor’s task is to grade students individually based on peer evaluations. The method of deriving individual grades from the matrix A is supposed to be psychometrically valid and reliable. We argue that mathematically it means that 1) the collective truth-telling is a strict Nash equilibrium and 2) individual grade of student i does not depend on the true contribution of student j for j ≠ i. Existing methods of peer evaluation commonly used in educational practice fail to satisfy at least one of these properties. We construct a new method of peer evaluation satisfying both desired properties for n ≥ 5. We share a large dataset (1201 students, 220 teams, 6619 evaluations) of peer evaluations collected in undergraduate courses taught by the author, outline some practical challenges, and show how these challenges can be addressed.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning in Teams: Peer Evaluation for Fair Assessment of Individual Contributions

Duzhin

2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

show abstract

“…These mechanisms incentivize coordination on similar evaluation reports by punishing evaluations that do not match each other. Thus, they do not necessarily incentivize accuracy (Prelec, 2004;Miller et al, 2005;Jurca and Faltings, 2009;Faltings et al, 2012;Dasgupta and Ghosh, 2013;Lev et al, 2023;Waggoner and Chen, 2014;Shnayder et al, 2016;Dhull et al, 2022). Any such mechanism introduces uninformative equilibria alongside the truth-telling one (Jurca and Faltings, 2009;Waggoner and Chen, 2014).…”

Section: Introductionmentioning

confidence: 99%

Removing Bias and Incentivizing Precision in Peer-grading

Chakraborty,

Jindal,

Nath

2024

jair

View full text Add to dashboard Cite

Most peer-evaluation practices rely on the evaluator’s goodwill and model them as potentially noisy evaluators. But what if graders are competitive, i.e., enjoy higher utility when their peers get lower scores? We model the setting as a multi-agent incentive design problem and propose a new mechanism, PEQA, that incentivizes these agents (peer-graders) through a score-assignment rule and a grading performance score. PEQA is designed in such a way that it makes grader-bias irrelevant and ensures grader-utility to be monotonically increasing with the grading-precision, despite competitiveness. When grading is costly and costs are private information of the individual graders, a modified version of PEQA implements the socially optimal grading-choices in equilibrium. Data from our classroom experiments is consistent with our theoretical assumptions and show that PEQA outperforms the popular median mechanism, which is used in several massive open online courses (MOOCs).

show abstract

Manipulation and peer mechanisms: A survey

Olckers,

Walsh

2024

Artificial Intelligence

View full text Add to dashboard Cite

PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments

Cited by 5 publications

References 36 publications

Learning in Teams: Peer Evaluation for Fair Assessment of Individual Contributions

Learning in Teams: Peer Evaluation for Fair Assessment of Individual Contributions

Removing Bias and Incentivizing Precision in Peer-grading

Manipulation and peer mechanisms: A survey

Contact Info

Product

Resources

About