Optimization of Scoring Rules

Hartline, Jason D.; Li, Yingkai; Shan, Liren; Wu, Yifan

doi:10.48550/arxiv.2007.02905

Cited by 2 publications

(7 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Information acquisition. Prior work of Hartline et al (2020); Chen and Yu (2021) considered similar problems, namely, designing proper scoring rules to maximize the incentive of an agent to acquire a signal before reporting a prediction. Our problem is framed slightly differently, as the cost of acquiring the signal is given and the goal is minimizing payment.…”

Section: Related Workmentioning

confidence: 99%

“…Limited liability has not yet been explored in the information acquisition literature. Instead, research so far has assumed boundedness of the payment (Hartline et al, 2020;Chen and Yu, 2021). While boundedness is a reasonable restriction for the principal in some settings, we believe limited liability is often a more natural assumption in that setting.…”

Section: Introductionmentioning

confidence: 97%

“…This problem has classically been addressed by proper scoring rules, which incent experts to truthfully predict events Savage (1971); Gneiting and Raftery (2007). Recent work has investigated the design of scoring rules to incent the agent to acquire information and report it (Hartline et al, 2020;Chen and Yu, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Contracts with Information Acquisition, via Scoring Rules

Papireddygari¹,

Waggoner²

2022

Preprint

View full text Add to dashboard Cite

We consider a principal-agent problem where the agent may privately choose to acquire relevant information prior to taking a hidden action. This model generalizes two special cases: a classic moral hazard setting, and a more recently studied problem of incentivizing information acquisition (IA). We show that all of these problems can be reduced to the design of a proper scoring rule. Under a limited liability condition, we consider the special cases separately and then the general problem. We give novel results for the special case of IA, giving a closed form "pointed polyhedral cone" solution for the general multidimensional problem. We also describe a geometric, scoring-rules based solution to the case of the classic contracts problem. Finally, we give an efficient algorithm for the general problem of Contracts with Information Acquisition.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 97%

See 1 more Smart Citation

Contracts with Information Acquisition, via Scoring Rules

Papireddygari¹,

Waggoner²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Properties beyond robustness against strategic behavior have also recently been examined in other domains of information elicitation that are related to, but distinct from, peer prediction. In probabilistic forecasting 4 , for example, Hartline et al [8] and Neyman et al [17] both consider optimizing for properties related to incentivizing effort when selecting a proper scoring rule with which to score forecasts. In contrast to peer prediction mechanisms, however, proper scoring rules (by definition) each satisfy the same particular theoretical notion of robustness against strategic behavior.…”

Section: Peer Prediction and Information Elicitationmentioning

confidence: 99%

“…where G is a Gamma distribution. 8 To reiterate, the simulated data in our experiments is always generated according to the model described in Section 3. But instead of estimating parameters of that underlying model, we estimate the parameters of model PG 1 .…”

Section: Parametric Mechanismsmentioning

confidence: 99%

Measurement Integrity in Peer Prediction: A Peer Assessment Case Study

Burrell¹,

Schoenebeck²

2021

Preprint

View full text Add to dashboard Cite

We propose measurement integrity, a property related to ex post reward fairness, as a novel desideratum for peer prediction mechanisms in many natural applications, including peer assessment. We then operationalize this notion to evaluate the measurement integrity of different peer prediction mechanisms in computational experiments. Our evaluations simulate the application of peer prediction mechanisms to peer assessment-a setting in which realistic models have been validated on real data and in which ex post fairness concerns are particularly salient. We find that peer prediction mechanisms, as proposed in the literature, largely fail to demonstrate measurement integrity in our experiments. However, we also find that supplementing the mechanisms with realistic parametric statistical models can, in some cases, improve their measurement integrity.In the same setting, we also evaluate an empirical notion of robustness against strategic behavior to complement the theoretical analyses of robustness against strategic behavior that have been the primary focus of the peer prediction literature. In this dimension of analysis, we again find that supplementing certain mechanisms with realistic parametric statistical models can improve their empirical performance. Even so, though, we find that theoretical guarantees of robustness against strategic behavior are somewhat noisy predictors of empirical robustness against strategic behavior.As a whole, our empirical methodology for quantifying desirable mechanism properties facilitates a more nuanced comparison between mechanisms than theoretical analysis alone. Ultimately, we find that there is a trade-off between our two dimensions of analysis. The best performing mechanisms for measurement integrity are highly susceptible to strategic behavior. On the other hand, certain parametric peer prediction mechanisms are robust against all the strategic manipulations that we consider while still achieving reasonable measurement integrity.

show abstract

Optimization of Scoring Rules

Cited by 2 publications

References 18 publications

Contracts with Information Acquisition, via Scoring Rules

Contracts with Information Acquisition, via Scoring Rules

Measurement Integrity in Peer Prediction: A Peer Assessment Case Study

Contact Info

Product

Resources

About