Informed Truthfulness in Multi-Task Peer Prediction

Shnayder, Victor; Agarwal, Amit; Frongillo, Rafael M.; Parkes, David C.

doi:10.1145/2940716.2940790

Cited by 108 publications

(172 citation statements)

References 16 publications

(25 reference statements)

Supporting

Mentioning

170

Contrasting

Order By: Relevance

“…This assumption is reasonable when agents receive tasks in independent random orders. Shnayder et al [17] show that the above assumption can be removed when the mechanism is linear in the joint distribution over agents' reports. However, the mechanism designed in this paper is not linear.…”

Section: Mechanism Design For Multi-task Peer Predictionmentioning

confidence: 99%

See 1 more Smart Citation

Dominantly Truthful Multi-task Peer Prediction with a Constant Number of Tasks

Kong¹

2020

Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms

View full text Add to dashboard Cite

In the setting where participants are asked multiple similar possibly subjective multi-choice questions (e.g. Do you like Panda Express? Y/N; do you like Chick-fil-A? Y/N), a series of peer prediction mechanisms are designed to incentivize honest reports and some of them achieve dominantly truthfulness: truth-telling is a dominant strategy and strictly dominate other "non-permutation strategy" with some mild conditions. However, a major issue hinders the practical usage of those mechanisms: they require the participants to perform an infinite number of tasks. When the participants perform a finite number of tasks, these mechanisms only achieve approximated dominant truthfulness. The existence of a dominantly truthful multi-task peer prediction mechanism that only requires a finite number of tasks remains to be an open question that may have a negative result, even with full prior knowledge.This paper answers this open question by proposing a new mechanism, Determinant based Mutual Information Mechanism (DMI-Mechanism), that is dominantly truthful when the number of tasks is ≥ 2C. C is the number of choices for each question (C = 2 for binary-choice questions). DMI-Mechanism also pays truth-telling higher than any strategy profile and strictly higher than uninformative strategy profiles (informed truthfulness). In addition to the truthfulness properties, DMI-Mechanism is also easy to implement since it does not require any prior knowledge (detailfree) and only requires ≥ 2 participants. The core of DMI-Mechanism is a novel information measure, Determinant based Mutual Information (DMI). DMI generalizes Shannon's mutual information and the square of DMI has a simple unbiased estimator. In addition to incentivizing honest reports, DMI-Mechanism can also be transferred into an information evaluation rule that identifies high-quality information without verification when there are ≥ 3 participants.To the best of our knowledge, DMI-Mechanism is both the first detail-free informed-truthful mechanism and the first dominantly truthful mechanism that works for a finite number of tasks, not to say a small constant number of tasks. arXiv:1911.00272v1 [cs.GT] 1 Nov 2019 1 Permutation strategy means always reporting a permuted version of the answer, e.g. say "like" when the honest answer is "dislike" while say "dislike" when the honest answer is "like".2 The "best" strategy profile has the highest amount of expected payment for every participant.3 ( , δ)-informed truthfulness: truth-telling is at least better than other strategy profiles, with 1 − δ probability.

show abstract

Section: Mechanism Design For Multi-task Peer Predictionmentioning

confidence: 99%

“…A non-detail-free version of CA also achieves informed truthfulness with 2 questions. Moreover, although the original paper of CA[17] does not claim the dominant truthfulness, its detail-free version is dominantly truthful.…”

mentioning

confidence: 99%

Dominantly Truthful Multi-task Peer Prediction with a Constant Number of Tasks

Kong¹

2020

Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms

View full text Add to dashboard Cite

show abstract

“…Publication date: June 2019. t (i.e., R c , Rĉ ). According to the original report [6] and a subsequent report for its generalization [32], DG13 can be formulated as…”

Section: Dg13 For Reward Computationmentioning

confidence: 99%

“…31 If we assume the cost of curation as c, the expected rewards in this example become E(k) = p(k − c) − (1 − p)(q + c). This extension shifts the condition for E(k ) = 0, from where [x] >0 and [x] <0 indicate that x is positive and negative, respectively 32 . It is apparent that E(θ G t c ) is maximized only when both c andĉ provide truthful reports (r (0) = 0, r (1) = 1) or opposite reports (r (0) = 1, r (1) = 0).…”

Section: B Proof Of the Strong Truthfulness Of The Dg13 Mechanismmentioning

confidence: 99%

Token-Curated Registry with Citation Graph

Ito

Tanaka

2019

ledger

View full text Add to dashboard Cite

In this study, we aim to incorporate the expertise of anonymous curators into a token-curated registry (TCR), a decentralized recommender system for collecting a list of high-quality content. This registry is important, because previous studies on TCRs have not specifically focused on technical content, such as academic papers and patents, whose effective curation requires expertise in relevant fields. To measure expertise, curation in our model focuses on both the content and its citation relationships, for which curator assignment uses the Personalized PageRank (PPR) algorithm while reward computation uses a multi-task peer-prediction mechanism. Our proposed CitedTCR bridges the literature on network-based and token-based recommender systems and contributes to the autonomous development of an evolving citation graph for high-quality content. Moreover, we experimentally confirm the incentive for registration and curation in CitedTCR using the simplification of a one-to-one correspondence between users and content (nodes). 11 For this property, PPR is often referred to as the random walk with restart (RWR) algorithm. 12 See the textbook [8], for more comprehensive review on peer-prediction method and other information elicitation models. 13 Uninformative equilibria are designated as a blind agreements in original DG13 [6].

show abstract

“…Bayesian markets avoid this issue at the cost of asking for a betting decision. Moreover, in Bayesian markets, degenerate equilibria are dominated by truth telling, unlike in many peer prediction mechanisms (17).…”

Section: Related Literaturementioning

confidence: 99%

Bayesian markets to elicit private information

Baillon

2017

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Financial markets reveal what investors think about the future, and prediction markets are used to forecast election results. Could markets also encourage people to reveal private information, such as subjective judgments (e.g., "Are you satisfied with your life?") or unverifiable facts? This paper shows how to design such markets, called Bayesian markets. People trade an asset whose value represents the proportion of affirmative answers to a question. Their trading position then reveals their own answer to the question. The results of this paper are based on a Bayesian setup in which people use their private information (their "type") as a signal. Hence, beliefs about others' types are correlated with one's own type. Bayesian markets transform this correlation into a mechanism that rewards truth telling. These markets avoid two complications of alternative methods: they need no knowledge of prior information and no elicitation of metabeliefs regarding others' signals.prediction markets | economic incentives | truth telling | mechanism design | Bayesianism W hen trading in a market or submitting a price to an auctioneer, people reveal the extent to which they value a good. In finance, option and future markets reveal investors' beliefs. Although such revelations of tastes and beliefs originally were a by-product of regular markets, they have led to prediction markets whose primary goal is to reveal beliefs. In such markets, people can buy or sell a simple asset whose value is 1 unit (e.g., $10) if a specified event occurs and is 0 otherwise. Buying a unit at a given price is a bet that the event is more likely than people think to be on average. The resulting market price reveals aggregate expectations. Prediction markets have been used by various public organizations and companies (1, 2); for example, social scientists use this type of market to predict the replicability of experiments (3, 4). Prediction markets also have been shown to outperform polls in predicting election results (5). Unfortunately, they can only be used for events whose occurrence can be objectively verified. When collecting personal data such as opinions and self-assessed measurements, objective verification often is conceptually or practically impossible. This paper introduces Bayesian markets, which are designed to elicit private information in binary settings (yes-or-no questions) when objective verification is impossible. Bayesian markets rely on the assumption that the private information that people possess influences their belief about others. Such inference is justified by Bayesian reasoning, a widely used theory of rational reasoning (6). Answering yes (Y) provides information (a signal) that can be used to update one's prior expectation about the proportion of Y answers. In Bayesian markets, the assets traded have a value determined by the proportion of Y answers to a given question, for example, "Are you satisfied with your life?." Using Bayesian reasoning, we can predict that people who actually answer Y will expect a higher as...

show abstract

Informed Truthfulness in Multi-Task Peer Prediction

Cited by 108 publications

References 16 publications

Dominantly Truthful Multi-task Peer Prediction with a Constant Number of Tasks

Dominantly Truthful Multi-task Peer Prediction with a Constant Number of Tasks

Token-Curated Registry with Citation Graph

Bayesian markets to elicit private information

Contact Info

Product

Resources

About