In the setting where information cannot be verified, we propose a simple yet powerful information theoretical framework-the Mutual Information Paradigm-for information elicitation mechanisms. Our framework pays every agent a measure of mutual information between her signal and a peer's signal. We require that the mutual information measurement has the key property that any "data processing" on the two random variables will decrease the mutual information between them. We identify such information measures that generalize Shannon mutual information.Our Mutual Information Paradigm overcomes the two main challenges in information elicitation without verification: (1) how to incentivize effort and avoid agents colluding to report random or identical responses (2) how to motivate agents who believe they are in the minority to report truthfully.Aided by the information measures we found, (1) we use the paradigm to design a family of novel mechanisms where truth-telling is a dominant strategy and any other strategy will decrease every agent's expected payment (in the multi-question, detail free, minimal setting where the number of questions is large); (2) we show the versatility of our framework by providing a unified theoretical understanding of existing mechanisms-Peer Prediction [Miller 2005], Bayesian Truth Serum [Prelec 2004], and Dasgupta and Ghosh [2013]-by mapping them into our framework such that theoretical results of those existing mechanisms can be reconstructed easily.We also give an impossibility result which illustrates, in a certain sense, the the optimality of our framework.
In the setting where participants are asked multiple similar possibly subjective multi-choice questions (e.g. Do you like Panda Express? Y/N; do you like Chick-fil-A? Y/N), a series of peer prediction mechanisms are designed to incentivize honest reports and some of them achieve dominantly truthfulness: truth-telling is a dominant strategy and strictly dominate other "non-permutation strategy" with some mild conditions. However, a major issue hinders the practical usage of those mechanisms: they require the participants to perform an infinite number of tasks. When the participants perform a finite number of tasks, these mechanisms only achieve approximated dominant truthfulness. The existence of a dominantly truthful multi-task peer prediction mechanism that only requires a finite number of tasks remains to be an open question that may have a negative result, even with full prior knowledge.This paper answers this open question by proposing a new mechanism, Determinant based Mutual Information Mechanism (DMI-Mechanism), that is dominantly truthful when the number of tasks is ≥ 2C. C is the number of choices for each question (C = 2 for binary-choice questions). DMI-Mechanism also pays truth-telling higher than any strategy profile and strictly higher than uninformative strategy profiles (informed truthfulness). In addition to the truthfulness properties, DMI-Mechanism is also easy to implement since it does not require any prior knowledge (detailfree) and only requires ≥ 2 participants. The core of DMI-Mechanism is a novel information measure, Determinant based Mutual Information (DMI). DMI generalizes Shannon's mutual information and the square of DMI has a simple unbiased estimator. In addition to incentivizing honest reports, DMI-Mechanism can also be transferred into an information evaluation rule that identifies high-quality information without verification when there are ≥ 3 participants.To the best of our knowledge, DMI-Mechanism is both the first detail-free informed-truthful mechanism and the first dominantly truthful mechanism that works for a finite number of tasks, not to say a small constant number of tasks. arXiv:1911.00272v1 [cs.GT] 1 Nov 2019 1 Permutation strategy means always reporting a permuted version of the answer, e.g. say "like" when the honest answer is "dislike" while say "dislike" when the honest answer is "like".2 The "best" strategy profile has the highest amount of expected payment for every participant.3 ( , δ)-informed truthfulness: truth-telling is at least better than other strategy profiles, with 1 − δ probability.
A central question 1 of crowdsourcing is how to elicit expertise from agents. This is even more difficult when answers cannot be directly verified. A key challenge is that sophisticated agents may strategically withhold effort or information when they believe their payoff will be based upon comparison with other agents whose reports will likely omit this information due to lack of effort or expertise.Our work defines a natural model for this setting based on the assumption that more sophisticated agents know the beliefs of less sophisticated agents.We then provide a mechanism design framework for this setting. From this framework, we design several novel mechanisms, for both the single and multiple tasks settings, that (1) encourage agents to invest effort and provide their information honestly; (2) output a correct "hierarchy" of the information when agents are rational. arXiv:1802.08312v2 [cs.GT] 22 May 2018 2 Mechanism Design ToolsWe use two key information theory ingredients in designing information elicitation mechanisms. The first ingredient is f -mutual information M I f (X; Y ) which measures the amount of information crossing two random variables X, Y . For example, if X is independent with Y -no information crosses X and Y , M I f (X; Y ) = 0. The second ingredient is proper scoring rule P S(x, p) which measures the accuracy of the prediction p even we only have one sample x of the outcome X.Both two ingredients have the information monotonicity property. If the information is measured by f -mutual information, any "data processing" on either of the random variables will decrease the amount of information crossing them. If the accuracy of a forecast is measured by a proper scoring rule, more information implies a more accurate forecast. f -mutual informationwhere f (⋅) is a convex function and f (1) = 0. Two commonly used f -divergences are KL divergence and total variation distance. Now we start to introduce f -mutual information.Given two random variables X, Y , let U X,Y and V X,Y be two probability measures where U X,Y is the joint distribution of (X, Y ) and V is the product of the marginal distributions of X and Y . Formally, for every pair of (x, y),If U X,Y is very different with V X,Y , the mutual information between X and Y should be high since knowing X changes the belief for Y a lot. If U X,Y equals to V X,Y , the mutual information between X and Y should be zero since X is independent with Y . Intuitively, the "distance" between U X,Y and V X,Y represents the mutual information between them.Definition 2.1 (f -mutual information [20]). The f -mutual information between X and Y is defined aswhere D f is f -divergence. Definition 2.2 (Conditional f -mutual information [20]). Given three random variables X, Y, Z, we define M I f (X; Y Z) as z P r[Z = z]M I f (X; Y Z = z) where M I f (X; Y Z = z) ∶= M I f (X ′ ; Y ′ ) where P r[X ′ = x, Y ′ = y] = P r[X = x, Y = y Z = z].
Peer-prediction [18] is a (meta-)mechanism which, given any proper scoring rule, produces a mechanism to elicit privately-held, non-verifiable information from self-interested agents. Formally, truth-telling is a strict Nash equilibrium of the mechanism. Unfortunately, there may be other equilibria as well (including uninformative equilibria where all players simply report the same fixed signal, regardless of their true signal) and, typically, the truth-telling equilibrium does not have the highest expected payoff. The main result of this paper is to show that, in the symmetric binary setting, by tweaking peer-prediction, in part by carefully selecting the proper scoring rule it is based on, we can make the truth-telling equilibrium focal-that is, truth-telling has higher expected payoff than any other equilibrium.Along the way, we prove the following: in the setting where agents receive binary signals we 1) classify all equilibria of the peer-prediction mechanism; 2) introduce a new technical tool for understanding scoring rules, which allows us to make truth-telling pay better than any other informative equilibrium; 3) leverage this tool to provide an optimal version of the previous result; that is, we optimize the gap between the expected payoff of truth-telling and other informative equilibria; and 4) show that with a slight modification to the peer-prediction framework, we can, in general, make the truth-telling equilibrium focal-that is, truth-telling pays more than any other equilibrium (including the uninformative equilibria).
Eliciting labels from crowds is a potential way to obtain large labeled data. Despite a variety of methods developed for learning from crowds, a key challenge remains unsolved: learning from crowds without knowing the information structure among the crowds a priori, when some people of the crowds make highly correlated mistakes and some of them label effortlessly (e.g. randomly). We propose an information theoretic approach, Max-MIG, for joint learning from crowds, with a common assumption: the crowdsourced labels and the data are independent conditioning on the ground truth. Max-MIG simultaneously aggregates the crowdsourced labels and learns an accurate data classifier. Furthermore, we devise an accurate data-crowds forecaster that employs both the data and the crowdsourced labels to forecast the ground truth. To the best of our knowledge, this is the first algorithm that solves the aforementioned challenge of learning from crowds. In addition to the theoretical validation, we also empirically show that our algorithm achieves the new state-of-the-art results in most settings, including the real-world data, and is the first algorithm that is robust to various information structures. Codes are available at https://github.com/Newbeeer/Max-MIG
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.