Robust forecast aggregation

Arieli, Itai; Babichenko, Yakov; Smorodinsky, Rann

doi:10.1073/pnas.1813934115

Cited by 24 publications

(55 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…See [CW07] and [DL14] for more recent high-level overviews of the subject, including discussion of both Bayesian and axiomatic approaches, as well as discussion of the axioms we discussed in Section 1.1. Recent work on Bayesian opinion pooling includes [CL13], [Sat+14], [FCK15], and [ABS18]. Finally, see [FES20] for work on learning weights for linear pooling under a quadratic loss function; this is an instance of our more general theory in Section 5.…”

Section: Related Workmentioning

confidence: 99%

From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation

Neyman¹,

Roughgarden²

2021

Preprint

View full text Add to dashboard Cite

This paper forges a strong connection between two seemingly unrelated forecasting problems: incentive-compatible forecast elicitation and forecast aggregation. Proper scoring rules are the well-known solution to the former problem. To each such rule s we associate a corresponding method of aggregation, mapping expert forecasts and expert weights to a "consensus forecast," which we call quasi-arithmetic (QA) pooling with respect to s. We justify this correspondence in several ways:• QA pooling with respect to the two most well-studied scoring rules (quadratic and logarithmic) corresponds to the two most well-studied forecast aggregation methods (linear and logarithmic).• Given a scoring rule s used for payment, a forecaster agent who sub-contracts several experts, paying them in proportion to their weights, is best off aggregating the experts' reports using QA pooling with respect to s, meaning this strategy maximizes its worst-case profit (over the possible outcomes).• The score of an aggregator who uses QA pooling is concave in the experts' weights. As a consequence, online gradient descent can be used to learn appropriate expert weights from repeated experiments with low regret.• The class of all QA pooling methods is characterized by a natural set of axioms (generalizing classical work by Kolmogorov on quasi-arithmetic means).

show abstract

Section: Related Workmentioning

confidence: 99%

From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation

Neyman¹,

Roughgarden²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A key distinction is that the aforementioned papers consider robust optimality from an interim approach, while we study the decision maker's robustly optimal ex-ante decision plan. Finally, Arieli, Babichenko, and Smorodinsky [2018] also study features of the robustly optimal ex-ante decision plans. An important difference is that they study robust aggregation in a specific decision problem while we characterize the robustly optimal ex-ante decision plan in general decision problems.…”

Section: Related Literaturementioning

confidence: 99%

“…An important difference is that they study robust aggregation in a specific decision problem while we characterize the robustly optimal ex-ante decision plan in general decision problems. 4 Moreover, Arieli, Babichenko, and Smorodinsky [2018] study robust aggregation when the decision maker has limited knowledge of the distribution of posteriors/signals generated by each expert. In contrast, in order to focus our analysis on robustness concerns about correlations between information sources, we assume in our model that the decision maker possesses a perfect understanding of the marginal distributions of signals of each expert/information source in isolation.…”

Section: Related Literaturementioning

confidence: 99%

Robust Merging of Information

Oliveira

Ishii

Xiao

2021

Preprint

View full text Add to dashboard Cite

When multiple sources of information are available, any decision must take into account their correlation. If information about this correlation is lacking, an agent may find it desirable to make a decision that is robust to possible correlations. Our main results characterize the strategies that are robust to possible hidden correlations. In particular, with two states and two actions, the robustly optimal strategy pays attention to a single information source, ignoring all others. More generally, the robustly optimal strategy may need to combine multiple information sources, but can be constructed quite simply by using a decomposition of the original problem into separate decision problems, each requiring attention to only one information source. An implication is that an information source generates value to the agent if and only if it is best for at least one of these decomposed problems.

show abstract

“…Our work is most similar to [ABS18], which likewise seeks to optimize an aggregation method against an adversarially selected information structure. However, the class of information structures that we consider is broader: while they consider the case of two Blackwell-ordered experts (i.e.…”

Section: Related Workmentioning

confidence: 99%

Are You Smarter Than a Random Expert? The Robust Aggregation of Substitutable Signals

Neyman¹,

Roughgarden²

2021

Preprint

View full text Add to dashboard Cite

The problem of aggregating expert forecasts is ubiquitous in fields as wide-ranging as machine learning, economics, climate science, and national security. Despite this, our theoretical understanding of this question is fairly shallow. This paper initiates the study of forecast aggregation in a context where experts' knowledge is chosen adversarially from a broad class of information structures. While in full generality it is impossible to achieve a nontrivial performance guarantee, we show that doing so is possible under a condition on the experts' information structure that we call projective substitutes. The projective substitutes condition is a notion of informational substitutes: that there are diminishing marginal returns to learning the experts' signals. We show that under the projective substitutes condition, taking the average of the experts' forecasts improves substantially upon the strategy of trusting a random expert. We then consider a more permissive setting, in which the aggregator has access to the prior. We show that by averaging the experts' forecasts and then extremizing the average by moving it away from the prior by a constant factor, the aggregator's performance guarantee is substantially better than is possible without knowledge of the prior. Our results give a theoretical grounding to past empirical research on extremization and help give guidance on the appropriate amount to extremize.

show abstract

Robust forecast aggregation

Cited by 24 publications

References 26 publications

From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation

From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation

Robust Merging of Information

Are You Smarter Than a Random Expert? The Robust Aggregation of Substitutable Signals

Contact Info

Product

Resources

About