In this paper we present methods for comparing and evaluating forecasters whose predictions are presented as their subjective probability distributions of various random variables that will be observed in the future, e.g. weather forecasters who each day must specify their own probabilities that it will rain in a particular location. We begin by reviewing the concepts of calibration and refinement, and describing the relationship between this notion of refinement and the notion of sufficiency in the comparison of statistical experiments. We also consider the question of interrelationships among forecasters and discuss methods by which an observer should combine the predictions from two or more different forecasters. Then we turn our attention to the concept of a proper scoring rule for evaluating forecasters, relating it to the concepts of calibration and refinement. Finally, we discuss conditions under which one forecaster can exploit the predictions of another forecaster to obtain a better score.
A person deciding on a career, a wife, or a place to live bases his choice on two factors: (1) How much do I like each of the available alternatives? and (2) What are the chances for a successful outcome of each alternative? These two factors comprise the utility of each outcome for the person making the choice. This notion of utility is fundamental to most current theories of decision behavior. According to the expected utility hypothesis, if we could know the utility function of a person, we could predict his choice from among any set of actions or objects. But the utility function of a given subject is almost impossible to measure directly. To circumvent this difficulty, stochastic models of choice behavior have been formulated which do not predict the subject's choices but make statements about the probabilities that the subject will choose a given action. This paper reports an experiment to measure utility and to test one stochastic model of choice behavior.
226
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.