A New Vector Partition of the Probability Score

Murphy, Allan H.

doi:10.1175/1520-0450(1973)012<0595:anvpot>2.0.co;2

Cited by 906 publications

(629 citation statements)

References 5 publications

Supporting

Mentioning

606

Contrasting

Unclassified

Order By: Relevance

“…Scoring Rules Several different forms of scoring rules have been proposed ( Brier, 1950;Murphy, 1972Murphy, , 1973Matheson and Winkler, 1976;DeGroot and Fienberg, 1983 ) but all essentially give a mathematical measure of the distance between the predicted probabilities and observed frequencies. Scoring rules were originally developed as aids in eliciting probabilities; they provided feedback to subjects in pilot studies and were intended to induce well -calibrated judgments in final elicitations ( Cooke, 1991 ).…”

Section: Calibration Methodsmentioning

confidence: 99%

Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene

Walker

Catalano

Hammitt

et al. 2003

J Expo Sci Environ Epidemiol

View full text Add to dashboard Cite

The recent movement of regulatory agencies toward probabilistic analyses of human health and environmental risks has focused greater attention on the quality of the estimates of variability and uncertainty that underlie them. Of particular concern is how uncertainty -a measure of what is not known -is characterized, as uncertainty can play an influential role in analyses of the need for regulatory controls or in estimates of the economic value of additional research. This paper reports the second phase of a study, conducted as an element of the National Human Exposure Assessment Survey ( NHEXAS ), to obtain and calibrate exposure assessment experts' judgments about uncertainty in residential ambient, residential indoor, and personal air benzene concentrations experienced by the nonsmoking, nonoccupationally exposed population in U.S. EPA's Region V. Subjective judgments ( i.e., the median, interquartile range, and 90% confidence interval ) about the means and 90th percentiles of each of the benzene distributions were elicited from the seven experts participating in the study. The calibration or quality of the experts' judgments was assessed by comparing them to the actual measurements from the NHEXAS Region V study using graphical techniques, a quadratic scoring rule, and surprise and interquartile indices. The results from both quantitative scoring methods suggested that, considered collectively, the experts' judgments were relatively well calibrated although on balance, underconfident. The calibration of individual expert judgments appeared variable, highlighting potential pitfalls in reliance on individual experts. In a surprising finding, the experts' judgments about the 90th percentiles of the benzene distributions were better calibrated than their predictions about the means; the experts tended to be overconfident in their ability to predict the means. This paper is also one of the first calibration studies to demonstrate the importance of taking into account intraexpert correlation on the statistical significance of the findings. When the judgments were assumed to be independent, analysis of the surprise and interquartile indices found evidence of poor calibration ( P < 0.05 ). However, when the intraexpert correlation in the study was taken into account, these findings were no longer statistically significant. The analysis further found that the experts' judgments scored better than estimates of Region V benzene concentrations simply drawn from earlier studies of ambient, indoor and personal benzene levels in other U.S. cities. These results suggest the value of careful elicitation of expert judgments in characterizing exposures in probabilistic form. Additional calibration studies need to be undertaken to corroborate and extend these findings.

show abstract

Section: Calibration Methodsmentioning

confidence: 99%

Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene

Walker

Catalano

Hammitt

et al. 2003

J Expo Sci Environ Epidemiol

View full text Add to dashboard Cite

show abstract

“…The first verification diagnostic used was the Brier skill score, BSS 5 1 2 (BS/BS ref ), where BS is the Brier score of the algorithm and BS ref is a reference Brier score (Murphy 1973). The Brier score, as defined by Wilks (2006), is…”

Section: A Brier Skill Scorementioning

confidence: 99%

Objective Estimation of the 24-h Probability of Tropical Cyclone Formation

Schumacher

DeMaria

Knaff

2009

Weather and Forecasting

View full text Add to dashboard Cite

A new product for estimating the 24-h probability of TC formation in individual 58 3 58 subregions of the North Atlantic, eastern North Pacific, and western North Pacific tropical basins is developed. This product uses environmental and convective parameters computed from best-track tropical cyclone (TC) positions, National Centers for Environmental Prediction (NCEP) Global Forecasting System (GFS) analysis fields, and water vapor (;6.7 mm wavelength) imagery from multiple geostationary satellite platforms. The parameters are used in a two-step algorithm applied to the developmental dataset. First, a screening step removes all data points with environmental conditions highly unfavorable to TC formation. Then, a linear discriminant analysis (LDA) is applied to the screened dataset. A probabilistic prediction scheme for TC formation is developed from the results of the LDA.Coefficients computed by the LDA show that the largest contributors to TC formation probability are climatology, 850-hPa circulation, and distance to an existing TC. The product was evaluated by its Brier and relative operating characteristic skill scores and reliability diagrams. These measures show that the algorithmgenerated probabilistic forecasts are skillful with respect to climatology, and that there is relatively good agreement between forecast probabilities and observed frequencies. As such, this prediction scheme has been implemented as an operational product called the National Environmental Satellite, Data, and Information Services (NESDIS) Tropical Cyclone Formation Probability (TCFP) product. The TCFP product updates every 6 h and displays plots of TC formation probability and input parameter values on its Web site. At present, the TCFP provides real-time, objective TC formation guidance used by tropical cyclone forecast offices in the Atlantic, eastern Pacific, and western Pacific basins.

show abstract

“…1. Schematic representation of overall probabilistic judgment accuracy and its elements as discerned in the Murphy (1973) decomposition of the mean probability score.…”

Section: Directional Forecastsmentioning

confidence: 99%

“…; K, where K is the total number of categories. Then the calibration index (CI) is given by (Murphy, 1973;Sanders, 1963):…”

Section: Murphy Decompositionmentioning

confidence: 99%

Professional vs. amateur judgment accuracy: The case of foreign exchange rates

Önkal

Yates

Şımga-Muğan

et al. 2003

Organizational Behavior and Human Decision Processes

View full text Add to dashboard Cite

Highly knowledgeable people often fail to achieve highly accurate judgments, a phenomenon sometimes called the ''processperformance paradox.'' The present research tested for this paradox in foreign exchange (FX) rate forecasting. Forty professional and 57 sophisticated amateur forecasters made one-day and one-week-ahead FX predictions in deterministic and probabilistic formats. Among the conclusions indicated by the results are: (a) professional accuracy usually surpasses amateur accuracy, although many amateurs outperform many professionals; (b) professionals appear to achieve high proficiency via heavy reliance on predictive information (unlike what has been observed before, e.g., for stock prices); (c) forecast format strongly affects judgment accuracy and processes; and (d) apparent overconfidence can transform itself into underconfidence depending on when and how forecasters must articulate their confidence.

show abstract

A New Vector Partition of the Probability Score

Cited by 906 publications

References 5 publications

Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene

Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene

Objective Estimation of the 24-h Probability of Tropical Cyclone Formation

Professional vs. amateur judgment accuracy: The case of foreign exchange rates

Contact Info

Product

Resources

About