Fast, Optimal, and Targeted Predictions Using Parameterized Decision Analysis

Kowal, Daniel R.

doi:10.1080/01621459.2021.1891926

Cited by 15 publications

(31 citation statements)

References 30 publications

(36 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our approach for evaluating predictive performance follows two core principles, which are similarly emphasized by Kowal: 46 (i) predictions should be evaluated out‐of‐sample and (ii) evaluations should be accompanied by full uncertainty quantification . Naturally, out‐of‐sample predictive evaluations are strongly preferred as more reliable assessments of the predictive capability of a model.…”

Section: Methodsmentioning

confidence: 99%

“…In conjunction with the concurrent work by Kowal, 46 we introduce two complementary notions of proximity in predictive performance, which are made precise below. First, let

η \geq 0

define a margin for acceptable performance: any model with predictive performance within

η

of the best model is considered acceptable.…”

Section: Methodsmentioning

confidence: 99%

“…The out‐of‐sample MSEs in () better capture the essence of the prediction task, where the predictand is not available for model‐fitting of ()‐() or solving (). Since these quantities may be sensitive to the specific testing set

ℐ_{k}

, we average () over all

K

testing sets:

{MSE}_{λ}^{o u t} = \frac{1}{K} \sum_{k = 1}^{K} {MSE}_{λ}^{o u t} (k), {\tilde{MSE}}_{λ}^{o u t} = \frac{1}{K} \sum_{k = 1}^{K} {\tilde{MSE}}_{λ}^{o u t} (k),

which is advocated in the literature 46,47 . When the holdout sets are mutually exclusive, each observation

i = 1, \dots, n

appears exactly once in ().…”

Section: Methodsmentioning

confidence: 99%

“…Next, consider the out‐of‐sample predictive variable

{\tilde{gapMSE}}_{λ}^{o u t} = {\tilde{MSE}}_{λ}^{o u t} - {\tilde{MSE}}_{λ_{m i n}}^{o u t},

which is the gap between model

λ

and the best model

λ_{m i n}

. The set of acceptable models 46 is

Λ_{η, ε} = {λ : ℙ ({\tilde{gapMSE}}_{λ}^{o u t} \leq η σ^{2}) \geq ε} .

…”

Section: Methodsmentioning

confidence: 99%

“…The acceptable set of models in () is equivalently derived using posterior predictive intervals: 46

λ \in Λ_{η, ε}

if and only if there exists a

(1 - ε)

lower posterior predictive credible interval for

{\tilde{gapMSE}}_{λ}^{o u t} / σ^{2}

that includes

η

. Naturally, if the predictive interval for the increase in out‐of‐sample MSE excludes

η

, such as

η = 0

, then this choice of

λ

is deemed unacceptable and omitted from

Λ_{η, ε}

.…”

Section: Methodsmentioning

confidence: 99%

See 4 more Smart Citations

Bayesian variable selection for understanding mixtures in environmental exposures

Kowal

Bravo

Leong

et al. 2021

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

Social and environmental stressors are crucial factors in child development. However, there exists a multitude of measurable social and environmental factors—the effects of which may be cumulative, interactive, or null. Using a comprehensive cohort of children in North Carolina, we study the impact of social and environmental variables on 4th end‐of‐grade exam scores in reading and mathematics. To identify the essential factors that predict these educational outcomes, we design new tools for Bayesian linear variable selection using decision analysis. We extract a predictive optimal subset of explanatory variables by coupling a loss function with a novel model‐based penalization scheme, which leads to coherent Bayesian decision analysis and empirically improves variable selection, estimation, and prediction on simulated data. The Bayesian linear model propagates uncertainty quantification to all predictive evaluations, which is important for interpretable and robust model comparisons. These predictive comparisons are conducted out‐of‐sample with a customized approximation algorithm that avoids computationally intensive model refitting. We apply our variable selection techniques to identify the joint collection of social and environmental stressors—and their interactions—that offer clear and quantifiable improvements in prediction of reading and mathematics exam scores.

show abstract

Section: Methodsmentioning

confidence: 99%

“…In conjunction with the concurrent work by Kowal, 46 we introduce two complementary notions of proximity in predictive performance, which are made precise below. First, let

η \geq 0

define a margin for acceptable performance: any model with predictive performance within

η

of the best model is considered acceptable.…”

Section: Methodsmentioning

confidence: 99%

ℐ_{k}

, we average () over all

K

testing sets:

{MSE}_{λ}^{o u t} = \frac{1}{K} \sum_{k = 1}^{K} {MSE}_{λ}^{o u t} (k), {\tilde{MSE}}_{λ}^{o u t} = \frac{1}{K} \sum_{k = 1}^{K} {\tilde{MSE}}_{λ}^{o u t} (k),

which is advocated in the literature 46,47 . When the holdout sets are mutually exclusive, each observation

i = 1, \dots, n

appears exactly once in ().…”

Section: Methodsmentioning

confidence: 99%

“…Next, consider the out‐of‐sample predictive variable

{\tilde{gapMSE}}_{λ}^{o u t} = {\tilde{MSE}}_{λ}^{o u t} - {\tilde{MSE}}_{λ_{m i n}}^{o u t},

which is the gap between model

λ

and the best model

λ_{m i n}

. The set of acceptable models 46 is

Λ_{η, ε} = {λ : ℙ ({\tilde{gapMSE}}_{λ}^{o u t} \leq η σ^{2}) \geq ε} .

…”

Section: Methodsmentioning

confidence: 99%

“…The acceptable set of models in () is equivalently derived using posterior predictive intervals: 46

λ \in Λ_{η, ε}

if and only if there exists a

(1 - ε)

lower posterior predictive credible interval for

{\tilde{gapMSE}}_{λ}^{o u t} / σ^{2}

that includes

η

. Naturally, if the predictive interval for the increase in out‐of‐sample MSE excludes

η

, such as

η = 0

, then this choice of

λ

is deemed unacceptable and omitted from

Λ_{η, ε}

.…”

Section: Methodsmentioning

confidence: 99%

See 3 more Smart Citations

Bayesian variable selection for understanding mixtures in environmental exposures

Kowal

Bravo

Leong

et al. 2021

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

show abstract

Decisions, decisions, decisions in an uncertain environment

Cressie

2022

Environmetrics

View full text Add to dashboard Cite

Decision‐makers abhor uncertainty, and it is certainly true that the less there is of it the better. However, recognizing that uncertainty is part of the equation, particularly for deciding on environmental policy, is a prerequisite for making wise decisions. Even making no decision is a decision that has consequences, and using the presence of uncertainty as the reason for failing to act is a poor excuse. Statistical science is the science of uncertainty, and it should play a critical role in the decision‐making process. This opinion piece focuses on the summit of the knowledge pyramid that starts from data and rises in steps from data to information, from information to knowledge, and finally from knowledge to decisions. Enormous advances have been made in the last 100 years ascending the pyramid, with deviations that have followed different routes. There has generally been a healthy supply of uncertainty quantification along the way but, in a rush to the top, where the decisions are made, uncertainty is often left behind. In my opinion, statistical science needs to be much more pro‐active in evolving classical decision theory into a relevant and practical area of decision applications. This article follows several threads, building on the decision‐theoretic foundations of loss functions and Bayesian uncertainty.

show abstract

Ultra-Efficient MCMC for Bayesian Longitudinal Functional Data Analysis

Sun,

Kowal

2024

Journal of Computational and Graphical Statistics

View full text Add to dashboard Cite

Fast, Optimal, and Targeted Predictions Using Parameterized Decision Analysis

Cited by 15 publications

References 30 publications

Bayesian variable selection for understanding mixtures in environmental exposures

Bayesian variable selection for understanding mixtures in environmental exposures

Decisions, decisions, decisions in an uncertain environment

Ultra-Efficient MCMC for Bayesian Longitudinal Functional Data Analysis

Contact Info

Product

Resources

About