A Simple and Effective Model-Based Variable Importance Measure

Greenwell, Brandon; Boehmke, Bradley C.; McCarthy, Andrew J.

doi:10.48550/arxiv.1805.04755

Cited by 54 publications

(79 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This method is based on partial dependency of input features. Essentially, it can be derived from PDPs that input features that have more variability in their PDP, are more influential in the final predictions made by the ML model (Greenwell, 2018). Consequently, the features for which the PDP is flat is likely to be less important than input variables with more variable PDP across range of their values.…”

Section: Feature Importancementioning

confidence: 99%

Forecasting Corn Yield With Machine Learning Ensembles

2020

View full text Add to dashboard Cite

The emergence of new technologies to synthesize and analyze big data with highperformance computing has increased our capacity to more accurately predict crop yields. Recent research has shown that machine learning (ML) can provide reasonable predictions faster and with higher flexibility compared to simulation crop modeling. However, a single machine learning model can be outperformed by a "committee" of models (machine learning ensembles) that can reduce prediction bias, variance, or both and is able to better capture the underlying distribution of the data. Yet, there are many aspects to be investigated with regard to prediction accuracy, time of the prediction, and scale. The earlier the prediction during the growing season the better, but this has not been thoroughly investigated as previous studies considered all data available to predict yields. This paper provides a machine leaning based framework to forecast corn yields in three US Corn Belt states (Illinois, Indiana, and Iowa) considering complete and partial inseason weather knowledge. Several ensemble models are designed using blocked sequential procedure to generate out-of-bag predictions. The forecasts are made in county-level scale and aggregated for agricultural district and state level scales. Results show that the proposed optimized weighted ensemble and the average ensemble are the most precise models with RRMSE of 9.5%. Stacked LASSO makes the least biased predictions (MBE of 53 kg/ha), while other ensemble models also outperformed the base learners in terms of bias. On the contrary, although random k-fold cross-validation is replaced by blocked sequential procedure, it is shown that stacked ensembles perform not as good as weighted ensemble models for time series data sets as they require the data to be non-IID to perform favorably. Comparing our proposed model forecasts with the literature demonstrates the acceptable performance of forecasts made by our proposed ensemble model. Results from the scenario of having partial in-season weather knowledge reveals that decent yield forecasts with RRMSE of 9.2% can be made as early as June 1 st. Moreover, it was shown that the proposed model performed better than individual models and benchmark ensembles at agricultural district and statelevel scales as well as county-level scale. To find the marginal effect of each input feature on the forecasts made by the proposed ensemble model, a methodology is suggested that is the basis for finding feature importance for the ensemble model. The findings

show abstract

Section: Feature Importancementioning

confidence: 99%

Forecasting Corn Yield With Machine Learning Ensembles

2020

View full text Add to dashboard Cite

show abstract

“…A bivariate importance measure, perhaps obtained by permuting pairs of variables, could be used in place of the H-statistic in the heatmap and network visualizations. It would also be interesting to explore the interaction measures of Hooker (2004) and Greenwell et al (2018) in our visualizations, and whether these measures avoid the issues identified with the use of H.…”

Section: Discussionmentioning

confidence: 99%

Visualizing Variable Importance and Variable Interaction Effects in Machine Learning Models

Inglis¹,

Parnell²,

Hurley³

2021

Preprint

View full text Add to dashboard Cite

Variable importance, interaction measures, and partial dependence plots are important summaries in the interpretation of statistical and machine learning models. In this paper we describe new visualization techniques for exploring these model summaries. We construct heatmap and graph-based displays showing variable importance and interaction jointly, which are carefully designed to highlight important aspects of the fit. We describe a new matrix-type layout showing all single and bivariate partial dependence plots, and an alternative layout based on graph Eulerians focusing on key subsets. Our new visualizations are model-agnostic and are applicable to regression and classification supervised learning settings. They enhance interpretation even in situations where the number of variables is large. Our R package vivid (variable importance and variable interaction displays) provides an implementation.

show abstract

“…It is also useful to summarize the main and interaction ALEs with a one-number summary that can be used to rank the importance of each effect. Following Greenwell et al (2018), we propose to measure overall variable importance (VI) for continuous covariates using the standard deviation of the ALE with respect to the marginal distribution of X, i.e.,…”

Section: I-spline Basis Expansionmentioning

confidence: 99%

Bayesian Non-parametric Quantile Process Regression and Estimation of Marginal Quantile Effects

Xu,

Reich

2021

Preprint

View full text Add to dashboard Cite

We propose a non-parametric method to simultaneously estimate non-crossing, non-linear quantile curves. We expand the conditional distribution function of the response in I-spline basis functions where the coefficients are further modeled as functions of the covariates using feed-forward neural networks. By leveraging the approximation power of splines and neural networks, our model can approximate any continuous quantile function. Compared to existing methods, our method estimates all rather than a finite subset of quantiles, scales well to high dimensions and accounts for estimation uncertainty. While the model is arbitrarily flexible, interpretable marginal quantile effects are estimated using accumulative local effect plots and variable importance measures. A simulation study shows that compared to existing methods, our model can better recover quantiles of the response distribution when the sample size is small, and illustrative applications to birth weight and tropical cyclone intensity are presented.

show abstract

A Simple and Effective Model-Based Variable Importance Measure

Cited by 54 publications

References 16 publications

Forecasting Corn Yield With Machine Learning Ensembles

Forecasting Corn Yield With Machine Learning Ensembles

Visualizing Variable Importance and Variable Interaction Effects in Machine Learning Models

Bayesian Non-parametric Quantile Process Regression and Estimation of Marginal Quantile Effects

Contact Info

Product

Resources

About