Assessing the accuracy and stability of variable selection methods for random forest modeling in ecology

Abstract

Random forest (RF) modeling has emerged as an important statistical learning method in ecology due to its exceptional predictive performance. However, for large and complex ecological data sets, there is limited guidance on variable selection methods for RF modeling. Typically, either a preselected set of predictor variables are used or stepwise procedures are employed which iteratively remove variables according to their importance measures. This paper investigates the application of variable selection methods to RF models for predicting probable biological stream condition. Our motivating data set consists of the good/poor condition of n = 1365 stream survey sites from the 2008/2009 National Rivers and Stream Assessment, and a large set (p = 212) of landscape features from the StreamCat data set as potential predictors. We compare two types of RF models: a full variable set model with all 212 predictors and a reduced variable set model selected using a backward elimination approach. We assess model accuracy using RF's internal out-of-bag estimate, and a cross-validation procedure with validation folds external to the variable selection process. We also assess the stability of the spatial predictions generated by the RF models to changes in the number of predictors and argue that model selection needs to consider both accuracy and stability. The results suggest that RF modeling is robust to the inclusion of many variables of moderate to low importance. We found no substantial improvement in cross-validated accuracy as a result of variable reduction. Moreover, the backward elimination procedure tended to select too few variables and exhibited numerous issues such as upwardly biased out-of-bag accuracy estimates and instabilities in the spatial predictions. We use simulations to further support and generalize results from the analysis of real data. A main purpose of this work is to elucidate issues of model selection bias and instability to ecologists interested in using RF to develop predictive models with large environmental data sets.

Predictive mapping of the biotic condition of conterminous U.S. rivers and streams

“…Fox et al. () found that variable selection did not substantially affect model performance, relative to the model decisions explored here. Rather, variable selection tended to introduce instability in predicted probabilities (Fox et al.…”

“…, Fox et al. ). Development of CONUS‐wide predictions of biological condition required several key decisions that are not normally described in papers using RF.…”

“…Fox et al. () tested a multinomial RF model of condition with good, fair, and poor classes. They found that the model could only correctly predict the condition of 24% of fair sites (see Supplement 2 of Fox et al.…”

“…A companion paper to this study by Fox et al. () provides an examination of variable reduction with RFs. Fox et al.…”

“…See Fox et al. () for additional details on model development, evaluation, and selection. Although other studies have focused primarily on model development and the interpretation of the response variable to predictors (e.g., Carlisle et al.…”