Sequential Selection Procedures and False Discovery Rate Control

G’Sell, Max; Wager, Stefan; Chouldechova, Alexandra; Tibshirani, Robert

doi:10.1111/rssb.12122

Cited by 112 publications

(90 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ongoing development, most notably in LASSO and RF, may further improve model performance. Recent research on LASSO includes alternative selection methods (Belloni, Chernozhukov & Wang 2011;G'Sell et al 2013;Lederer & M€ uller 2015) Despite our general optimism about smoothing methods, we did find some important limitations. First, our ability to identify climate drivers is limited by the number of independent observations of climate history.…”

Section: Discussionmentioning

confidence: 70%

Linking demography with drivers: climate and competition

et al. 2016

View full text Add to dashboard Cite

1. In observational demographic data, the number of measured factors that could potentially drive demography (such as daily weather records between two censuses) can easily exceed the number of independent observations. Thus, identifying the important drivers requires alternatives to standard model selection and variable selection methods.2. Spline methods that estimate smooth functions over continuous domains (such as space or time) have the potential to resolve high-dimensional problems in ecological systems. We consider two examples that are important for many plant populations: competition with neighbours that vary in size and distance from the focal individual and climate variables during a window of time before a response (growth, survival, etc.) is measured. 3. For competition covariates, we use a simulation study based on empirical data to show that a monotone spline estimate of competition kernels via approximate AIC returns very accurate estimates. We then apply the method to long-term, mapped quadrat data on the four dominant species in an Idaho (US) sagebrush steppe community. 4. For climate predictors and their temporal lags, we use simulated data sets to compare functional smoothing methods with competing linear (LASSO) or machine learning (random forests) methods. Given sufficient data, functional smoothing methods outperformed the other two methods. 5. Functional smoothing methods can advance data-driven population modelling by providing alternatives to specifying competition kernels a priori and to arbitrarily aggregating continuous environmental covariates. However, there are important open questions related to modelling of nonlinear climate responses and size 9 climate interactions.

show abstract

Section: Discussionmentioning

confidence: 70%

Linking demography with drivers: climate and competition

et al. 2016

View full text Add to dashboard Cite

show abstract

“…• Covariance testing (Lockhart et al, 2014), which we use in conjunction with the forward stopping rules proposed by G'Sell et al (2016) to control the pathwise false discovery rate at 10%.…”

Section: Comparison With Other Methodsmentioning

confidence: 99%

“…Here, we limit the first‐stage selections to 20 variables so that in the second stage, the model contains 10 events per variable (Peduzzi, Concato, Feinstein, & Holford, ). Covariance testing (Lockhart et al., ), which we use in conjunction with the forward stopping rules proposed by G'Sell et al. () to control the pathwise false discovery rate at 10%. Univariate testing, where we fit separate, unpenalized regression models to each variable individually, and then adjust the resulting P ‐values using the Benjamini‐Hochberg procedure to control the false discovery rate at 10%. Cross‐validation (CV), where 10‐fold CV is used to choose λ for the lasso model; note that this approach makes no attempt to control the false discovery rate.…”

Section: Simulation Studiesmentioning

confidence: 99%

“…More recently, Lockhart, Taylor, Tibshirani, and Tibshirani (2014) and Tibshirani, Taylor, Lockhart, and Tibshirani (2016) have proposed a family of methods known as selective inference, which test the significance of each variable selection along the lasso solution path as is decreased, conditional upon the other variables already active in the model. Based on this sequence of tests, formal stopping rules can be derived in order to control the false discovery rate at a specified level (G'Sell, Wager, Chouldechova, & Tibshirani, 2016). Several other approaches exist -for example, procedures based on the bootstrap (Dezeure, Bühlmann, & Zhang, 2017) or the idea of debiasing (Javanmard & Montanari, 2014).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Marginal false discovery rate control for likelihood‐based penalized regression models

Miller

Breheny

2019

Biometrical J

View full text Add to dashboard Cite

The popularity of penalized regression in high‐dimensional data analysis has led to a demand for new inferential tools for these models. False discovery rate control is widely used in high‐dimensional hypothesis testing, but has only recently been considered in the context of penalized regression. Almost all of this work, however, has focused on lasso‐penalized linear regression. In this paper, we derive a general method for controlling the marginal false discovery rate that can be applied to any penalized likelihood‐based model, such as logistic regression and Cox regression. Our approach is fast, flexible and can be used with a variety of penalty functions including lasso, elastic net, MCP, and MNet. We derive theoretical results under which the proposed method is valid, and use simulation studies to demonstrate that the approach is reasonably robust, albeit slightly conservative, when these assumptions are violated. Despite being conservative, we show that our method often offers more power to select causally important features than existing approaches. Finally, the practical utility of the method is demonstrated on gene expression datasets with binary and time‐to‐event outcomes.

show abstract

“…Recognizing these limitations, several methods have been proposed to reduce the model space, although this reduction might lead to a lack of interpretability and usability [7]. Some authors have proposed rules for shrinkage methods, suggesting that one could safely evaluate problems when there are at most 10 to 15 times more variables than observations, while others have established criteria to delete variables, for example, Least Absolute Shrinkage and Selection Operator (LASSO) [11, 16–18]. …”

Section: Introductionmentioning

confidence: 99%

FARMS: A New Algorithm for Variable Selection

Pérez-Álvarez

Gómez

Berthou

2015

BioMed Research International

View full text Add to dashboard Cite

Large datasets including an extensive number of covariates are generated these days in many different situations, for instance, in detailed genetic studies of outbreed human populations or in complex analyses of immune responses to different infections. Aiming at informing clinical interventions or vaccine design, methods for variable selection identifying those variables with the optimal prediction performance for a specific outcome are crucial. However, testing for all potential subsets of variables is not feasible and alternatives to existing methods are needed. Here, we describe a new method to handle such complex datasets, referred to as FARMS, that combines forward and all subsets regression for model selection. We apply FARMS to a host genetic and immunological dataset of over 800 individuals from Lima (Peru) and Durban (South Africa) who were HIV infected and tested for antiviral immune responses. This dataset includes more than 500 explanatory variables: around 400 variables with information on HIV immune reactivity and around 100 individual genetic characteristics. We have implemented FARMS in R statistical language and we showed that FARMS is fast and outcompetes other comparable commonly used approaches, thus providing a new tool for the thorough analysis of complex datasets without the need for massive computational infrastructure.

show abstract

Sequential Selection Procedures and False Discovery Rate Control

Cited by 112 publications

References 34 publications

Linking demography with drivers: climate and competition

Linking demography with drivers: climate and competition

Marginal false discovery rate control for likelihood‐based penalized regression models

FARMS: A New Algorithm for Variable Selection

Contact Info

Product

Resources

About