Post model‐fitting exploration via a “Next‐Door” analysis

Abstract

We propose a simple method for evaluating the model that has been chosen by an adaptive regression procedure, our main focus being the lasso. This procedure deletes each chosen predictor and refits the lasso to get a set of models that are "close" to the one chosen, referred to as "base model". If the deletion of a predictor leads to significant deterioration in the model's predictive power, the predictor is called indispensable; otherwise, the nearby model is called acceptable and can serve as a good alternative to the base model. This provides both an assessment of the predictive contribution of each variable and a set of alternative models that may be used in place of the chosen model.In this paper, we will focus on the cross-validation (CV) setting and a model's predictive power is measured by its CV error, with base model tuned by cross-validation. We propose a method for comparing the error rates of the base model with that of nearby models, and a p-value for testing whether a predictor is dispensable. We also propose a new quantity called model score which works similarly as the p-value for the control of type I error. Our proposal is closely related to the LOCO (leave-one-covarate-out) methods of (Rinaldo et al. (2016)) and less so, to Stability Selection (Meinshausen & Bühlmann (2010)).We call this procedure "Next-Door analysis" since it examines models close to the base model. It can be applied to Gaussian regression data, generalized linear models, and other supervised learning problems with 1 penalization. It could also be applied to best subset and stepwise regression procedures. We have implemented it in the R language as a library to accompany the well-known glmnet library. *

ctDNA Predicts Overall Survival in Patients With NSCLC Treated With PD-L1 Blockade or With Chemotherapy

“…Feature importance was measured as how often a feature was retained in a lasso model (frequency) and improvement on model fitting from next-door analysis (gain). 33 All P values are two-sided without multiple testing correction.…”

A longitudinal circulating tumor DNA-based model associated with survival in metastatic non-small-cell lung cancer

“…Feature interaction was addressed in survival random forest modeling (R package Nature Medicine Article https://doi.org/10.1038/s41591-023-02226-6 randomForestSRC v 2.9.3) with default parameters, but no improvements in model performance were detected. Feature importance was assessed as the number of cross-validations a feature was retained, and by the 'Gain' metric as assessed by the average worsening statistic from the next-door analysis 40 across LOOCV. Model performance, measured by c-index, was estimated after pooling all LOOCV predictions together to reconstruct the original training dataset.…”

Towards a Better Understanding of Linear Models for Recommendation

“…Furthermore, once the parameter searching algorithm is available, we consider to utilize it for searching nearby models for an existing model (Subsection 4.2). As we mentioned before, this can be considered as a post-model fitting exploration in statistical learning [12].…”

“…In this subsection, we consider how to further leverage the new learning procedure for other linear models to help identify the (hyper)parameters. Inspired by the recent efforts of post-model fitting exploration [12], we consider to augment the existing learned 𝑊 from any existing models (or adding on top of the aforementioned closed-form solution) with two types of parameters:…”

“…To help further understand how the singular values of user-item interaction matrix can be adjusted (at individual level), we introduce a novel learning algorithm which can search through high dimensional continuous (hyper)parameter space. This learning algorithm also enables us to perform the post-model fitting exploration [12] for existing linear recommendation models. Our approach is to augment (existing) linear models with additional parameters to help further improve the model accuracy.…”