The Sequential Bootstrap

Pathak, Pramod K.; Rao, C. R.

doi:10.1016/b978-0-444-53859-8.00001-1

Cited by 2 publications

(2 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Though the bagged models nominally include more participants, resampling is performed with replacement. Thus, only 63.2%, or 253 out of 400, are unique participants in a given bootstrap on average ( Pathak and Rao, 2013 ). On the other hand, the subsample 200 and 300 models are created without replacement, and therefore only contain unique participants.…”

Section: Resultsmentioning

confidence: 99%

Resample aggregating improves the generalizability of connectome predictive modeling

et al. 2021

View full text Add to dashboard Cite

It is a longstanding goal of neuroimaging to produce reliable, generalizable models of brain behavior relationships. More recently, data driven predictive models have become popular. However, overfitting is a common problem with statistical models, which impedes model generalization. Cross validation (CV) is often used to estimate expected model performance within sample. Yet, the best way to generate brain behavior models, and apply them out-of-sample, on an unseen dataset, is unclear. As a solution, this study proposes an ensemble learning method, in this case resample aggregating, encompassing both model parameter estimation and feature selection. Here we investigate the use of resampled aggregated models when used to estimate fluid intelligence (fIQ) from fMRI based functional connectivity (FC) data. We take advantage of two large openly available datasets, the Human Connectome Project (HCP), and the Philadelphia Neurodevelopmental Cohort (PNC). We generate aggregated and non-aggregated models of fIQ in the HCP, using the Connectome Prediction Modelling (CPM) framework. Over various test-train splits, these models are evaluated in sample, on left-out HCP data, and out-of-sample, on PNC data. We find that a resample aggregated model performs best both within- and out-of-sample. We also find that feature selection can vary substantially within-sample. More robust feature selection methods, as detailed here, are needed to improve cross sample performance of CPM based brain behavior models.

show abstract

Section: Resultsmentioning

confidence: 99%

Resample aggregating improves the generalizability of connectome predictive modeling

et al. 2021

View full text Add to dashboard Cite

show abstract

“…This method may not result in samples that are equally informative [50]. The sample size, following the sequential bootstrap resampling scheme, is kept as a constant, which is equal to the preassigned value 0.632n [51]. To select the sample,…”

Section: Confidence Intervalmentioning

confidence: 99%

Probabilistic Evaluation of the Multicategory Seasonal Precipitation Re-Forecast

2022

Meteorology

View full text Add to dashboard Cite

The Meteo-France seasonal forecasting system 7 provides a 7-month forecast range with 25 ensembles. The seasonal precipitation re-forecast (from May to November 1993–2015) was evaluated by the Brier score in terms of accuracy and reliability based on tercile probabilities. Multiple analyses were performed to assess the robustness of the score. These results show that the spatial distribution of the Brier score depends significantly on tercile thresholds, reference data, sampling methods, and ensemble types. Large probabilistic errors over the dry regions on land and the Nino regions in the Pacific can be reduced by adjusting the tercile thresholds. The forecast errors were identified when they were insensitive to different analysis methods. All the analyses detected that the errors increase/decrease with the lead time over the tropical Indian/Pacific Ocean. The intra-seasonal analysis reveals that some of these errors are inherited from monthly forecasts, which may be related to large-scale, short-term variability modes. A new confidence interval calculation was formulated for the “uncertain” case in the reference data. The confidence interval at a 95% level for the mean Brier score over the entire tropical region was quantified. The best estimations are ~6% the mean Brier score for both the above and below-normal terciles.

show abstract

The Sequential Bootstrap

Cited by 2 publications

References 28 publications

Resample aggregating improves the generalizability of connectome predictive modeling

Resample aggregating improves the generalizability of connectome predictive modeling

Probabilistic Evaluation of the Multicategory Seasonal Precipitation Re-Forecast

Contact Info

Product

Resources

About