Using and understanding cross-validation strategies. Perspectives on Saeb et al.

Little, Max A.; Varoquaux, Gaël; Saeb, Sohrab; Lonini, Luca; Jayaraman, Arun; Mohr, David C.; Kording, Konrad

doi:10.1093/gigascience/gix020

Cited by 107 publications

(91 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In both the train and test set the ridge regression was the most predictive model (post-hoc analysis will show how model prediction was affected by our sample size). The test set may be lower because cross-validation is not a panacea for overfitting as cross-validation still capitalizes on stochastic error 32 .…”

Section: Phenotypic Predictionmentioning

confidence: 99%

Inferring the Genetic Influences on Psychological Traits Using MRI Connectivity Predictive Models: Demonstration with Cognition

Hatoum

Reineberg

Kragel

et al. 2019

Preprint

View full text Add to dashboard Cite

Genetic correlations between brain and behavioral phenotypes in analyses from major genetic consortia have been weak and mostly non-significant. fMRI models of systems-level brain patterns may help improve our ability to link genes, brains, and behavior by identifying reliable and reproducible endophenotypes. Work using connectivity-based predictive modeling (CBPM) has generated brain-based proxies of behavioral and neuropsychological variables. If such models capture activity in inherited brain systems, they may offer a more powerful link between genes and behavior. As a proof of concept, we develop models predicting intelligence (IQ) based on fMRI connectivity and test their effectiveness as endophenotypes. We link brain and IQ in a model development dataset of N=3,000 individuals; and test the genetic correlations between brain models and measured IQ in a genetic validation sample of N=13,092 individuals from the UKBiobank. We compare an additive connectivity-based model to multivariate LASSO and ridge models phenotypically and genetically. We also compare these approaches to single "candidate" brain areas. We find that predictive brain models were significantly phenotypically correlated with IQ and showed much stronger correlations than individual edges. Further, brain models were more heritable than single brain regions (h 2 =.155-.181) and capture about half of the genetic variance in IQ (rG=.422-.576), while rGs with single brain measures were smaller and non-significant. For the different approaches, LASSO and Ridge were similarly predictive, with slightly weaker performance of the additive model. LASSO model weights were highly theoretically interpretable and replicated known brain IQ associations. Finally, functional connectivity models trained in midlife showed genetic correlations with early life correlates of IQ, suggesting some stability in the prediction of fMRI models. We conclude that multi-system predictive models hold promise as imaging endophenotypes that offer complex and theoretically relevant conclusions for future imaging genetics research.

show abstract

Section: Phenotypic Predictionmentioning

confidence: 99%

Inferring the Genetic Influences on Psychological Traits Using MRI Connectivity Predictive Models: Demonstration with Cognition

Hatoum

Reineberg

Kragel

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The pdfs shown use different band‐widths, each computed from the associated delimiter placements through multiple iterations of leave‐subject‐out Monte Carlo cross‐validation (CV), utilizing a train‐test split of 90% to 10%. Leave‐subject‐out CV is an established blocked CV approach with theoretic optimality that accounts for dependencies within subject responses [XH12, SLJ∗17, RBC∗17, LVS∗17]. Peaks in the resulting pdfs highlight consistencies across participants' placed delimiters.…”

Section: Resultsmentioning

confidence: 99%

Examining Implicit Discretization in Spectral Schemes

Quinan

Padilla

Creem-Regehr

et al. 2019

Computer Graphics Forum

View full text Add to dashboard Cite

Two of the primary reasons rainbow color maps are considered ineffective trace back to the idea that they implicitly discretize encoded data into hue‐based bands, yet no research addresses what this discretization looks like or how consistent it is across individuals. This paper presents an exploratory study designed to empirically investigate the implicit discretization of common spectral schemes and explore whether the phenomenon can be modeled by variations in lightness, chroma, and hue. Our results suggest that three commonly used rainbow color maps are implicitly discretized with consistency across individuals. The results also indicate, however, that this implicit discretization varies across different datasets, in a way that suggests the visualization community's understanding of both rainbow color maps, and more generally effective color usage, remains incomplete.

show abstract

“…Unfortunately, in this case, we do not believe leave-one-out subject-wise cross validation does not address identity confounding [23], although it is commonly used for this purpose with health care-related data. We believe identity confounding is best addressed by larger data sets representing more diversity in data requiring significant improvement in how current clinical data collection is achieved.…”

Section: Discussionmentioning

confidence: 99%

“…These recordings were chosen at random from each subject. We did not choose to use leave-one-subject-out cross validation in order to incorporate more of the data due to the concerns of within subject variation in data, which is well known in similar data sets [23]. Particularly with voice features, there tends to be large variation in features within subjects which violates the primary assumption of within subject consistency at the core of leave-one-subject-out cross validation.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Depression Screening from Voice Samples of Patients Affected by Parkinson’s Disease

et al. 2019

View full text Add to dashboard Cite

Depression is a common mental health problem leading to significant disability worldwide. It is not only common but also commonly co-occurs with other mental and neurological illnesses. Parkinson’s disease (PD) gives rise to symptoms directly impairing a person’s ability to function. Early diagnosis and detection of depression can aid in treatment, but diagnosis typically requires an interview with a health provider or a structured diagnostic questionnaire. Thus, unobtrusive measures to monitor depression symptoms in daily life could have great utility in screening depression for clinical treatment. Vocal biomarkers of depression are a potentially effective method of assessing depression symptoms in daily life, which is the focus of the current research. We have a database of 921 unique PD patients and their self-assessment of whether they felt depressed or not. Voice recordings from these patients were used to extract paralinguistic features, which served as inputs to machine learning and deep learning techniques to predict depression. The results are presented here, and the limitations are discussed given the nature of the recordings which lack language content. Our models achieved accuracies as high as 0.77 in classifying depressed and nondepressed subjects accurately using their voice features and PD severity. We found depression and severity of PD had a correlation coefficient of 0.3936, providing a valuable feature when predicting depression from voice. Our results indicate a clear correlation between feeling depressed and PD severity. Voice may be an effective digital biomarker to screen for depression among PD patients.

show abstract

Using and understanding cross-validation strategies. Perspectives on Saeb et al.

Cited by 107 publications

References 16 publications

Inferring the Genetic Influences on Psychological Traits Using MRI Connectivity Predictive Models: Demonstration with Cognition

Inferring the Genetic Influences on Psychological Traits Using MRI Connectivity Predictive Models: Demonstration with Cognition

Examining Implicit Discretization in Spectral Schemes

Depression Screening from Voice Samples of Patients Affected by Parkinson’s Disease

Contact Info

Product

Resources

About