Validating digital phenotyping technologies for clinical use: the critical importance of “resolution”

Cohen, Alex S.; Schwartz, Elana K.; Le, Thanh P.; Cowan, Tovah; Cox, Christopher R.; Tucker, Raymond; Foltz, Peter W.; Holmlund, Terje B.; Elvevåg, Brita

doi:10.1002/wps.20703

Cited by 43 publications

(27 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Given that BvA and alogia reflect overt behaviors that can be quantified, it has long been proposed that computerized acoustic analysis could be used to measure them [9][10][11][12] . Presumably, this type of "digital phenotyping" 13 could be automated to provide a relatively efficient and sensitive "state" measure of negative symptoms; with applications for improving diagnostic accuracy and for efficiently tracking symptom severity, relapse risk, treatment response, and pharmacological side effects [14][15][16] . Moreover, acoustic analysis is based on speech analysis technologies that are freely available, well-validated for a variety of applications, and can be collected using a wide range of unobtrusive and in situ remote technologies (e.g., smartphones, archived videos) 17 .…”

Section: Introductionmentioning

confidence: 99%

Using machine learning of computerized vocal expression to measure blunted vocal affect and alogia

Cohen

Cox

et al. 2020

npj Schizophr

Self Cite

View full text Add to dashboard Cite

Negative symptoms are a transdiagnostic feature of serious mental illness (SMI) that can be potentially “digitally phenotyped” using objective vocal analysis. In prior studies, vocal measures show low convergence with clinical ratings, potentially because analysis has used small, constrained acoustic feature sets. We sought to evaluate (1) whether clinically rated blunted vocal affect (BvA)/alogia could be accurately modelled using machine learning (ML) with a large feature set from two separate tasks (i.e., a 20-s “picture” and a 60-s “free-recall” task), (2) whether “Predicted” BvA/alogia (computed from the ML model) are associated with demographics, diagnosis, psychiatric symptoms, and cognitive/social functioning, and (3) which key vocal features are central to BvA/Alogia ratings. Accuracy was high (>90%) and was improved when computed separately by speaking task. ML scores were associated with poor cognitive performance and social functioning and were higher in patients with schizophrenia versus depression or mania diagnoses. However, the features identified as most predictive of BvA/Alogia were generally not considered critical to their operational definitions. Implications for validating and implementing digital phenotyping to reduce SMI burden are discussed.

show abstract

Section: Introductionmentioning

confidence: 99%

Using machine learning of computerized vocal expression to measure blunted vocal affect and alogia

Cohen

Cox

et al. 2020

npj Schizophr

Self Cite

View full text Add to dashboard Cite

show abstract

“…As a result, it can be challenging to empirically evaluate the impact of extraneous factors and thus demonstrate robust internal validity, especially because there can be substantial intraindividual variability in many important symptoms and behaviors. One solution is to exploit the ability of these tools to capture high-resolution data [22]. High intraindividual variability can inflate the sample SD; increasing the number of data points per participant can increase the precision of estimates and improve statistical power [71][72][73][74].…”

Section: How To Validate High-frequency Assessments For Remote Researchmentioning

confidence: 99%

“…This increased level of detail can be incorporated into the statistical analyses to improve the signal-to-noise ratio. As a result, high-frequency assessments can be used to achieve more representative baselines and develop novel digital phenotypes to improve the precision and accuracy of diagnosis and outcomes [ 22 ]. It should be noted that brief assessments do not necessarily have to be administered at high frequencies to be valid measurement tools.…”

Section: Determining the Suitability Of An Assessment For Remote Researchmentioning

confidence: 99%

Developing Digital Tools for Remote Clinical Research: How to Evaluate the Validity and Practicality of Active Assessments in Field Settings

Ferrar¹,

Griffith²,

Skirrow³

et al. 2021

J Med Internet Res

View full text Add to dashboard Cite

The ability of remote research tools to collect granular, high-frequency data on symptoms and digital biomarkers is an important strength because it circumvents many limitations of traditional clinical trials and improves the ability to capture clinically relevant data. This approach allows researchers to capture more robust baselines and derive novel phenotypes for improved precision in diagnosis and accuracy in outcomes. The process for developing these tools however is complex because data need to be collected at a frequency that is meaningful but not burdensome for the participant or patient. Furthermore, traditional techniques, which rely on fixed conditions to validate assessments, may be inappropriate for validating tools that are designed to capture data under flexible conditions. This paper discusses the process for determining whether a digital assessment is suitable for remote research and offers suggestions on how to validate these novel tools.

show abstract

“…MLM can accommodate data with a "nested" structure, such that observations are hierarchically organized within individuals, settings, and times/days. MLM, for example, has been used to understand how acoustics and facial features change as a function of time, social context and setting in schizophrenia (24,(84)(85)(86).…”

Section: How Should Resolution Be Addressed?mentioning

confidence: 99%

Validating Biobehavioral Technologies for Use in Clinical Psychiatry

et al. 2021

Self Cite

View full text Add to dashboard Cite

The last decade has witnessed the development of sophisticated biobehavioral and genetic, ambulatory, and other measures that promise unprecedented insight into psychiatric disorders. As yet, clinical sciences have struggled with implementing these objective measures and they have yet to move beyond “proof of concept.” In part, this struggle reflects a traditional, and conceptually flawed, application of traditional psychometrics (i.e., reliability and validity) for evaluating them. This paper focuses on “resolution,” concerning the degree to which changes in a signal can be detected and quantified, which is central to measurement evaluation in informatics, engineering, computational and biomedical sciences. We define and discuss resolution in terms of traditional reliability and validity evaluation for psychiatric measures, then highlight its importance in a study using acoustic features to predict self-injurious thoughts/behaviors (SITB). This study involved tracking natural language and self-reported symptoms in 124 psychiatric patients: (a) over 5–14 recording sessions, collected using a smart phone application, and (b) during a clinical interview. Importantly, the scope of these measures varied as a function of time (minutes, weeks) and spatial setting (i.e., smart phone vs. interview). Regarding reliability, acoustic features were temporally unstable until we specified the level of temporal/spatial resolution. Regarding validity, accuracy based on machine learning of acoustic features predicting SITB varied as a function of resolution. High accuracy was achieved (i.e., ~87%), but only when the acoustic and SITB measures were “temporally-matched” in resolution was the model generalizable to new data. Unlocking the potential of biobehavioral technologies for clinical psychiatry will require careful consideration of resolution.

show abstract

Validating digital phenotyping technologies for clinical use: the critical importance of “resolution”

Cited by 43 publications

References 11 publications

Using machine learning of computerized vocal expression to measure blunted vocal affect and alogia

Using machine learning of computerized vocal expression to measure blunted vocal affect and alogia

Developing Digital Tools for Remote Clinical Research: How to Evaluate the Validity and Practicality of Active Assessments in Field Settings

Validating Biobehavioral Technologies for Use in Clinical Psychiatry

Contact Info

Product

Resources

About