Smartphone video nystagmography using convolutional neural networks: ConVNG

Friedrich, M; Schneider, Erich; Buerklein, Miriam; Taeger, Johannes; Hartig, Johannes; Volkmann, Jens; Peach, Robert L.; Zeller, Daniel

doi:10.1007/s00415-022-11493-1

Cited by 12 publications

(42 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Whilst adaptions to TWSTRS have incorporated tremor related features [40, 41], there is still a growing demand for more objective and granular disease metrics [3, 18]. Already widely used for quantitative phenotyping in experimental neuroscience, computer vision approaches have recently emerged as a promising new tool for clinical assessments [20, 23, 24, 42].…”

Section: Discussionmentioning

confidence: 99%

“…The generalised dystonia cohort, used for independent validation, comprised 30 patients from the same centres. Individual datasets were included if (i) they contained at least one pre-operative clinical rating video showing the full dystonic phenotype and a video from the chronic postoperative phase (3-15 months post surgery) documenting the effects of clinically programmed DBS and (ii) both videos fulfilled minimal criteria ensuring video quality, which were chosen to reflect the current best practice in clinical computer vision approaches [20, 23, 24]. These were: (i) front view perspective of a single individual sitting on a chair, (ii) no significantly obscuring items on patients (e. g. excessive head dressings with externalised DBS device), (ii) no excessive camera movements, variable zoom depths or lighting insufficient to identify typical body landmarks (e.g., eyes), (iii) continuous presence of head and neck in the camera frame.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Quantitative assessment of head movement dynamics in dystonia using visual perceptive deep learning: a multi-centre retrospective longitudinal cohort study

Peach,

Friedrich,

Fronemann

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

BackgroundDystonia is a neurological movement disorder characterised by abnormal involuntary movements and postures, particularly affecting the head and neck. However, current clinical assessment methods for dystonia rely on simplified rating scales which lack the ability to capture the intricate spatiotemporal features of dystonic phenomena, hindering clinical management and limiting understanding of the underlying neurobiology. To address this, we developed a visual perceptive deep learning framework that utilizes standard clinical videos to comprehensively evaluate and quantify disease states and the impact of therapeutic interventions, specifically deep brain stimulation. This framework overcomes the limitations of traditional rating scales and offers an efficient and accurate method that is rater-independent for evaluating and monitoring dystonia patients.MethodsTo evaluate the framework, we leveraged semi-standardized clinical video data collected in three retrospective, longitudinal cohort studies across seven academic centres in Germany. We extracted static head angle excursions for clinical validation and derived kinematic variables reflecting naturalistic head dynamics to predict dystonia severity, subtype, and neuromodulation effects. The framework was validated in a fully independent cohort of generalised dystonia patients.FindingsComputer vision-derived measurements of head angle excursions showed a strong correlation with clinically assigned scores, outperforming previous approaches employing specialised camera equipment. Across comparisons, we discovered a consistent set of kinematic features derived from full video assessments, which encoded information relevant to disease severity, subtype, and effects of neural circuit intervention more strongly and independently of static head angle deviations predominantly used for scoring.InterpretationThe proposed visual perceptive machine learning framework reveals kinematic pathosignatures of dystonia which may be utilized to augment clinical management, facilitate scientific translation and inform personalised and precision approaches in Neurology.Research in contextEvidence before this studyClinical assessment of dystonia, a neurological movement disorder, has traditionally relied on rating scales that aim to simplify complex phenomenology into lowerdimensional rating items. However, these score-based assessments have significant clinimetric limitations and do not fully capture the rich spatiotemporal dynamics of dystonic phenomena, which are crucial for clinical judgment and pathophysiological understanding. In contrast, recent investigations in animal models of dystonia have already demonstrated the utility and relevance of quantitative methods for phenotyping, which gradually supersedes previous observer-dependent behavioural analyses. Taken together, this has led to a need for more objective and detailed clinical evaluation methods of dystonia.We performed a PubMed search up to July 2023 combining the terms “dystonia” AND (”deep learning” OR “machine learning” or “computer vision” OR “vision-based” OR “video-based”) AND (”angle” OR “kinematic” OR “rating” OR “scoring” OR “movement analysis”) including abstracts in English or German. The search yielded three studies that validated vision-based frameworks for automating the assessment of cervical dystonia severity compared to clinician-annotated ratings. Two of these studies focused on deriving head angle deviations from specialised camera setups, while the third study utilised computer vision in a retrospective video dataset recorded using conventional equipment. These studies reported fair to moderately strong correlations between vision-based head angle measurements and clinical scores. Additionally, two studies investigated computer vision for assessing head tremor in the context of cervical dystonia: one single case report demonstrated the clinical validity of computer vision-derived head angle and head tremor metrics, while a retrospective cross-sectional study reported moderately strong clinical agreement of computer vision-derived head oscillation metrics across different dystonia subgroups. Two additional studies used computer visionbased kinematics to quantify dystonia-like phenomena in rodent models of monogenetic dystonia, demonstrating utility in both phenotype and genotype predictions.However, most of the clinical studies were limited to static task conditions, where patients attempted to hold a neutral position of the head, thus not providing a naturalistic account of dystonia. Moreover, beyond head angular deviations and oscillation metrics, no study explored a broader kinematic feature space that reflects the true spatiotemporal complexity of dystonic movements. Additionally, the studies assessed patients at single time points without considering different therapy conditions, particularly the effects of deep brain stimulation, which is a highly effective intervention targeting brain circuits. Nor did they compare dystonia sub-types, such as cervical and generalised systonia.Added value of this studyIn this study, we present a comprehensive visual perceptive deep learning framework that addresses the gaps in current dystonia assessments. We use this framework to retrospectively analyse a unique dataset from three multi-centric, studies encompassing video examinations of patients along the dystonic severity continuum, including different deep brain stimulation states. Our framework goes beyond the automation of suboptimal symptom severity assessments by reverse engineering a set of clinically inspired kinematic features. The resulting high dimensional, yet intuitively interpretable kinematic feature space enabled us to explore disease states and effects of brain circuit therapies in a level of detail comparable to experimental neuroscientific investigations. Through a data-driven approach, we have identified a consistent set of only four dynamic parameters that encode dystonia severity, subtype, and the efficacy of brain circuit interventions. Notably, these features are independent of static head angle deviations, which play a central role in dystonia severity scores, pointing to the involvement of partially distinct neurobiological processes not captured by these scores. Our findings align with emerging concepts of symptom-specific brain circuits and findings in rodent models of dystonia, thereby exemplifying the visual perceptive framework’s potential to augment clinical management and bridge translational gaps in movement disorders research. By providing a more comprehensive and precise assessment of the disorder, our study offers valuable insights for improved treatment strategies and further understanding of dystonia’s complex neurobiology.Implications of all the available evidenceThe available evidence collectively underscores the limitations of traditional rating scales in capturing the informative spatiotemporal dynamics of dystonic movements, emphasizing the need for more objective and granular evaluation methods. In line with recent animal studies using computer vision for dystonia quantification, recent clinical studies have shown the potential of computer vision-based frameworks in automating cervical dystonia severity assessment and capturing head tremor metrics. However, their underlying study designs may inadvertently reinforce limitations associated with the clinical scoring process.In this study, we introduce a comprehensive visual perceptive deep learning framework that serves as a powerful platform to augment clinical judgement and generate valuable pathophysiological insights by extracting a set of clinically inspired, interpretable kinematic features. Our findings have implications beyond dystonia, showcasing the utility of visual perceptive frameworks in enhancing clinical management and fostering integration with advanced neuroimaging and neurotechnological methods. This study opens doors for future translational research to explore the broader application of computer vision and deep learning techniques to derive kinematic signatures of movement disorders across species and experimental conditions, promising more precise and personalised assessments that can significantly improve therapeutic strategies and patient outcomes.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Quantitative assessment of head movement dynamics in dystonia using visual perceptive deep learning: a multi-centre retrospective longitudinal cohort study

Peach,

Friedrich,

Fronemann

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recent advancements have introduced mobile-centric models for nystagmus tracking, such as ConVNG ( 9 ) and EyePhone ( 20 ), which are designed for smartphones. Our model, however, is not merely a standalone tool for nystagmus detection but rather a foundational model making significant contributions to this field.…”

Section: Discussionmentioning

confidence: 99%

“…However, we have reached the conclusion that any model that seeks to address these problems must be lightweight and patient-centric design to facilitate efficient diagnostic workflow. Recent studies have utilized webcams ( 8 ) and smartphones ( 9 ) as well as video-oculographic devices ( 10 ) for detection of nystagmus. Thus, we have developed a new technology capable of continuously tracking eye movements by recognizing regions of the ocular.…”

Section: Introductionmentioning

confidence: 99%

Feasibility of video-based real-time nystagmus tracking: a lightweight deep learning model approach using ocular object segmentation

Cho,

Park,

et al. 2024

Front. Neurol.

View full text Add to dashboard Cite

BackgroundEye movement tests remain significantly underutilized in emergency departments and primary healthcare units, despite their superior diagnostic sensitivity compared to neuroimaging modalities for the differential diagnosis of acute vertigo. This underutilization may be attributed to a potential lack of awareness regarding these tests and the absence of appropriate tools for detecting nystagmus. This study aimed to develop a nystagmus measurement algorithm using a lightweight deep-learning model that recognizes the ocular regions.MethodThe deep learning model was used to segment the eye regions, detect blinking, and determine the pupil center. The model was trained using images extracted from video clips of a clinical battery of eye movement tests and synthesized images reproducing real eye movement scenarios using virtual reality. Each eye image was annotated with segmentation masks of the sclera, iris, and pupil, with gaze vectors of the pupil center for eye tracking. We conducted a comprehensive evaluation of model performance and its execution speeds in comparison to various alternative models using metrics that are suitable for the tasks.ResultsThe mean Intersection over Union values of the segmentation model ranged from 0.90 to 0.97 for different classes (sclera, iris, and pupil) across types of images (synthetic vs. real-world images). Additionally, the mean absolute error for eye tracking was 0.595 for real-world data and the F1 score for blink detection was ≥ 0.95, which indicates our model is performing at a very high level of accuracy. Execution speed was also the most rapid for ocular object segmentation under the same hardware condition as compared to alternative models. The prediction for horizontal and vertical nystagmus in real eye movement video revealed high accuracy with a strong correlation between the observed and predicted values (r = 0.9949 for horizontal and r = 0.9950 for vertical; both p < 0.05).ConclusionThe potential of our model, which can automatically segment ocular regions and track nystagmus in real time from eye movement videos, holds significant promise for emergency settings or remote intervention within the field of neurotology.

show abstract

“…Non-infrared oculography systems may use visible light reflections from the cornea or pupil to monitor eye movements. 23 It is particularly useful for studying eye movements in well-lit environments and can provide high-resolution tracking of gaze points. 24 VOG is widely used in both research and clinical settings for diagnosing and monitoring eye conditions like strabismus and nystagmus.…”

Section: Methods Of Recording Eye Movementsmentioning

confidence: 99%

Detecting Abnormal Eye Movements in Patients with Neurodegenerative Diseases – Current Insights

Sekar,

Panouillères,

Kaski

2024

View full text Add to dashboard Cite

This review delineates the ocular motor disturbances across a spectrum of neurodegenerative disorders, including Alzheimer's Disease (AD) and related disorders (ADRD), Parkinson's Disease (PD), atypical parkinsonism, and others, leveraging advancements in eye-tracking technology for enhanced diagnostic precision. We delve into the different classes of eye movements, their clinical assessment, and specific abnormalities manifesting in these diseases, highlighting the nuanced differences and shared patterns. For instance, AD and ADRD are characterized by increased saccadic latencies and instability in fixation, while PD features saccadic hypometria and mild smooth pursuit impairments. Atypical parkinsonism, notably Progressive Supranuclear Palsy (PSP) and Corticobasal Syndrome (CBS), presents with distinct ocular motor signatures such as vertical supranuclear gaze palsy and saccadic apraxia, respectively. Our review underscores the diagnostic value of eye movement analysis in differentiating between these disorders and also posits the existence of underlying common pathological mechanisms. We discuss how eye movements have potential as biomarkers for neurodegenerative diseases but also some of the existing limitations.

show abstract

Smartphone video nystagmography using convolutional neural networks: ConVNG

Cited by 12 publications

References 78 publications

Quantitative assessment of head movement dynamics in dystonia using visual perceptive deep learning: a multi-centre retrospective longitudinal cohort study

Quantitative assessment of head movement dynamics in dystonia using visual perceptive deep learning: a multi-centre retrospective longitudinal cohort study

Feasibility of video-based real-time nystagmus tracking: a lightweight deep learning model approach using ocular object segmentation

Detecting Abnormal Eye Movements in Patients with Neurodegenerative Diseases – Current Insights

Contact Info

Product

Resources

About