The availability of smartphone and wearable sensor technology is leading to a rapid accumulation of human subject data, and machine learning is emerging as a technique to map those data into clinical predictions. As machine learning algorithms are increasingly used to support clinical decision making, it is vital to reliably quantify their prediction accuracy. Cross-validation (CV) is the standard approach where the accuracy of such algorithms is evaluated on part of the data the algorithm has not seen during training. However, for this procedure to be meaningful, the relationship between the training and the validation set should mimic the relationship between the training set and the dataset expected for the clinical use. Here we compared two popular CV methods: record-wise and subject-wise. While the subject-wise method mirrors the clinically relevant use-case scenario of diagnosis in newly recruited subjects, the record-wise strategy has no such interpretation. Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms. We also conducted a systematic review of the relevant literature, and found that this overly optimistic method was used by almost half of the retrieved studies that used accelerometers, wearable sensors, or smartphones to predict clinical outcomes. As we move towards an era of machine learning-based diagnosis and treatment, using proper methods to evaluate their accuracy is crucial, as inaccurate results can mislead both clinicians and data scientists.

Machine learning algorithms that use data streams captured from soft wearable sensors have the potential to automatically detect PD symptoms and inform clinicians about the progression of disease. However, these algorithms must be trained with annotated data from clinical experts who can recognize symptoms, and collecting such data are costly. Understanding how many sensors and how much labeled data are required is key to successfully deploying these models outside of the clinic. Here we recorded movement data using 6 flexible wearable sensors in 20 individuals with PD over the course of multiple clinical assessments conducted on 1 day and repeated 2 weeks later. Participants performed 13 common tasks, such as walking or typing, and a clinician rated the severity of symptoms (bradykinesia and tremor). We then trained convolutional neural networks and statistical ensembles to detect whether a segment of movement showed signs of bradykinesia or tremor based on data from tasks performed by other individuals. Our results show that a single wearable sensor on the back of the hand is sufficient for detecting bradykinesia and tremor in the upper extremities, whereas using sensors on both sides does not improve performance. Increasing the amount of training data by adding other individuals can lead to improved performance, but repeating assessments with the same individuals—even at different medication states—does not substantially improve detection across days. Our results suggest that PD symptoms can be detected during a variety of activities and are best modeled by a dataset incorporating many individuals.

Objective: Controlling the spread of the COVID-19 pandemic largely depends on scaling up the testing infrastructure for identifying infected individuals. Consumer-grade wearables may present a solution to detect the presence of infections in the population, but the current paradigm requires collecting physiological data continuously and for long periods of time on each individual, which poses limitations in the context of rapid screening. Technology: Here, we propose a novel paradigm based on recording the physiological responses elicited by a short (~2 minutes) sequence of activities (i.e. “snapshot”), to detect symptoms associated with COVID-19. We employed a novel body-conforming soft wearable sensor placed on the suprasternal notch to capture data on physical activity, cardio-respiratory function, and cough sounds. Results: We performed a pilot study in a cohort of individuals (n=14) who tested positive for COVID-19 and detected altered heart rate, respiration rate and heart rate variability, relative to a group of healthy individuals (n=14) with no known exposure. Logistic regression classifiers were trained on individual and combined sets of physiological features (heartbeat and respiration dynamics, walking cadence, and cough frequency spectrum) at discriminating COVID-positive participants from the healthy group. Combining features yielded an AUC of 0.94 (95% CI=[0.92, 0.96]) using a leave-one-subject-out cross validation scheme. Conclusions and Clinical Impact: These results, although preliminary, suggest that a sensor-based snapshot paradigm may be a promising approach for non-invasive and repeatable testing to alert individuals that need further screening.

This three-part review takes a detailed look at the complexities of cross-validation, fostered by the peer review of Saeb et al.’s paper entitled “The need to approximate the use-case in clinical machine learning.” It contains perspectives by reviewers and by the original authors that touch upon cross-validation: the suitability of different strategies and their interpretation.

Background: Parkinson's disease (PD) is a progressive neurological disease, with characteristic motor symptoms such as tremor and bradykinesia. There is a growing interest to continuously monitor these and other symptoms through body-worn sensor technology. However, limited battery life and memory capacity hinder the potential for continuous, long-term monitoring with these devices. There is little information available on the relative value of adding sensors, increasing sampling rate, or computing complex signal features, all of which may improve accuracy of symptom detection at the expense of computational resources. Here we build on a previous study to investigate the relationship between data measurement characteristics and accuracy when using wearable sensor data to classify tremor and bradykinesia in patients with PD. Methods: Thirteen individuals with PD wore a flexible, skin-mounted sensor (collecting tri-axial accelerometer and gyroscope data) and a commercial smart watch (collecting tri-axial accelerometer data) on their predominantly affected hand. The participants performed a series of standardized motor tasks, during which a clinician scored the severity of tremor and bradykinesia in that limb. Machine learning models were trained on scored data to classify tremor and bradykinesia. Model performance was compared when using different types of sensors (accelerometer and/or gyroscope), different data sampling rates (up to 62.5 Hz), and different categories of pre-engineered features (up to 148 features). Performance was also compared between the flexible sensor and smart watch for each analysis. Results: First, there was no effect of device type for classifying tremor symptoms (p > 0.34), but bradykinesia models incorporating gyroscope data performed slightly better (up to 0.05 AUROC) than other models (p = 0.01). Second, model performance decreased with sampling frequency (p < 0.001) for tremor, but not bradykinesia (p > 0.47). Finally, model performance for both symptoms was maintained after substantially reducing the feature set.

BackgroundAutomatically detecting falls with mobile phones provides an opportunity for rapid response to injuries and better knowledge of what precipitated the fall and its consequences. This is beneficial for populations that are prone to falling, such as people with lower limb amputations. Prior studies have focused on fall detection in able-bodied individuals using data from a laboratory setting. Such approaches may provide a limited ability to detect falls in amputees and in real-world scenarios.ObjectiveThe aim was to develop a classifier that uses data from able-bodied individuals to detect falls in individuals with a lower limb amputation, while they freely carry the mobile phone in different locations and during free-living.MethodsWe obtained 861 simulated indoor and outdoor falls from 10 young control (non-amputee) individuals and 6 individuals with a lower limb amputation. In addition, we recorded a broad database of activities of daily living, including data from three participants’ free-living routines. Sensor readings (accelerometer and gyroscope) from a mobile phone were recorded as participants freely carried it in three common locations—on the waist, in a pocket, and in the hand. A set of 40 features were computed from the sensors data and four classifiers were trained and combined through stacking to detect falls. We compared the performance of two population-specific models, trained and tested on either able-bodied or amputee participants, with that of a model trained on able-bodied participants and tested on amputees. A simple threshold-based classifier was used to benchmark our machine-learning classifier.ResultsThe accuracy of fall detection in amputees for a model trained on control individuals (sensitivity: mean 0.989, 1.96*standard error of the mean [SEM] 0.017; specificity: mean 0.968, SEM 0.025) was not statistically different (P=.69) from that of a model trained on the amputee population (sensitivity: mean 0.984, SEM 0.016; specificity: mean 0.965, SEM 0.022). Detection of falls in control individuals yielded similar results (sensitivity: mean 0.979, SEM 0.022; specificity: mean 0.991, SEM 0.012). A mean 2.2 (SD 1.7) false alarms per day were obtained when evaluating the model (vs mean 122.1, SD 166.1 based on thresholds) on data recorded as participants carried the phone during their daily routine for two or more days. Machine-learning classifiers outperformed the threshold-based one (P<.001).ConclusionsA mobile phone-based fall detection model can use data from non-amputee individuals to detect falls in individuals walking with a prosthesis. We successfully detected falls when the mobile phone was carried across multiple locations and without a predetermined orientation. Furthermore, the number of false alarms yielded by the model over a longer period of time was reasonably low. This moves the application of mobile phone-based fall detection systems closer to a real-world use case scenario.

Recent advancements in deep learning have produced significant progress in markerless human pose estimation, making it possible to estimate human kinematics from single camera videos without the need for reflective markers and specialized labs equipped with motion capture systems. Such algorithms have the potential to enable the quantification of clinical metrics from videos recorded with a handheld camera. Here we used DeepLabCut, an open-source framework for markerless pose estimation, to fine-tune a deep network to track 5 body keypoints (hip, knee, ankle, heel, and toe) in 82 below-waist videos of 8 patients with stroke performing overground walking during clinical assessments. We trained the pose estimation model by labeling the keypoints in 2 frames per video and then trained a convolutional neural network to estimate 5 clinically relevant gait parameters (cadence, double support time, swing time, stance time, and walking speed) from the trajectory of these keypoints. These results were then compared to those obtained from a clinical system for gait analysis (GAITRite®, CIR Systems). Absolute accuracy (mean error) and precision (standard deviation of error) for swing, stance, and double support time were within 0.04 ± 0.11 s; Pearson’s correlation with the reference system was moderate for swing times (<i>r</i> = 0.4–0.66), but stronger for stance and double support time (<i>r</i> = 0.93–0.95). Cadence mean error was −0.25 steps/min ± 3.9 steps/min (<i>r</i> = 0.97), while walking speed mean error was −0.02 ± 0.11 m/s (<i>r</i> = 0.92). These preliminary results suggest that single camera videos and pose estimation models based on deep networks could be used to quantify clinically relevant gait metrics in individuals poststroke, even while using assistive devices in uncontrolled environments. Such development opens the door to applications for gait analysis both inside and outside of clinical settings, without the need of sophisticated equipment.

Abstract-Tactile feedback is pivotal for grasping and manipulation in humans. Providing functionally effective sensory feedback to prostheses users is an open challenge. Past paradigms were mostly based on vibroor electrotactile stimulations. However, the tactile sensitivity on the targeted body parts (usually the forearm) is greatly less than that of the hand/fingertips, restricting the amount of information that can be provided through this channel. Visual feedback is the most investigated technique in motor learning studies, where it showed positive effects in learning both simple and complex tasks; however, it was not exploited in prosthetics due to technological limitations. Here, we investigated if visual information provided in the form of augmented reality (AR) feedback can be integrated by able-bodied participants in their sensorimotor control of a pick-and-lift task while controlling a robotic hand. For this purpose, we provided visual continuous feedback related to grip force and hand closure to the participants. Each variable was mapped to the length of one of the two ellipse axes visualized on the screen of wearable single-eye display AR glasses. We observed changes in behavior when subtle (i.e., not announced to the participants) manipulation of the AR feedback was introduced, which indicated that the participants integrated the artificial feedback within the sensorimotor control of the task. These results demonstrate that it is possible to deliver effective information through AR feedback in a compact and wearable fashion. This feedback modality may be exploited for delivering sensory feedback to amputees in a clinical scenario.