Review: Are we stumbling in our quest to find the best predictor? Over‐optimism in sensor‐based models for predicting falls in older adults

Abstract

The field of fall risk testing using wearable sensors is bustling with activity. In this Letter, the authors review publications which incorporated features extracted from sensor signals into statistical models intended to estimate fall risk or predict falls in older people. A review of these studies raises concerns that this body of literature is presenting over-optimistic results in light of small sample sizes, questionable modelling decisions and problematic validation methodologies (e.g. inherent problems with the overly-popular cross-validation technique, lack of external validation). There seem to be substantial issues in the feature selection process, whereby researchers select features before modelling begins based on their relation to the target, and either perform no validation or test the models on the same data used for their training. This, together with potential issues related to the large number of features and their correlations, inevitably leads to models with inflated accuracy that are unlikely to maintain their reported performance during everyday use in relevant populations. Indeed, the availability of rich sensor data and many analytical options provides intellectual and creative freedom for researchers, but should be treated with caution, and such pitfalls must be avoided if we desire to create generalisable prognostic tools of any clinical value.

Wearable Inertial Sensors for Fall Risk Assessment and Prediction in Older Adults: A Systematic Review and Meta-Analysis

“…accelerometers and gyroscopes) attached to the subject's body during specific assessment tasks (e.g. walking, quiet standing, sit-to-stand transitions) [9]- [11]. In those studies, machine learning methods were used to automatically identify fallers (F) and non-fallers (NF).…”

“…Howcroft et al [9] and Shany et al [11] have presented insightful accounts of features, classification models and validation strategies related to sensor-based fall-risk testing (SFRT). In their investigations, these authors found large heterogeneity in terms of sensor placement, tasks assessed, and sensor-based features.…”

“…The latter is particularly relevant to overcome the limitations imposed by the curse of dimensionality (i.e. the difficulty and risk of training learning algorithms to discriminate between non-fallers and fallers in a high-dimensional feature space on the basis of a small pool of available data) [11]. Increasing the sample size would seem to be the logical solution.…”

Unintended Consequences of Wearable Sensor Use in Healthcare

“…For example, in geriatric fall prediction [18], pooling activity and falls data collected by wearable sensors across multiple studies, and making them open access, may enable the development of better prediction models. In fact, 'intellectual crowdsourcing' is the basis of Kaggle [19], an online platform where researchers and companies, both within and outside of healthcare, post datasets to crowdsource statistical and data mining labor via competitions to find the best predictive models.…”

Health-Enabling and Ambient Assistive Technologies: Past, Present, Future

“…In the analysis of sensor data for ambulation and falls we learnt that many research groups worldwide fell into analysis and methodological traps that limit the generalizability and usefulness of some of the analyses and models reported in the literature. Some of the associated issues and lessons learnt are summarized in [119] and include overly-optimistic results in light of small sample sizes, questionable modelling decisions, and problematic validation methodologies. With the enormous increase in availability of rich sensor data and many possible analytical approaches, researchers are provided with the intellectual and creative freedom to explore datasets without constraints.…”