A Non-Laboratory Gait Dataset of Full Body Kinematics and Egocentric Vision

Sharma, Abhishek; Rai, Vijeth; Calvert, Melissa; Dai, Zhongyi; Guo, Zhou; Boe, David

doi:10.1038/s41597-023-01932-7

Cited by 7 publications

(7 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used the multimodal dataset by [6] that includes fullbody kinematics from inertial sensors and RGB images with gaze from smart glasses. A detailed description is provided in [7]. Data was collected with 23 healthy young adults walking in 4 different annotated environments, including transitions, for a total of ∼12 hours.…”

Section: Methodsmentioning

confidence: 99%

“…Computer vision systems like smart glasses can inform about the walking environment and sense obstacles before they are encountered, which other technologies like inertial measurement units (IMUs) are not fully capable of. Recent research [6]- [7] has shown that adding computer vision data can result in 7.9% and 7.0% improvement in root mean squared error (RMSE) of knee and ankle joint angle predictions, respectively, compared to using only inertial sensors; this was possible by pairing the inertial data with optical flow [8] mappings from the smart glasses. However, due to the high computational requirements of the proposed long short-term memory (LSTM)-based networks, their model was not capable of running real-time inference on an embedded system.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Continuous Prediction of Leg Kinematics during Walking using Inertial Sensors, Smart Glasses, and Embedded Computing

Tsepa

Burakov

Laschowski

et al. 2023

Preprint

View full text Add to dashboard Cite

Unlike traditional hierarchical controllers for robotic leg prostheses and exoskeletons, continuous systems could allow persons with mobility impairments to walk more naturally in real-world environments without requiring high-level switching between locomotion modes. To support these next-generation controllers, we developed a new system called KIFNet (Kinematics and Image Fusing Network) that uses lightweight and efficient deep learning models to continuously predict the leg kinematics during walking. We tested different sensor fusion methods to combine kinematics data from inertial sensors and computer vision data from smart glasses and found that adaptive instance normalization achieved the lowest RMSE predictions for knee and ankle joint kinematics. We also deployed our model on an embedded device. Without inference optimization, our model was 20 times faster than the previous state-of-the-art and achieved 20% higher prediction accuracies, and during some locomotor activities like stair descent, decreased RMSE up to 300%. With inference optimization, our best model achieved 125 FPS on an NVIDIA Jetson Nano. These results demonstrate the potential to build fast and accurate deep learning models for continuous prediction of leg kinematics during walking based on sensor fusion and embedded computing, therein providing a foundation for real-time continuous controllers for robotic leg prostheses and exoskeletons.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Continuous Prediction of Leg Kinematics during Walking using Inertial Sensors, Smart Glasses, and Embedded Computing

Tsepa

Burakov

Laschowski

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Subjects provided verbal and written informed consent approved by the University of Nebraska Medical Center's Institutional Review Board (# 0762-21-EP). This study does not contain sensitive data (i.e., from children) and all participants were between 19 to 31 years old [72][73][74][75][76][77][78][79][80][81][82][83][84][85] .…”

Section: Methodsmentioning

confidence: 99%

NONAN GaitPrint: An IMU gait database of healthy young adults

Wiles,

Mangalam,

Sommerfeld

et al. 2023

Sci Data

View full text Add to dashboard Cite

An ongoing thrust of research focused on human gait pertains to identifying individuals based on gait patterns. However, no existing gait database supports modeling efforts to assess gait patterns unique to individuals. Hence, we introduce the Nonlinear Analysis Core (NONAN) GaitPrint database containing whole body kinematics and foot placement during self-paced overground walking on a 200-meter looping indoor track. Noraxon Ultium MotionTM inertial measurement unit (IMU) sensors sampled the motion of 35 healthy young adults (19–35 years old; 18 men and 17 women; mean ± 1 s.d. age: 24.6 ± 2.7 years; height: 1.73 ± 0.78 m; body mass: 72.44 ± 15.04 kg) over 18 4-min trials across two days. Continuous variables include acceleration, velocity, position, and the acceleration, velocity, position, orientation, and rotational velocity of each corresponding body segment, and the angle of each respective joint. The discrete variables include an exhaustive set of gait parameters derived from the spatiotemporal dynamics of foot placement. We technically validate our data using continuous relative phase, Lyapunov exponent, and Hurst exponent—nonlinear metrics quantifying different aspects of healthy human gait.

show abstract

“…However, the methods are application agnostic. With the recent rise in multimodal gait datasets [8], [14]- [19], we can also learn representations of hard to define factors in a similar way the representations of style were learned in this manuscript. Similarly, we could encode terrain in an appropriate latent space and then compose different representations using decoder.…”

Section: Beyond Personalizationmentioning

confidence: 97%

Optimizing representations of multiple simultaneous attributes for gait generation using deep learning

Sharma¹

2023

Preprint

View full text Add to dashboard Cite

<p>Rich variations in gait are generated according to several attributes of the individual and environment, such as age, athleticism, terrain, speed, personal ”style”, mood, etc. The effects of these attributes can be hard to quantify explicitly, but relatively straightforward to sample. We seek to generate gait that expresses these attributes, creating synthetic gait samples that exemplify a custom mix of attributes. This is difficult to perform manually, and generally restricted to simple, human-interpretable and handcrafted rules. In this manuscript, we present neural network architectures to learn representations of hard to quantify attributes from data, and generate gait trajectories by composing multiple desirable attributes. We demonstrate this method for the two most commonly desired attribute classes: individual style and walking speed. We show that two methods, cost function design and latent space regularization, can be used individually or combined. We also show two uses of machine learning classifiers that recognize individuals and speeds. Firstly, they can be used as quantitative measures of success; if a synthetic gait fools a classifier, then it is considered to be a good example of that class. Secondly, we show that classifiers can be used in the latent space regularizations and cost functions to improve training beyond a typical squared-error cost. </p>

show abstract

A Non-Laboratory Gait Dataset of Full Body Kinematics and Egocentric Vision

Cited by 7 publications

References 35 publications

Continuous Prediction of Leg Kinematics during Walking using Inertial Sensors, Smart Glasses, and Embedded Computing

Continuous Prediction of Leg Kinematics during Walking using Inertial Sensors, Smart Glasses, and Embedded Computing

NONAN GaitPrint: An IMU gait database of healthy young adults

Optimizing representations of multiple simultaneous attributes for gait generation using deep learning

Contact Info

Product

Resources

About