Fusion of video and inertial sensing data via dynamic optimization of a biomechanical model

Pearl, Owen Douglas; Shin, Soyong; Godura, Ashwin; Bergbreiter, Sarah; Halilaj, Eni

doi:10.1016/j.jbiomech.2023.111617

Cited by 8 publications

(5 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Alternatively, the kinematic error can be minimized by compensating sensor noise and bias (e.g., sensor drift or magnetic disturbances) which affects methods using IMU-sensors for motion analysis ( Cockcroft et al, 2014 ; Allen et al, 2017 ; Laidig et al, 2017 ). Some methods investigate if a multimodal motion analysis approach (combination of IMU-sensors with either RGB-video- or depth-camera) enhances tracking performance thus minimizing the kinematic error ( Atrsaei et al, 2016 ; Halilaj et al, 2021 ; Pearl et al, 2023 ). Other methods explicitly aim to generate simulation results without undesired residuals ( Vaughan et al, 1982 ; Koopman et al, 1995 ; Kuo, 1998 ; Cahouët et al, 2002 ; Mazzà and Cappozzo, 2004 ; Riemer et al, 2008 ; Remy and Thelen, 2009 ; Riemer and Hsiao-Wecksler, 2009 ; Jackson et al, 2015 ; Samaan et al, 2016 ; Faber et al, 2018 ; Muller et al, 2018 ; Noamani et al, 2018 ; Fritz et al, 2019 ; Pallarès-López et al, 2019 ; Sturdy et al, 2022 ; Werling et al, 2023 ).…”

Section: Resultsmentioning

confidence: 99%

“…They hypothesise that through this approach, individual weaknesses (marker occlusion and sensor drift) of the two measurement and sensor technologies compensate each other. Analogous, Pearl et al (2023) fused camera and IMU data to track human gait. But instead of using a Kalman Filter, the authors used dynamic optimization to analyze experimentally measured motion.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Bridging the sim2real gap. Investigating deviations between experimental motion measurements and musculoskeletal simulation results—a systematic review

Wechsler,

Wolf,

Shanbhag

et al. 2024

Front. Bioeng. Biotechnol.

View full text Add to dashboard Cite

Musculoskeletal simulations can be used to estimate biomechanical variables like muscle forces and joint torques from non-invasive experimental data using inverse and forward methods. Inverse kinematics followed by inverse dynamics (ID) uses body motion and external force measurements to compute joint movements and the corresponding joint loads, respectively. ID leads to residual forces and torques (residuals) that are not physically realistic, because of measurement noise and modeling assumptions. Forward dynamic simulations (FD) are found by tracking experimental data. They do not generate residuals but will move away from experimental data to achieve this. Therefore, there is a gap between reality (the experimental measurements) and simulations in both approaches, the sim2real gap. To answer (patho-) physiological research questions, simulation results have to be accurate and reliable; the sim2real gap needs to be handled. Therefore, we reviewed methods to handle the sim2real gap in such musculoskeletal simulations. The review identifies, classifies and analyses existing methods that bridge the sim2real gap, including their strengths and limitations. Using a systematic approach, we conducted an electronic search in the databases Scopus, PubMed and Web of Science. We selected and included 85 relevant papers that were sorted into eight different solution clusters based on three aspects: how the sim2real gap is handled, the mathematical method used, and the parameters/variables of the simulations which were adjusted. Each cluster has a distinctive way of handling the sim2real gap with accompanying strengths and limitations. Ultimately, the method choice largely depends on various factors: available model, input parameters/variables, investigated movement and of course the underlying research aim. Researchers should be aware that the sim2real gap remains for both ID and FD approaches. However, we conclude that multimodal approaches tracking kinematic and dynamic measurements may be one possible solution to handle the sim2real gap as methods tracking multimodal measurements (some combination of sensor position/orientation or EMG measurements), consistently lead to better tracking performances. Initial analyses show that motion analysis performance can be enhanced by using multimodal measurements as different sensor technologies can compensate each other’s weaknesses.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Bridging the sim2real gap. Investigating deviations between experimental motion measurements and musculoskeletal simulation results—a systematic review

Wechsler,

Wolf,

Shanbhag

et al. 2024

Front. Bioeng. Biotechnol.

View full text Add to dashboard Cite

show abstract

“…If the errors in the experimental measurement data are too large to obtain accurate results, one possible solution could be to develop strategies for enhancing the accuracy of the input data. Previous research has shown that fusing IMU data with either RGB- [44,45] or depth-camera [46] data can enhance motion tracking performance in comparison to single modality approaches. So, using multimodal motion data may enhance the method's robustness by mitigating measurement errors inherent to one sensor type (IMU drift) through complementary data acquisition from another sensor type (e.g.…”

Section: Discussionmentioning

confidence: 99%

IMU-based joint axis identification method for arbitrary joints in OpenSim - a simulation study

Wechsler,

Shanbhag,

Wartzack

et al. 2024

Preprint

View full text Add to dashboard Cite

In musculoskeletal simulation, individualized joint axes enhance the accuracy and reliability of kinematic and kinetic simulation results. We investigated the correctness and performance of an analytical method for identifying the instantaneous axis of rotation between two bodies based on motion data in OpenSim. The instantaneous center of rotation is the point at which two bodies have the same velocity. The relative linear and angular velocity between the two bodies, as well as their relative position to each another, are required as inputs to calculate it. Using the instantaneous center of rotation, fixed or moving joint centers of rotation can be identified. To prove the general applicability of the method, the instantaneous centers of rotation of a revolute joint of a simple double pendulum model and the hip and knee joint of a more complex musculoskeletal model were investigated. The hip joint is defined as a ball joint. The knee joint is defined as an OpenSim custom joint which describes the motion of the child segment in relation to the parent segment as a function of generalized coordinates. To verify the correctness of the approach in OpenSim, the moving centers of rotation were calculated using synthetic noisefree data. The results were compared to the implementation of the respective joints in the model which act as the ground truth. White Gaussian noise was added to the synthetic data to analyze its effect on the quality of the calculated centers of rotation. We were able to correctly identify the center of rotation of each joint using noisefree data. In the case of noisy data, joint centers of rotation can be determined by applying additional filtering or optimization methods to the calculated instantaneous centers of rotation. Consequently, we are able to determine the center of rotation for arbitrary joints based on noisy synthetic data. This approach is applicable for both fixed and moving centers of rotation which distinguishes it from commonly used methods in the field of biomechanical simulation.

show abstract

“…Intelligent fusion of data from multimodal wearables has remained elusive, despite its promise to revolutionize human motion tracking in the wild. Sensor-fusion algorithms applied to data from IMUs are unable to reach high accuracy in human applications, despite their success in robotics [39] , due to the unique challenges posed by sensor-to-body calibration and soft-tissuemotion uncertainties. Although the formfactor of wearables is improving rapidly, with band-aid-like inertial sensors on the market, clinical translation remains sparse because accurate estimation of human kinematics and kinetics in natural environments is still out of reach.…”

Section: Capacitive Sensing As An Integral Component Of Multimodal We...mentioning

confidence: 99%

Capacitive Sensing for Natural Environment Biomechanics Monitoring

Pearl,

Rokhmanova,

Dankovich

et al. 2024

Preprint

View full text Add to dashboard Cite

Inexpensive wearable sensors could transform how we monitor patient movement outside of the laboratory and help personalize the treatment of mobility impairments[1]. To meet these expectations, wearable sensors must be benchmarked against clinical standards, provide reliable data over extended periods of time, be robust to placement errors by non-experts, and offer tangible translational potential. Inertial sensing is one of the only wearable technologies that have been comprehensively characterized and benchmarked against gold-standard biomechanical measurements, but it remains sensitive to both drift and placement error[2–4] and does not capture muscle activity, which is relevant to both mobility impairments and the development of assistive wearables. Here we investigate capacitive sensing[5] as a muscle-activity monitoring technology, finding for the first time that it can accurately track changes in muscle bulging and fiber length with the fidelity of laboratory tools, in natural environments, over long durations, on multiple body parts, and across people of varying compositions. We also demonstrate translational potential in two clinical applications and show that capacitive sensing complements inertial sensing better than electromyography in multimodal wearable-sensing systems, achieving state-of-the-art accuracy for wearable motion tracking. Our results indicate that capacitive sensing wearables could be used synergistically with other emerging wearable technologies to provide real-time feedback and make rehabilitation monitoring outside of laboratory environments more feasible[6]. We expect this foundational study of capacitive sensing for natural environment biomechanics monitoring—combining wearable sensor design, human experiments, biomechanical modeling, multimodal data fusion, magnetic resonance imaging, and clinical research—to be applicable to a wide range of mobility-related pathologies and emerging human-in-the-loop assistive devices[7–9].

show abstract

Fusion of video and inertial sensing data via dynamic optimization of a biomechanical model

Cited by 8 publications

References 40 publications

Bridging the sim2real gap. Investigating deviations between experimental motion measurements and musculoskeletal simulation results—a systematic review

Bridging the sim2real gap. Investigating deviations between experimental motion measurements and musculoskeletal simulation results—a systematic review

IMU-based joint axis identification method for arbitrary joints in OpenSim - a simulation study

Capacitive Sensing for Natural Environment Biomechanics Monitoring

Contact Info

Product

Resources

About