Sequential Decision Fusion for Environmental Classification in Assistive Walking

Abstract

Powered prostheses are effective for helping amputees walk on level ground, but these devices are inconvenient to use in complex environments. Prostheses need to understand the motion intent of amputees to help them walk in complex environments. Recently, researchers have found that they can use vision sensors to classify environments and predict the motion intent of amputees. Previous researchers can classify environments accurately in the offline analysis, but they neglect to decrease the corresponding time delay. To increase the accuracy and decrease the time delay of environmental classification, we propose a new decision fusion method in this paper. We fuse sequential decisions of environmental classification by constructing a hidden Markov model and designing a transition probability matrix. We evaluate our method by inviting ablebodied subjects and amputees to implement indoor and outdoor experiments. Experimental results indicate that our method can classify environments more accurately and with less time delay than previous methods. Besides classifying environments, the proposed decision fusion method may also optimize sequential predictions of the human motion intent in the future.

ExoNet Database: Wearable Camera Images of Human Locomotion Environments

“…In comparison, the previous largest dataset contained approximately 402,000 images (Massalin et al, 2018). While most environment recognition systems have included fewer than 6 classes (Khademi and Simon, 2019; Krausz and Hargrove, 2015; Krausz et al, 2015; 2019; Laschowski et al, 2019b; Massalin et al, 2018; Novo-Torres et al, 2019; Varol and Massalin, 2016; Zhang et al, 2019b; 2019c; 2019d; 2020), the ExoNet database features a 12-class hierarchical labelling architecture. These differences have practical implications given that learning-based algorithms like deep convolutional neural networks require significant and diverse training images (LeCun et al, 2015).…”

“…One subject was instrumented with a lightweight wearable smartphone camera system (iPhone XS Max); photograph shown in Figure 1A. Unlike limb-mounted systems (Da Silva et al, 2020; Diaz et al, 2018; Hu et al, 2018; Kleiner et al, 2018; Massalin et al, 2018; Rai and Rombokas, 2018; Varol and Massalin, 2016; Zhang et al, 2011; 2019b; 2019c), chest-mounting can provide more stable video recording and allow users to wear pants and long dresses without obstructing the sampled field-of-view. The chest-mount height was approximately 1.3 m from the ground when the participant stood upright.…”

“…While most environment recognition systems have been limited to controlled indoor environments and/or prearranged walking circuits (Du et al, 2012; Hu et al, 2018; Khademi and Simon, 2019; Kleiner et al, 2018; Krausz et al, 2015; 2019; Liu et al, 2016; Wang et al, 2013; Zhang et al, 2011; 2019b; 2019c; 2019d), our subject walked around unknown outdoor and indoor real-world environments while collecting images with occlusions, signal noise, and intraclass variations. Data were collected at various times throughout the day to incorporate different lighting conditions.…”

“…However, wearable vision-based systems can provide more detailed information about the field-of-view and detect physical obstacles in peripheral locations. Most environment recognition systems have included either RGB cameras (Da Silva et al, 2020; Diaz et al, 2018; Khademi and Simon, 2019; Krausz and Hargrove, 2015; Laschowski et al, 2019b; Novo-Torres et al, 2019; Zhong et al, 2020) or 3D depth cameras (Hu et al, 2018; Krausz et al, 2015; 2019; Massalin et al, 2018; Varol and Massalin, 2016; Zhang et al, 2019b; 2019c; 2019d).…”

“…For image classification, researchers have used learning-based algorithms like support vector machines (Massalin et al, 2018; Varol and Massalin, 2016) and deep convolutional neural networks (Khademi and Simon, 2019; Laschowski et al, 2019b; Novo-Torres et al, 2019; Rai and Rombokas, 2018; Zhang et al, 2019b; 2019c; 2019d; Zhong et al, 2020). Although convolutional neural networks typically outperform support vector machines for image classification (LeCun et al, 2015), deep learning requires significant and diverse training images to prevent overfitting and promote generalization.…”