ExoNet Database: Wearable Camera Images of Human Locomotion Environments

Laschowski, Brock; McNally, William; Wong, Alexander; McPhee, John

doi:10.3389/frobt.2020.562061

Cited by 30 publications

(27 citation statements)

References 34 publications

(92 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, smallscale and private training datasets have hindered the development of image classification algorithms for environment recognition [20]. To address these limitations, we developed ExoNet -the largest and most diverse open-source dataset of wearable camera images of walking environments [11]. Unparalleled in both scale and diversity, ExoNet contains over 5.6 million images of indoor and outdoor real-world environments, of which ~923,000 images were annotated using a 12class hierarchical labelling architecture; these design features are important since deep learning requires significant and diverse training data.…”

Section: Discussionmentioning

confidence: 99%

“…Data were collected throughout the summer, fall, and winter seasons to incorporate different weathered surfaces like snow, grass, and multicolored leaves. The image database, which we named ExoNet, was deposited in the IEEE DataPort repository and is now publicly available for download [11]. The file size of the uncompressed videos is ~140 GB.…”

Section: A Experimental Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Computer Vision and Deep Learning for Environment-Adaptive Control of Robotic Lower-Limb Exoskeletons

Laschowski

McNally

Wong

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Robotic exoskeletons require human control and decision making to switch between different locomotion modes, which can be inconvenient and cognitively demanding. To support the development of automated locomotion mode recognition systems (i.e., high-level controllers), we designed an environment recognition system using computer vision and deep learning. We collected over 5.6 million images of indoor and outdoor real-world walking environments using a wearable camera system, of which ~923,000 images were annotated using a 12-class hierarchical labelling architecture (called the ExoNet database). We then trained and tested the EfficientNetB0 convolutional neural network, designed for efficiency using neural architecture search, to predict the different walking environments. Our environment recognition system achieved ~73% image classification accuracy. While these preliminary results benchmark EfficientNetB0 on the ExoNet database, further research is needed to compare different image classification algorithms to develop an accurate and real-time environment-adaptive locomotion mode recognition system for robotic exoskeleton control.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: A Experimental Datasetmentioning

confidence: 99%

Computer Vision and Deep Learning for Environment-Adaptive Control of Robotic Lower-Limb Exoskeletons

Laschowski

McNally

Wong

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Many researchers have likewise used a wearable RGB camera for passive environment sensing (Da Silva et al, 2020;Diaz et al, 2018;Khademi and Simon, 2019;Krausz and Hargrove, 2015;Laschowski et al, 2019b;2020b;Novo-Torres et al, 2019;Zhong et al, 2020). Although multicamera systems could be used to capture 3D information (i.e., comparable to how the human visual system uses triangulation for depth perception) (Patla, 1997), each pixel in an RGB image contains only light intensity information.…”

Section: Discussionmentioning

confidence: 99%

“…Compared to radar and laser rangefinders, cameras can provide more detailed information about the field-of-view and detect physical obstacles and terrain changes in peripheral locations (example shown in Figure 3). Most environment recognition systems have used RGB cameras (Da Silva et al, 2020;Diaz et al, 2018;Khademi and Simon, 2019;Krausz and Hargrove, 2015;Laschowski et al, 2019b;2020b;2021b;Novo-Torres et al, 2019;Zhong et al, 2020) and/or 3D depth cameras (Hu et al, 2018;Krausz et al, 2015;Krausz and Hargrove, 2021;Massalin et al, 2018;Varol and Massalin, 2016;Zhang et al, 2019b;2019c;2019d, 2020 mounted on the chest (Krausz et al, 2015;Laschowski et al, 2019b;2020b;2021b), waist (Khademi andSimon, 2019;Krausz et al, 2019;Krausz and Hargrove, 2021;Zhang et al, 2019d), or lower-limbs (Da Silva et al, 2020;Diaz et al, 2018;Massalin et al, 2018;Varol and Massalin, 2016;Zhang et al, 2019b;2019c;2020;Zhong et al, 2020) (see Table 1). Few studies have adopted head-mounted cameras for biomimicry (Novo-Torres et al, 2019;Zhong et al, 2020).…”

Section: Literature Reviewmentioning

confidence: 99%

“…Images were collected during the summer, fall, and winter seasons to capture different weathered surfaces like snow, grass, and multicolored leaves. The image database, which we named "ExoNet", was uploaded to the IEEE DataPort repository and is publicly available for download (Laschowski et al, 2020b). The file size of the uncompressed videos is ~140 GB.…”

Section: Environment Sensingmentioning

confidence: 99%

See 1 more Smart Citation

Environment Classification for Robotic Leg Prostheses and Exoskeletons using Deep Convolutional Neural Networks

Laschowski

McNally

Wong

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Robotic leg prostheses and exoskeletons can provide powered locomotor assistance to older adults and/or persons with physical disabilities. However, the current locomotion mode recognition systems being developed for intelligent high-level control and decision-making use mechanical, inertial, and/or neuromuscular data, which inherently have limited prediction horizons (i.e., analogous to walking blindfolded). Inspired by the human vision-locomotor control system, we designed and evaluated an advanced environment classification system that uses computer vision and deep learning to forward predict the oncoming walking environments prior to physical interaction, therein allowing for more accurate and robust locomotion mode transitions. In this study, we first reviewed the development of the ExoNet database – the largest and most diverse open-source dataset of wearable camera images of indoor and outdoor real-world walking environments, which were annotated using a hierarchical labelling architecture. We then trained and tested over a dozen state-of-the-art deep convolutional neural networks (CNNs) on the ExoNet database for large-scale image classification of the walking environments, including: EfficientNetB0, InceptionV3, MobileNet, MobileNetV2, VGG16, VGG19, Xception, ResNet50, ResNet101, ResNet152, DenseNet121, DenseNet169, and DenseNet201. Lastly, we quantitatively compared the benchmarked CNN architectures and their environment classification predictions using an operational metric called NetScore, which balances the image classification accuracy with the computational and memory storage requirements (i.e., important for onboard real-time inference). Although we designed this environment classification system to support the development of next-generation environment-adaptive locomotor control systems for robotic prostheses and exoskeletons, applications could extend to humanoids, autonomous legged robots, powered wheelchairs, and assistive devices for persons with visual impairments.

show abstract

Convolutional neural network and sensor fusion for obstacle classification in the context of powered prosthetic leg applications

Contreras-Cruz

Novo-Torres

Villarreal

et al. 2023

Computers and Electrical Engineering

View full text Add to dashboard Cite

ExoNet Database: Wearable Camera Images of Human Locomotion Environments

Cited by 30 publications

References 34 publications

Computer Vision and Deep Learning for Environment-Adaptive Control of Robotic Lower-Limb Exoskeletons

Computer Vision and Deep Learning for Environment-Adaptive Control of Robotic Lower-Limb Exoskeletons

Environment Classification for Robotic Leg Prostheses and Exoskeletons using Deep Convolutional Neural Networks

Convolutional neural network and sensor fusion for obstacle classification in the context of powered prosthetic leg applications

Contact Info

Product

Resources

About