OpenMPR: Recognize places using multimodal data for people with visual impairments

Abstract

Place recognition plays a crucial role in navigational assistance, and is also a challenging issue of assistive technology. The place recognition is prone to erroneous localization owing to various changes between database and query images. Aiming at the wearable assistive device for visually impaired people, we propose an open-sourced place recognition algorithm OpenMPR, which utilizes the multimodal data to address the challenging issues of place recognition.Compared with conventional place recognition, the proposed OpenMPR not only leverages multiple effective descriptors, but also assigns different weights to those descriptors in image matching. Incorporating GNSS data into the algorithm, the cone-based sequence searching is used for robust place recognition. The experiments illustrate that the proposed algorithm manages to solve the place recognition issue in the real-world scenarios and surpass the state-of-the-art algorithms in terms of assistive navigation performance. On the real-world testing dataset, the online OpenMPR achieves 88.7% precision at 100% recall without illumination changes, and achieves 57.8% precision at 99.3% recall with illumination changes. The OpenMPR is available at https://github.com/chengricky/OpenMultiPR.

Uncertainty-Aware Visual Perception System for Outdoor Navigation of the Visually Challenged

“…In detail, the overall accuracy of the system based on the proposed method was estimated to be 85.7%, when the methodology proposed in [38] produced an accuracy of 72.6%, based on the dataset described in Section 4.1. Additionally, in contrast to other methodologies such as [2,26,27,31,32], the proposed obstacle detection and recognition system is solely based on visual cues obtained using only an RGB-D sensor, minimizing the computational and energy resources required for the integration, fusion, and synchronization of multiple sensors.…”

“…In [26], a scene perception system based on a multi-modal fusion-based framework for object detection and classification was proposed. The authors of [27] aimed to the development of a method integrated in a wearable device for the efficient place recognition using multimodal data. In [28], a unifying terrain awareness framework was proposed, extending the basic vision system based on an IR RGB-D sensor proposed in [10] and aiming at achieving efficient semantic understanding of the environment.…”

Unifying Visual Localization and Scene Recognition for People With Visual Impairment

“…The key position prediction algorithm [33] uses conventional image descriptors based on multimodal images and GNSS (global navigation satellite system) data to localize the users at the user-defined key positions. Subsequently, we proposed the improved localization approach OpenMPR [34], where the off-the-shelf CNN descriptors along with other multimodal descriptors are optimized in the sequence matching pipeline by the genetic algorithm. Hu et al [35] introduced the panoramic annular camera to visual odometry so as to robustify the positioning and mapping performance in the assistive navigation.…”

“…Finally, the matching score of the latent pair is evaluated by windowed uniqueness thresholding [37] to remove low-confidence matching results. The sequence matching parameters follow those optimized parameters in [34]. The parameters of cone region boundaries v min and v max are set as 0.4 and 2.5 respectively.…”

A Panoramic Localizer Based on Coarse-to-Fine Descriptors for Navigation Assistance

“…In addition, it is possible to effectively tackle both "which point you are" and "what you can perceive in the surroundings" issues. Efficient and robust VPR algorithms can be integrated on navigation devices with mobile computer platforms [14][15][16][17]. VPR may perfectly alleviate the potential hazards caused by inaccurate localization, providing navigation assistance for high-level applications like self-driving cars [2] and guidance of VIP [1].…”