Multimodal Gait Recognition Under Missing Modalities

Abstract

Multimodal systems for gait recognition have gained a lot of attention. However, there is a clear gap in the study of missing modalities, which represents real-life scenarios where sensors fail or data get corrupted. Here, we investigate how to handle missing modalities for gait recognition. We propose a single and flexible framework that uses a variable number of input modalities. For each modality, it consists of a branch and a binary unit indicating whether the modality is available; these are gated and merged together. Finally, it generates a single and compact 'multimodal' gait signature that encodes biometric information of the input. Our framework outperforms the state of the art on TUM-GAID and extensive experiments reveal its effectiveness for handling missing modalities even in the multiview setup of CASIA-B. The code is available online: https://github.com/avagait/gaitmiss.

Human Gait Recognition Based on Frontal-View Walking Sequences Using Multi-modal Feature Representations and Learning

“…Comparisons with other existing multi-modal gait recognition methods are given in this section as well on the CASIA-B gait database. Table 6 compares the average recognition results of the proposed method under the aforementioned three walking conditions with the ones of existing other gait recognition that used different combinations of modalities, including MissGait [46], mmGaitSet [47], and GaitPartHybrid [48]. The modalities used in these methods include grayscale images, optical flow images, silhouette images, pose heatmaps, and skeleton images.…”