Local and Holistic Feature Fusion for Occlusion-Robust 3D Ear Recognition

Abstract

Occlusion over ear surfaces results in performance degradation of ear registration and recognition systems. In this paper, we propose an occlusion-resistant three-dimensional (3D) ear recognition system consisting of four primary components: (1) an ear detection component, (2) a local feature extraction and matching component, (3) a holistic matching component, and (4) a decision-level fusion algorithm. The ear detection component is implemented based on faster region-based convolutional neural networks. In the local feature extraction and matching component, a symmetric space-centered 3D shape descriptor based on the surface patch histogram of indexed shapes (SPHIS) is used to generate a set of keypoints and a feature vector for each keypoint. Then, a two-step noncooperative game theory (NGT)-based method is proposed. The proposed symmetric game-based method is effectively applied to determine a set of keypoints that satisfy the rigid constraints from initial keypoint correspondences. In the holistic matching component, a proposed variant of breed surface voxelization is used to calculate the holistic registration error. Finally, the decision-level fusion algorithm is applied to generate the final match scores. Evaluation results from experiments conducted show that the proposed method produces competitive results for partial occlusion on a dataset consisting of natural and random occlusion.

PointNet++ and Three Layers of Features Fusion for Occlusion Three-Dimensional Ear Recognition Based on One Sample per Person

“…A set of point correspondence, which satisfies the joint rigidity constraint, can be obtained from the surface of the objects by implementing a GT framework. Zhu and Mu [4] extended this method using SPHIS and long-distance occlusion-independent SPHIS (LOSPHIS) descriptors to surmount the OSPP challenge in ear recognition. They proposed a two-step non-cooperative game-based (TNG) recognition method.…”

“…As mentioned in previously conducted studies [4], the objective of the TNG method is to obtain a set of keypoint correspondences that can maintain rigid constraints. However, when the TNG method is complete, a set of strategy pairs may exist in which each strategy pair satisfies the following conditions.…”

“…The first image from left to right is an ear image with long-distance object occlusion, whereas the second, third, and fourth images are ear images with short-distance object occlusion. The majority of the existing ear biometrics either exhibit considerably low performances or fail to work with partial data and under OSPP conditions [3], i.e., the per-person registered sample size required for training the majority of the existing ear recognition methods based on deep learning is difficult to satisfy when only OSPP is registered in the gallery [4]. Thus, developing an ear recognition system that can handle partial data problems using OSPP is necessary.…”

“…Thus, developing an ear recognition system that can handle partial data problems using OSPP is necessary. The depth information can effectively assist the separation of occlusion from the ear when the occlusion is far from the ear [4]. However, the separation of the ear from the occlusion as well as the extraction and recognition of the ear surface features become challenging when large occlusions are close to the ear.…”

“…For a given cloud data point, the segmentation component can provide a set of 3D points to represent the ear. For the local and LJS feature extraction and matching component, Herein, inspired by our previous work [4] and PointNet++ [7], we present a PointNet++, local feature, local joint structural (LJS) feature, and holistic feature fusion based on a fully automatic system for OSPP normalization-free 3D ear recognition when the large occlusion is near the ear surface. This system can be referred to as the PointNet++ and three layers of features fusion (PTLFF) system.…”