SeqViews2SeqLabels: Learning 3D Global Features via Aggregating Sequential Views by RNN With Attention

Abstract

Learning 3D global features by aggregating multiple views has been introduced as a successful strategy for 3D shape analysis. In recent deep learning models with end-to-end training, pooling is a widely adopted procedure for view aggregation. However, pooling merely retains the max or mean value over all views, which disregards the content information of almost all views and also the spatial information among the views. To resolve these issues, we propose Sequential Views To Sequential Labels (SeqViews2SeqLabels) as a novel deep learning model with an encoder-decoder structure based on recurrent neural networks (RNNs) with attention. SeqViews2SeqLabels consists of two connected parts, an encoder-RNN followed by a decoder-RNN, that aim to learn the global features by aggregating sequential views and then performing shape classification from the learned global features, respectively. Specifically, the encoder-RNN learns the global features by simultaneously encoding the spatial and content information of sequential views, which captures the semantics of the view sequence. With the proposed prediction of sequential labels, the decoder-RNN performs more accurate classification using the learned global features by predicting sequential labels step by step. Learning to predict sequential labels provides more and finer discriminative information among shape classes to learn, which alleviates the overfitting problem inherent in training using a limited number of 3D shapes. Moreover, we introduce an attention mechanism to further improve the discriminative ability of SeqViews2SeqLabels. This mechanism increases the weight of views that are distinctive to each shape class, and it dramatically reduces the effect of selecting the first view position. Shape classification and retrieval results under three large-scale benchmarks verify that SeqViews2SeqLabels learns more discriminative global features by more effectively aggregating sequential views than state-of-the-art methods.

View N-Gram Network for 3D Object Retrieval

“…In addition, VNN outperforms DeepPano [44], GIFT [2] and RED [4] by 8.6%, 1.7%, and 0.6% in terms of mAP, respectively. The comparison with SeqViews [17] is especially valuable since SeqView leverages a stronger backbone network for view feature extraction. Nevertheless, the proposed method surpasses it by 1.4% in terms of mAP.…”

“…Dai et al [12] propose a siamese CNN-BiLSTM for 3D shape representation learning, where they use BiLSTM to capture features across different views of a 3D shape. In addition, Han et al [17] propose to use an RNN with attention to aggregate sequential views of each 3D object and promising results on several 3D shape retrieval benchmarks are obtained. Leng et al [31] propose a score generation unit to evaluate the quality of the projected image and weight the view image features.…”

Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds From Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction

“…Deep learning models have led to significant progress in feature learning for 3D shapes [13,12,15,14,18,19,10,20,16,11]. Here, we focus on reviewing studies on point clouds.…”

CF-SIS: Semantic-Instance Segmentation of 3D Point Clouds by Context Fusion with Self-Attention

“…In the field of 3D computer vision, there are many studies concerning various representation form of 3D shapes (e.g. voxels [2,17,29], view [3][4][5][6]8] and point cloud [7,15,32]), and in this paper we concern the segmentation task on the specific form of point cloud. Instance segmentation.The studies concerning 3D instance segmentation can be roughly divided into two directions.…”