Multi-view dual attention network for 3D object recognition

Abstract

The existing view-based 3D object classification and recognition methods ignore the inherent hierarchical correlation and distinguishability of views, making it difficult to further improve the classification accuracy. In order to solve this problem, this paper proposes an end-to-end multi-view dual attention network framework for high-precision recognition of 3D objects. On one hand, we obtain three feature layers of query, key, and value through the convolution layer. The spatial attention matrix is generated by the key-value pairs of query and key, and each feature in the value of the original feature space branch is assigned different importance, which clearly captures the prominent detail features in the view, generates the view space shape descriptor, and focuses on the detail part of the view with the feature of category discrimination. On the other hand, a channel attention vector is obtained by compressing the channel information in different views, and the attention weight of each view feature is scaled to find the correlation between the target views and focus on the view with important features in all views. Integrating the two feature descriptors together to generate global shape descriptors of the 3D model, which has a stronger response to the distinguishing features of the object model and can be used for high-precision 3D object recognition. The proposed method achieves an overall accuracy of 96.6% and an average accuracy of 95.5% on the open-source ModelNet40 dataset, compiled by Princeton University when using Resnet50 as the basic CNN model. Compared with the existing deep learning methods, the experimental results demonstrate that the proposed method achieves state-of-the-art performance in the 3D object classification accuracy.

Trajectory-Aware Body Interaction Transformer for Multi-Person Pose Forecasting

“…All the above methods only model local pose dynamics, ignoring global body translation and inter-individual body interaction. However, learning both local and global pose dynamics and modeling fine-grained human-human interaction are essential for comprehending human behavior in a complex 3D environment [2,39].…”

“…Guo et al [15] present a collaborative prediction task and use a two-branch attention network for the prediction of two interacted persons. Wang et al [39] present a Transformer-based framework to forecast multi-person motion in a scenario with more people. Furthermore, this method produces unrealistic poses since they solely concen-Figure 2.…”

“…For simplicity, we omit subscript p when p only represents an arbitrary person, e.g., taking x p 1:t as x 1:t . Instead of absolute joint positions in the world coordinate, we use y i = x i+1 − x i to obtain instantaneous pose displacement at time i, which will provides more valuable dynamics information [34,39]. The whole displacement sequence is defined as Y 1:T = {y 1 , y 2 , ..., y T }.…”

“…For example, Guo et al [15] propose a collaborative prediction task and perform future motion prediction for only two interacted dancers, which inevitably ignores low interaction influence on one's future behavior. Wang et al [39] use local and global Transformers to learn indi-vidual motion and social interactions separately in a crowd scene. The aforementioned methods ignore the interactive influences of body parts and only learn temporal and social relationships without modeling fine-grained body interaction, which makes it difficult to capture complex interaction dependencies.…”

Show Me What I Like: Detecting User-Specific Video Highlights Using Content-Based Multi-Head Attention

“…An attention mechanism adaptively weighs the keys of different key-value pairs based on their relative importance to a given query to predict the most suitable responses to the query [45]. Depending on the data paradigm of the key, the value, and the query, attention mechanisms are used in a wide variety of tasks, including tasks in natural language understanding [9], text-based image and video retrieval [4], object and action recognition in images and videos [46,40], and visual question answering [57]. In the case of userspecific highlight detection, the key, value, and query need to be based on the video contents, i.e., follow the paradigm of content-based highlight detection [42,37,2] to perform meaningful retrieval of the highlightable clips per user.…”