Dual-level Embedding Alignment Network for 2D Image-Based 3D Object Retrieval

Zhou, Heyu; Liu, An-An; Nie, Weizhi

doi:10.1145/3343031.3351011

Cited by 37 publications

(25 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The learning rate of feature extractor, MI, MDD and SC are set to 0.004, 0.001, 0.04 and 0.0003 under the weight decay 0.0005, 0.8, 0.005 and 0.05, respectively. The view number is empirically set as 12 by following [4]. For the retrieval stage, the pairwise similarity between 2D image and 3D shape can be calculated simply based on the Euclidean distance, and the smaller distance means higher similarity.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…With widespread use of 3D equipments and software [1][2][3], 3D shape retrieval, the task of matching the object (such as 2D image [4], 3D shape [5] or text [6]) from the gallery 3D shape dataset, has drawn especial attention from multimedia and computer vision communities. Despite the deep learning technique has already achieved good performance in 3D shape retrieval, there still exist some issues to be solved , especially for 2D image-based 3D shape retrieval (2D-to-3D) task.…”

Section: Introductionmentioning

confidence: 99%

“…1, the flowerpot can be similar to vase if the upper part of flowerpot is distorted during the translation. Zhou et al [4] solved this problem by narrowing the Euclidean distance between the class centers from two domains. However, the proximity of the class centers is a necessary condition for semantic consistency.…”

Section: Introductionmentioning

confidence: 99%

“…How to enhance the capability of feature representation of each instance for domain adaptation. Most existing 2D-to-3D works [4,11] concentrate on eliminating the domain shift via domain adversarial training [4] and environmental style transfer [11], while paying less attention to enhance the capability of feature representation of each instance. Specifically, they extract features through powerful CNN, which can build up the connection between the feature representation and the category label, and consequently the feature representations tend to preserve the commonality of categories.…”

Section: Introductionmentioning

confidence: 99%

“…When the domain classifier and feature extractor reach an equilibrium, the representation can be domain-invariant. • Dual-level Embedding Alignment (DLEA)[4]: Unlike previous methods, which only eliminate the domain-level shift, this method also considers the class-level alignment by matching the centroid of each category in source and target domain.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Semantic Consistency Guided Instance Feature Alignment for 2D Image-Based 3D Shape Retrieval

Zhou

Nie

Song

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

2D image-based 3D shape retrieval (2D-to-3D) investigates the problem of matching the relevant 3D shapes from gallery dataset when given a query image. Recently, adversarial training and environmental style transfer learning have been successful applied to this task and achieved state-of-the-art performance. However, there still exist two problems. First, previous works only concentrate on the connection between the label and representation, where the unique visual characteristics of each instance are paid less attention. Second, the confused features or the transformed images can only cheat the discriminator but can not guarantee the semantic consistency. In another words, features of 2D desk may be mapped nearby the features of 3D chair. In this paper, we propose a novel semantic consistency guided instance feature alignment network (SC-IFA) to address these limitations. SC-IFA mainly consists of two parts, instance visual feature extraction and cross-domain instance feature adaptation. For the first module, unlike previous methods, which merely employ 2D CNN to extract the feature, we additionally maximize the mutual information between the input and feature to enhance the capability of feature representation for each instance. For the second module, we first introduce the margin disparity discrepancy model to mix up the cross-domain features in an adversarial training way. Then, we design two feature translators to transform the feature from one domain to another domain, and impose the translation loss and correlation loss on the transformed features to preserve the semantic consistency. Extensive experimental results on two benchmarks, MI3DOR and MI3DOR-2, verify SC-IFA is superior to the state-of-the-art methods.

show abstract

Section: Implementation Detailsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Semantic Consistency Guided Instance Feature Alignment for 2D Image-Based 3D Shape Retrieval

Zhou

Nie

Song

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

show abstract

Source‐enhanced prototypical alignment for single image 3D model retrieval

Song

Wang

Zhang

et al. 2022

Computer Animation & Virtual

View full text Add to dashboard Cite

Single image 3D model retrieval has attracted a lot of attentions with the convenience of organizing large-scale unlabeled 3D models. Existing methods transfer the knowledge from well-annotated 2D images (i.e., source domain) to unlabeled 3D models (i.e., target domain) to improve the discriminability of 3D models and align the feature distributions of 2D images and 3D models. However, during the alignment, the feature learning target of improving the discriminability of 3D models sometimes confuses the boundaries between 2D image categories, where prior methods ignore keeping the discriminability of 2D images. Motivated by this observation, we propose a source-enhanced prototypical alignment framework to first remain the discriminability of 2D images and then guide the category-level cross-domain alignment with better image representations. Specifically, a novel separation and compactness loss is proposed for images to separate the samples from different categories and compact the samples within the same category. Then we perform prototypical alignment to make 2D image features assist in the discriminative feature learning for 3D models. We evaluate the proposed method on the commonly used cross-domain 3D model retrieval benchmarks, namely MI3DOR and MI3DOR-2, and the results demonstrate the effectiveness of the proposed method.

show abstract

Instance-level 3D shape retrieval from a single image by hybrid-representation-assisted joint embedding

Zou

Liu

2020

Vis Comput

View full text Add to dashboard Cite

Dual-level Embedding Alignment Network for 2D Image-Based 3D Object Retrieval

Cited by 37 publications

References 14 publications

Semantic Consistency Guided Instance Feature Alignment for 2D Image-Based 3D Shape Retrieval

Semantic Consistency Guided Instance Feature Alignment for 2D Image-Based 3D Shape Retrieval

Source‐enhanced prototypical alignment for single image 3D model retrieval

Instance-level 3D shape retrieval from a single image by hybrid-representation-assisted joint embedding

Contact Info

Product

Resources

About