Learning by Association - A versatile semi-supervised training method for neural networks

Häusser, Philip; Mordvintsev, Alexander; Cremers, Daniel

doi:10.48550/arxiv.1706.00909

Cited by 2 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, when the two are trained jointly (Full-TST), we see a significant boost in performance. Lastly, in the original LBA work [11], the authors found that additional roundtrips do not boost performance for same-modal data. On the contrary, in working with multimodal data, we can see that using additional roundtrips in the MM model results in much higher retrieval performance.…”

Section: C1 Quantitative Retrieval Resultsmentioning

confidence: 96%

“…We "a tall brown table" "a brown table with four legs" Text Encoder Shape Encoder would like our joint embedding to: i) cluster similar text together and similar shapes together, ii) keep text descriptions close to their associated shape instance, and iii) separate text from shapes that are not similar. To address i), we generalize the learning by association approach [11] to handle multiple modalities (Sec. 4.1) and to use instance-level associations between a description and a shape (Sec.…”

Section: Joint Text-3d Shape Representation Learningmentioning

confidence: 99%

“…We generalize the same-modal data association approach of Haeusser et al [11] to learn cross-modal text-to-shape associations (see Fig. 3).…”

Section: Learning Cross-modal Associationsmentioning

confidence: 99%

“…Our approach significantly outperforms baselines from prior work. We compare our model with the deep symmetric structured joint embedding (DS-SJE) by Reed et al [4] and the original formulation of learning by association [11] with only TST round trips (LBA-TST). We also decompose our full model into the following: association learning only with both round trips (LBA-MM), metric learning (ML) only, combined LBA and ML with TST round trip only (Full-TST), and the full combined model (Full-MM).…”

Section: Retrieval Taskmentioning

confidence: 99%

“…To do this, we take inspiration from recent work on learning by association [11] to establish implicit cross-modal links between similar descriptions and shape instances, and combine this with a metric learning scheme [12] which strengthens links between similar instances in each modality (see Fig. 1(b)).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

Chen

Choy

Savva

et al. 2019

Lecture Notes in Computer Science

106

View full text Add to dashboard Cite

We present a method for generating colored 3D shapes from natural language. To this end, we first learn joint embeddings of freeform text descriptions and colored 3D shapes. Our model combines and extends learning by association and metric learning approaches to learn implicit cross-modal connections, and produces a joint representation that captures the many-to-many relations between language and physical properties of 3D shapes such as color and shape. To evaluate our approach, we collect a large dataset of natural language descriptions for physical 3D objects in the ShapeNet dataset. With this learned joint embedding we demonstrate text-to-shape retrieval that outperforms baseline approaches. Using our embeddings with a novel conditional Wasserstein GAN framework, we generate colored 3D shapes from text. Our method is the first to connect natural language text with realistic 3D objects exhibiting rich variations in color, texture, and shape detail.

show abstract

Section: C1 Quantitative Retrieval Resultsmentioning

confidence: 96%

Section: Joint Text-3d Shape Representation Learningmentioning

confidence: 99%

“…We generalize the same-modal data association approach of Haeusser et al [11] to learn cross-modal text-to-shape associations (see Fig. 3).…”

Section: Learning Cross-modal Associationsmentioning

confidence: 99%

Section: Retrieval Taskmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

Chen

Choy

Savva

et al. 2019

Lecture Notes in Computer Science

106

View full text Add to dashboard Cite

show abstract

A Review of Multi-Modal Learning from the Text-Guided Visual Processing Viewpoint

Ullah

Lee

et al. 2022

Sensors

View full text Add to dashboard Cite

For decades, co-relating different data domains to attain the maximum potential of machines has driven research, especially in neural networks. Similarly, text and visual data (images and videos) are two distinct data domains with extensive research in the past. Recently, using natural language to process 2D or 3D images and videos with the immense power of neural nets has witnessed a promising future. Despite the diverse range of remarkable work in this field, notably in the past few years, rapid improvements have also solved future challenges for researchers. Moreover, the connection between these two domains is mainly subjected to GAN, thus limiting the horizons of this field. This review analyzes Text-to-Image (T2I) synthesis as a broader picture, Text-guided Visual-output (T2Vo), with the primary goal being to highlight the gaps by proposing a more comprehensive taxonomy. We broadly categorize text-guided visual output into three main divisions and meaningful subdivisions by critically examining an extensive body of literature from top-tier computer vision venues and closely related fields, such as machine learning and human–computer interaction, aiming at state-of-the-art models with a comparative analysis. This study successively follows previous surveys on T2I, adding value by analogously evaluating the diverse range of existing methods, including different generative models, several types of visual output, critical examination of various approaches, and highlighting the shortcomings, suggesting the future direction of research.

show abstract

Learning by Association - A versatile semi-supervised training method for neural networks

Cited by 2 publications

References 0 publications

Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

A Review of Multi-Modal Learning from the Text-Guided Visual Processing Viewpoint

Contact Info

Product

Resources

About