Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation

Wei, Yuxiang; Shi, Yupeng; Liu, Xiao; Ji, Zhilong; Gao, Yuan; Wu, Zhongqin; Zuo, Wangmeng

doi:10.1109/iccv48922.2021.00665

Cited by 32 publications

(45 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Jacobi orthogonal regularization search algorithm [ 28 ] exploits the gradient relationship between the input and output feature maps of the generator network rather than the parameter information of a particular layer, which ensures the integrity of the method on the model. Jacobian Decomposition GAN [ 29 ] discovered and demonstrated that the eigenvectors obtained by the Jacobian matrix decomposition of the feature maps generated by the GAN generator network correspond to the semantic attribute direction of the image.…”

Section: Methodsmentioning

confidence: 99%

“…Hessian Penalty GAN [ 27 ] adopts the idea of the Jacobian matrix to restrict each dimension of the input latent space by orthogonal regularization, the method achieves different semantic attributes of the generated image controlled by different dimensions of the latent space, but the restriction of different dimensions to control a single semantics is too strict, and the editing does not work well on models with large hidden space dimensions such as StyleGAN. OroJaRGAN [ 28 ] relaxes the orthogonal regularization constraints on the basis of Hessian Penalty GAN, the method achieves good results on large GAN models.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Generated Image Editing Method Based on Global-Local Jacobi Disentanglement for Machine Learning

Zhang

Wang

et al. 2023

Sensors

View full text Add to dashboard Cite

Accurate semantic editing of the generated images is extremely important for machine learning and sample enhancement of big data. Aiming at the problem of semantic entanglement in generated image latent space of the StyleGAN2 network, we proposed a generated image editing method based on global-local Jacobi disentanglement. In terms of global disentanglement, we extract the weight matrix of the style layer in the pre-trained StyleGAN2 network; obtain the semantic attribute direction vector by using the weight matrix eigen decomposition method; finally, utilize this direction vector as the initialization vector for the Jacobi orthogonal regularization search algorithm. Our method improves the speed of the Jacobi orthogonal regularization search algorithm with the proportion of effective semantic attribute editing directions. In terms of local disentanglement, we design a local contrast regularized loss function to relax the semantic association local area and non-local area and utilize the Jacobi orthogonal regularization search algorithm to obtain a more accurate semantic attribute editing direction based on the local area prior MASK. The experimental results show that the proposed method achieves SOTA in semantic attribute disentangled metrics and can discover more accurate editing directions compared with the mainstream unsupervised generated image editing methods.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Generated Image Editing Method Based on Global-Local Jacobi Disentanglement for Machine Learning

Zhang

Wang

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…Wei et al [64] propose an orthogonal Jacobian regularization (OroJaR) to enforce disentanglement for generative models. They employ the Jacobian matrix of the output with respect to the input (i.e., latent variables for representation) to measure the output changes caused by the variations in the input.…”

Section: Generative Adversarial Network (Gan) Based Approachesmentioning

confidence: 99%

“…Vectorwise two or more MAP-IVR [83], DRNET [84], DR-GAN [65], DRANet [21], Lee et al [8], Liu et al [22], Singh et al [73] each latent variable aligns to one coarse-grained semantic meaning real scenes Dimensionwise one VAE-based methods, InfoGAN [9], IB-GAN [61], Zhu et al [19], InfoGAN-CR [62], PS-SC GAN [63], Wei et al [64], DNA-GAN [66] each dimension aligns to one fine-grained semantic meaning synthetic and simple datasets empirical case and the desired case. Explicitness focuses on the coverage of latent representation with respect to generative factors.…”

Section: Dimension Of Each Latent Factor Representative Work Semantic...mentioning

confidence: 99%

Disentangled Representation Learning for Recommendation

Wang

Chen

Zhou

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Disentangled Representation Learning (DRL) aims to learn a model capable of identifying and disentangling the underlying factors hidden in the observable data in representation form. The process of separating underlying factors of variation into variables with semantic meaning benefits in learning explainable representations of data, which imitates the meaningful understanding process of humans when observing an object or relation. As a general learning strategy, DRL has demonstrated its power in improving the model explainability, controlability, robustness, as well as generalization capacity in a wide range of scenarios such as computer vision, natural language processing, data mining etc. In this article, we comprehensively review DRL from various aspects including motivations, definitions, methodologies, evaluations, applications and model designs. We discuss works on DRL based on two well-recognized definitions, i.e., Intuitive Definition and Group Theory Definition. We further categorize the methodologies for DRL into four groups, i.e., Traditional Statistical Approaches, Variational Auto-encoder Based Approaches, Generative Adversarial Networks Based Approaches, Hierarchical Approaches and Other Approaches. We also analyze principles to design different DRL models that may benefit different tasks in practical applications. Finally, we point out challenges in DRL as well as potential research directions deserving future investigations. We believe this work may provide insights for promoting the DRL research in the community.

show abstract

“…Representation disentanglement is an important line of research in TST, which disentangles content and attribute representations (John et al, 2018b). Many disentanglement approaches are proposed to minimize the dependence between these two representations, such as mutual information (Yuan et al, 2020) and orthogonality (Wei et al, 2021). CTG controls the text generation of language models by smart prompt design (Li & Liang, 2021;Shin et al, 2020) or training conditioned on the controllable variables (Li et al, 2020;Hu & Li, 2021).…”

Section: Related Workmentioning

confidence: 99%

Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation

Li¹,

Qu²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we propose a variational autoencoder with disentanglement priors, VAE-DPRIOR , for conditional natural language generation with none or a handful of task-specific labeled examples. In order to improve compositional generalization, our model performs disentangled representation learning by introducing a prior for the latent content space and another prior for the latent label space. We show both empirically and theoretically that the conditional priors can already disentangle representations even without specific regularizations as in the prior work. We can also sample diverse content representations from the content space without accessing data of the seen tasks, and fuse them with the representations of novel tasks for generating diverse texts in the low-resource settings. Our extensive experiments demonstrate the superior performance of our model over competitive baselines in terms of i) data augmentation in continuous zero/fewshot learning, and ii) text style transfer in both zero/few-shot settings.

show abstract

Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation

Cited by 32 publications

References 17 publications

Generated Image Editing Method Based on Global-Local Jacobi Disentanglement for Machine Learning

Generated Image Editing Method Based on Global-Local Jacobi Disentanglement for Machine Learning

Disentangled Representation Learning for Recommendation

Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation

Contact Info

Product

Resources

About