Toward Characteristic-Preserving Image-Based Virtual Try-On Network

Abstract

Image-based virtual try-on systems for fitting a new in-shop clothes into a person image have attracted increasing research attention, yet is still challenging. A desirable pipeline should not only transform the target clothes into the most fitting shape seamlessly but also preserve well the clothes identity in the generated image, that is, the key characteristics (e.g. texture, logo, embroidery) that depict the original clothes. However, previous image-conditioned generation works fail to meet these critical requirements towards the plausible virtual try-on performance since they fail to handle large spatial misalignment between the input image and target clothes. Prior work explicitly tackled spatial deformation using shape context matching, but failed to preserve clothing details due to its coarse-to-fine strategy. In this work, we propose a new fully-learnable Characteristic-Preserving Virtual Try-On Network (CP-VTON) for addressing all real-world challenges in this task. First, CP-VTON learns a thin-plate spline transformation for transforming the in-shop clothes into fitting the body shape of the target person via a new Geometric Matching Module (GMM) rather than computing correspondences of interest points as prior works did. Second, to alleviate boundary artifacts of warped clothes and make the results more realistic, we employ a Try-On Module that learns a composition mask to integrate the warped clothes and the rendered image to ensure smoothness. Extensive experiments on a fashion dataset demonstrate our CP-VTON achieves the state-ofthe-art virtual try-on performance both qualitatively and quantitatively. Code is available at https://github.com/sergeywong/cp-vton.

Fashion-AttGAN: Attribute-Aware Fashion Editing With Multi-Objective GAN

“…The comparative results of fashion-editing between our Fashion-AttGAN and AttGAN are depicted in Fig.1. From the figures we can observe that: (1) Color edits: the original AttGAN can edit colors of clothes with lighter shades, but not so well for darker shades (row: (1,4,5), column: (7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22)). Whereas our Fashion-AttGAN can modify clothes of almost any shade to other colors as shown in Fig.1(b); (2) Sleeve length: original AttGAN does not present any sleeve-length changes even with careful parameter tuning ( Fig.1(a),column: (3,4,5,6)).…”

“…In this study, we constructed a dataset based on the VI-TON dataset [4,9] which is publically available . Each entry in the dataset consists of an image from VITON, and a vector of attributes, such as "no-sleeve", "shortsleeve","red","blue" and so on.…”

Towards Multi-Pose Guided Virtual Try-On Network

“…Pasting the warped clothes onto the target person directly may lead to generate the artifacts. Learning the composition mask between the warped clothes image and the coarse results also generates the artifacts [8,29] due to the diversity of pose. To solve the above issues, we present a refinement render utilizing multi-pose composition masks to recover the texture details and remove some artifacts.…”

“…However, recent image synthesis approaches [8,29] for virtual try-on mainly focus on the fixed pose and fail to preserve the fine details, such as the clothing of lower-body and the hair of the person lose the details and style, as shown in Figure 4. In order to generate the realistic image, those methods apply a coarse-to-fine network to produce the image conditioned on clothes only.…”

“…In this paper, we study the problem of virtual try-on conditioned on 2D images and arbitrary poses, which aims to learn a mapping function from an input image of a person to another image of the same person with a new outfit and diverse pose, by manipulating the target clothes and pose. Although the image-based virtual try-on with the fixed pose has been studied widely [8,29,37], the task of multi-pose virtual try-on is less explored. In addition, without modeling the mapping of the intricate interplay among of the appearance, the clothes, and the pose, directly using the existing virtual try-on methods to synthesized image based on different poses often result in blurry and artifacts.…”