Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval

Liu, Qing; Xie, Lingxi; Wang, Huiyu; Yuille, Alan

doi:10.1109/iccv.2019.00376

Cited by 99 publications

(103 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The application of zero-shot in super-resolution, i.e, the ZSSR prescription [23], is among the most widely used models in superresolution and has gained increasing interest recently. In addition, the majority of zero-shot methods that have provided a large number of excellent results in recent years are mainly based on segmentation [39], emotion recognition [40], object detection [41], image retrieval [42][43][44][45], image classification [46][47][48] and intelligent learning in machines or robots [49]. In the ZSSR formalism, LR images are downsampled to generate many lower-resolution images (I = I 0 , I 1 , I 2 , ..., I n ), which serve as the HR supervision information called "HR fathers, " then, each HR father is downscaled by the required scale factor s to obtain the corresponding "LR sons.…”

Section: Zero-shotmentioning

confidence: 99%

A Single Historical Painting Super-Resolution via a Reference-Based Zero-Shot Network

Shi¹,

Xu²,

Zhang³

et al. 2021

IJCIS

View full text Add to dashboard Cite

As an important part of human cultural heritage, many ancient paintings have suffered from various deteriorations that have led to texture blurring, color fading, etc. Single image super-resolution (SISR) which aims to recover a high-resolution (HR) version from a low-resolution (LR) input is actively engaged in the digital preservation of cultural relics. Currently, only traditional superresolution is widely studied and used in cultural heritage, and it is difficult to apply learning-based SISR to unique historical paintings because of the absence of both ground truth and datasets. Fortunately, the recently proposed ZSSR method suggests that it is feasible to generate a small dataset and extract self-supervised information from a single image. However, when applied to the preservations of historical paintings, the performance of ZSSR is highly limited due to the lack of image knowledge. To address the above issues and to unleash the great potential of learning-based methods in heritage conservation, we present Ref-ZSSR, which is the first attempt to combine zero-shot and reference-based methods to achieve SISR. In our model, both global and local multi-scale similar information is fully exploited from the painting itself. In an end-to-end manner, this information provides consistent artistic style image knowledge and helps synthesize SR images with sharp texture details. Compared with the ZSSR method, our approach shows improvement in both precision (approximately 4.68 dB for scale ×2) and visual perception. It is worth mentioning that all image knowledge required in our method can be extracted from the painting itself, i.e., external examples are not required. Therefore, this approach can be easily generalized to any damaged historical paintings, broken murals, noisy old photos, incomplete art works, etc.

show abstract

Section: Zero-shotmentioning

confidence: 99%

A Single Historical Painting Super-Resolution via a Reference-Based Zero-Shot Network

Shi¹,

Xu²,

Zhang³

et al. 2021

IJCIS

View full text Add to dashboard Cite

show abstract

“…Learning a compatibility or a matching function between multiple modalities in zero-shot scenario (Shen et al 2018;Dey et al 2019;Liu et al 2019) requires structure in the class embedding space where the image features are mapped to. Attributes provide one such a structured class embedding space (Lampert et al 2014), however obtaining attributes requires costly human annotation.…”

Section: Selection Of Side Informationmentioning

confidence: 99%

Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-Based Image Retrieval

Dutta

Akata

2020

Int J Comput Vis

View full text Add to dashboard Cite

Low-shot sketch-based image retrieval is an emerging task in computer vision, allowing to retrieve natural images relevant to hand-drawn sketch queries that are rarely seen during the training phase. Related prior works either require aligned sketch-image pairs that are costly to obtain or inefficient memory fusion layer for mapping the visual information to a semantic space. In this paper, we address any-shot, i.e. zero-shot and few-shot, sketch-based image retrieval (SBIR) tasks, where we introduce the few-shot setting for SBIR. For solving these tasks, we propose a semantically aligned paired cycle-consistent generative adversarial network (SEM-PCYC) for any-shot SBIR, where each branch of the generative adversarial network maps the visual information from sketch and image to a common semantic space via adversarial training. Each of these branches maintains cycle consistency that only requires supervision at the category level, and avoids the need of aligned sketch-image pairs. A classification criteria on the generators’ outputs ensures the visual to semantic space mapping to be class-specific. Furthermore, we propose to combine textual and hierarchical side information via an auto-encoder that selects discriminating side information within a same end-to-end model. Our results demonstrate a significant boost in any-shot SBIR performance over the state-of-the-art on the extended version of the challenging Sketchy, TU-Berlin and QuickDraw datasets.

show abstract

“…To the best of our knowledge [5,4,21,25] are presently the state-of-the-art methodologies that exist in the literature for zero-shot SBIR. It may be noted that while [5,21] use a generative adversarial network (GAN) based approach for this task, [4] use a graph-convolution network for aligning sketches and images in the shared semantic space.…”

Section: Related Workmentioning

confidence: 99%

CrossATNet - a novel cross-attention based framework for sketch-based image retrieval

Chaudhuri

Banerjee

Bhattacharya

et al. 2020

Image and Vision Computing

View full text Add to dashboard Cite

We propose a novel framework for cross-modal zero-shot learning (ZSL) in the context of sketch-based image retrieval (SBIR). Conventionally, the SBIR schema mainly considers simultaneous mappings among the two image views and the semantic side information. Therefore, it is desirable to consider fine-grained classes mainly in the sketch domain using highly discriminative and semantically rich feature space. However, the existing deep generative modelling based SBIR approaches majorly focus on bridging the gaps between the seen and unseen classes by generating pseudo-unseen-class samples. Besides, violating the ZSL protocol by not utilizing any unseen-class information during training, such techniques do not pay explicit attention to modelling the discriminative nature of the shared space. Also, we note that learning a unified feature space for both the multi-view visual data is a tedious task considering the significant domain difference between sketches and the colour images. In this respect, as a remedy, we introduce a novel framework for zero-shot SBIR. While we define a cross-modal triplet loss to ensure the discriminative nature of the shared space, an innovative cross-modal attention learning strategy is also proposed to guide feature extraction from the image domain exploiting information from the respective sketch counterpart. In order to preserve the semantic consistency of the shared space, we consider a graph CNN based module which propagates the semantic class topology to the shared space. To ensure an improved response time during inference, we further explore the possibility of representing the shared space in terms of hash-codes. Experimental results obtained on the benchmark TU-Berlin and the Sketchy datasets confirm the superiority of CrossATNet in yielding the state-of-the-art results.

show abstract

Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval

Cited by 99 publications

References 38 publications

A Single Historical Painting Super-Resolution via a Reference-Based Zero-Shot Network

A Single Historical Painting Super-Resolution via a Reference-Based Zero-Shot Network

Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-Based Image Retrieval

CrossATNet - a novel cross-attention based framework for sketch-based image retrieval

Contact Info

Product

Resources

About