Similarity Reasoning and Filtration for Image-Text Matching

Diao, Haiwen; Zhang, Ying; Ma, Lin; Lu, Huchuan

doi:10.48550/arxiv.2101.01368

Cited by 5 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unsupervised Learning for Dense Prediction Learning fine-grained semantic correspondence is essential for dense prediction tasks, e.g., object detection [15,38], semantic segmentation [6], etc.Recent studies have proposed various paradigms to tackle this problem [33,28,36,11,59]. However, most of these methods require a pre-trained object detector to generate proposals of interested objects.…”

Section: Related Workmentioning

confidence: 99%

Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

Wang¹,

Zhou²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning medical visual representations directly from paired radiology reports has become an emerging topic in representation learning. However, existing medical image-text joint learning methods are limited by instance or local supervision analysis, ignoring disease-level semantic correspondences. In this paper, we present a novel Multi-Granularity Cross-modal Alignment (MGCA) framework for generalized medical visual representation learning by harnessing the naturally exhibited semantic correspondences between medical image and radiology reports at three different levels, i.e., pathological region-level, instance-level, and disease-level. Specifically, we first incorporate the instance-wise alignment module by maximizing the agreement between image-report pairs. Further, for token-wise alignment, we introduce a bidirectional cross-attention strategy to explicitly learn the matching between fine-grained visual tokens and text tokens, followed by contrastive learning to align them. More important, to leverage the high-level inter-subject relationship semantic (e.g., disease) correspondences, we design a novel cross-modal disease-level alignment paradigm to enforce the cross-modal cluster assignment consistency. Extensive experimental results on seven downstream medical image datasets covering image classification, object detection, and semantic segmentation tasks demonstrate the stable and superior performance of our framework.

show abstract

Section: Related Workmentioning

confidence: 99%

Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

Wang¹,

Zhou²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Following up, [20] proposed a stacked cross attention network to model the latent alignments between image regions and words. Additional models have explored the roll of attention mechanisms [24,35,44,45,47], and Graph Convolutional Neural Networks (GCN) [7,19,21,25]. External modules have been explored to improve retrieval results such as the usage of an iterative recurrent attention module [3] and an external consensus knowledge base [41].…”

Section: Related Workmentioning

confidence: 99%

“…In this section, we compare the behaviour of existing systems by evaluating them on the newly proposed metrics. We evaluate the following state-of-the-art models: VSE++ [9], SCAN [20], VSRN [21], CVSE [41], SGR and SAF [7]. The experiment depicting the top-5 text-to-image retrieval scores for non-ground truth relevant items is shown in It is worth noting in Figure 3 that according to the recall (R@5), the models have a steady raise in recall scores as the number of relevant images m increase.…”

Section: Insights On State-of-the-art Retrievalmentioning

confidence: 99%

“…We evaluate all the models on the standard 1K test set split of each dataset. We employ two state-of-theart methods, CVSE [41] and SGR [7] for experimentation. Similar to the previous section, all the experiments are performed with public available code as described by the authors disregarding the adoption or not of our formulation.…”

Section: Reduced Data Scenariomentioning

confidence: 99%

See 1 more Smart Citation

Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching

Biten¹,

Mafla²,

Gómez³

et al. 2021

Preprint

View full text Add to dashboard Cite

The task of image-text matching aims to map representations from different modalities into a common joint visualtextual embedding. However, the most widely used datasets for this task, MSCOCO and Flickr30K, are actually image captioning datasets that offer a very limited set of relationships between images and sentences in their ground-truth annotations. This limited ground truth information forces us to use evaluation metrics based on binary relevance: given a sentence query we consider only one image as relevant. However, many other relevant images or captions may be present in the dataset. In this work, we propose two metrics that evaluate the degree of semantic relevance of retrieved items, independently of their annotated binary relevance. Additionally, we incorporate a novel strategy that uses an image captioning metric, CIDEr, to define a Semantic Adaptive Margin (SAM) to be optimized in a standard triplet loss. By incorporating our formulation to existing models, a large improvement is obtained in scenarios where available training data is limited. We also demonstrate that the performance on the annotated image-caption pairs is maintained while improving on other non-annotated relevant items when employing the full training set. Code with our metrics and adaptive margin formulation will be made public.

show abstract

“…Most existing instance retrieval solutions use Deep Metric Learning methodology [1,3,6,7,13,16], in which a deep learning model is trained to transform images to a vector representation, so that samples from the same class are close to each other. At the retrieval stage, the query embedding is scored against all gallery embeddings and the most similar ones are returned.…”

Section: Introductionmentioning

confidence: 99%

On the Unreasonable Effectiveness of Centroids in Image Retrieval

Wieczorek¹,

Rychalska²,

Dąbrowski³

2021

Preprint

View full text Add to dashboard Cite

Image retrieval task consists of finding similar images to a query image from a set of gallery (database) images. Such systems are used in various applications e.g. person re-identification (ReID) or visual product search. Despite active development of retrieval models it still remains a challenging task mainly due to large intra-class variance caused by changes in view angle, lighting, background clutter or occlusion, while inter-class variance may be relatively low. A large portion of current research focuses on creating more robust features and modifying objective functions, usually based on Triplet Loss. Some works experiment with using centroid/proxy representation of a class to alleviate problems with computing speed and hard samples mining used with Triplet Loss. However, these approaches are used for training alone and discarded during the retrieval stage. In this paper we propose to use the mean centroid representation both during training and retrieval. Such an aggregated representation is more robust to outliers and assures more stable features. As each class is represented by a single embeddingthe class centroid -both retrieval time and storage requirements are reduced significantly. Aggregating multiple embeddings results in a significant reduction of the search space due to lowering the number of candidate target vectors, which makes the method especially suitable for production deployments. Comprehensive experiments conducted on two ReID and Fashion Retrieval datasets demonstrate effectiveness of our method, which outperforms the current stateof-the-art. We propose centroid training and retrieval as a viable method for both Fashion Retrieval and ReID applications.

show abstract

Similarity Reasoning and Filtration for Image-Text Matching

Cited by 5 publications

References 0 publications

Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching

On the Unreasonable Effectiveness of Centroids in Image Retrieval

Contact Info

Product

Resources

About