Semantic enhanced Top-k similarity search on weighted HIN

Zhang, Yun; Yu, Minghe; Zhang, Tiancheng; Yu, Ge

doi:10.1007/s00521-022-07339-6

Cited by 6 publications

(8 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Concretely, these PETL methods insert lightweight adaptation modules into the pertrained models, freeze the pretrained weights, and finetune these modules end-to-end to adapt to downstream tasks. Recent work has verified the effectiveness of these PETL methods on ViT (Jia et al 2022;Zhang, Zhou, and Liu 2022), but we raise the question: Are these modules designed for the language models optimal for the vision models as well?…”

Section: Introductionmentioning

confidence: 88%

“…NOAH (Zhang, Zhou, and Liu 2022) is a newly proposed PETL method for ViT, which combines the above three module together and performs neural architecture search on hidden dimension h of Adapter, rank r of LoRA, and prompt length l of VPT.…”

Section: Parameter-efficient Transfer Learningmentioning

confidence: 99%

“…VTAB-1k Following (Jia et al 2022;Zhang, Zhou, and Liu 2022), we resize the images to 224 × 224, and then normalize them with ImageNet's mean and standard deviation.…”

Section: B3 Data Augmentationmentioning

confidence: 99%

See 2 more Smart Citations

Convolutional Bypasses Are Better Vision Transformer Adapters

Jie¹,

Deng²

2022

Preprint

View full text Add to dashboard Cite

The pretrain-then-finetune paradigm has been widely adopted in computer vision. But as the size of Vision Transformer (ViT) grows exponentially, the full finetuning becomes prohibitive in view of the heavier storage overhead. Motivated by parameter-efficient transfer learning (PETL) on language transformers, recent studies attempt to insert lightweight adaptation modules (e.g., adapter layers or prompt tokens) to pretrained ViT and only finetune these modules while the pretrained weights are frozen. However, these modules were originally proposed to finetune language models. Although ported well to ViT, their design lacks prior knowledge for visual tasks. In this paper, we propose to construct Convolutional Bypasses (Convpass) in ViT as adaptation modules, introducing only a small amount (less than 0.5% of model parameters) of trainable parameters to adapt the large ViT. Different from other PETL methods, Convpass benefits from the hard-coded inductive bias of convolutional layers and thus is more suitable for visual tasks, especially in the lowdata regime. Experimental results on VTAB-1k benchmark and few-shot learning datasets demonstrate that Convpass outperforms current language-oriented adaptation modules, demonstrating the necessity to tailor vision-oriented adaptation modules for vision models.

show abstract

Section: Introductionmentioning

confidence: 88%

Section: Parameter-efficient Transfer Learningmentioning

confidence: 99%

See 1 more Smart Citation

Convolutional Bypasses Are Better Vision Transformer Adapters

Jie¹,

Deng²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For all experiments with SAFT, we use Vit-B [12], a 12 block Vision Transformer, pretrained on the ImageNet-21K dataset as the backbone model. To train our models, we follow in the footsteps of previous works [20], [11], using the AdamW optimizer with a learning rate of 1e-3 and a cosine learning rate scheduler with a decay cycle of 0.9, and a minimum learning rate of 1e-5. For the hyper-parameters, we fix rank at 24, with a scale of 1 for Oxford Flowers and 10 for Caltech101.…”

Section: Implementation Detailsmentioning

confidence: 99%

Self-Attention Factor-Tuning for Parameter Efficient Fine-Tuning

Abohwo

2023

Preprint

View full text Add to dashboard Cite

Transformers have revolutionized the fields of natural language processing and computer vision - a result of their ability to capture long-range dependencies with their key innovation: the attention mechanism. Despite the success of these models, their growing complexity has led to an ever-increasing need for processing power, making their practical applications less feasible. In recent years, tensor decomposition based parameter-efficient fine-tuning techniques have emerged as a promising solution to the computational bottleneck. In this research, we investigate the use of a specialized version of Factor Tuning that focuses exclusively on tensor decomposition for attention layers, which we refer to as Self-Attention Factor-Tuning. This selective decomposition not only allows for more inter-layer weight redundancy reduction, but also improves the model's efficiency and scalability. To evaluate the effectiveness of our approach, we conduct experiments with Vision Transformers using several benchmark datasets for image classification. The results demonstrate that the proposed framework effectively reduces the number of parameters required to train a transformer, outperforming the original Factor-Tuning paradigm as well as various other state-of-the-art techniques such as Low-Rank Adaptation whilst using significantly less parameters.

show abstract

“…In particular, CLIP [28], a recently developed vision language model, demonstrates its generalization capability through a remarkable zero-shot performance in decoding the high-level semantics from the textual and visual data. In addition, when it comes to transferring knowledge of the pre-trained foundation model to downstream tasks, the prompt engineering [29][30][31][32][33] has shown efficacy. The underlying idea is to learn a suitable textual context as prompts to build around the main text for quarrying the text encoder of the model.…”

Section: Introductionmentioning

confidence: 99%

Distribution-Aware Prompt Generator for Long-Tailed Vision Recognition

Liao,

Fang,

et al. 2024

Preprint

View full text Add to dashboard Cite

We present a prompt learning framework designed to enhance the performance in computer vision task considering a particular use case where the training image dataset is confronted with highly imbalanced categorical distributions. By formulating the prompt learning as a variational problem, our model is capable of generating multiple prompts to describe a semantic (i.e, class). The motivation behind generating multiple prompts originates from the heuristic that the voting ensemble establishes a more robust aggregated learning algorithm which potentially benefits learning the tail classes where the number of the training sample is scarce. Unlike previous prompt learning techniques, which are often restricted by using a fixed set of prompts during the training and the test phase, we propose to learn the prompt distribution from which an arbitrary number of prompts can be sampled whenever required, and we named our method as “Prompt Distribution Learning (PDL)”. We will discuss and contrast various ways to formulate the variation model and thoroughly compare their performances against state-of-the-art solutions for long-tailed visual recognition. Our empirical study suggests that the proposed prompt-learning framework is beneficial for transferring a pre-trained vision-language model to long-tailed downstream visual recognition tasks while being sufficiently flexible in accommodating different designs of prompt-generating functions. Our code is publicly available at https: //github.com/Walter-pixel/Prompt-Distribution-of-CLIP-Long-Tailed-Data.

show abstract

Semantic enhanced Top-k similarity search on weighted HIN

Cited by 6 publications

References 51 publications

Convolutional Bypasses Are Better Vision Transformer Adapters

Convolutional Bypasses Are Better Vision Transformer Adapters

Self-Attention Factor-Tuning for Parameter Efficient Fine-Tuning

Distribution-Aware Prompt Generator for Long-Tailed Vision Recognition

Contact Info

Product

Resources

About