Forward Compatible Training for Large-Scale Embedding Retrieval Systems

Ramanujan, Vivek; Vasu, Pavan Kumar Anasosalu; Farhadi, Ali; Tuzel, Oncel; Pouransari, Hadi

doi:10.1109/cvpr52688.2022.01878

Cited by 9 publications

(6 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…( 1) for the new model, as they suffer from the inherent trade-off between maximizing the new model performance and at the same time its compatibility to the weaker old model. Other works have thus proposed learning a separate mapping from old model embedding space to new model embedding space to align their features (Wang et al, 2020;Ramanujan et al, 2022). Wang et al (2020) and both still modify the training of the new model which is undesirable because it can limit the performance gain of the new model and is intrusive and complicated for practitioners.…”

Section: Methodsmentioning

confidence: 99%

“…This has led rise to the study of compatible representation learning. Shen et al (2020); Budnik & Avrithis (2021); ; Ramanujan et al (2022); Hu et al (2022); Zhao et al (2022); Duggal et al (2021) all proposed methods to update the embedding model to achieve better performance whilst still being compatible with features generated by the old model (see Figure 1-left-top). Despite relative success, compatibility learning is not perfect: performing retrieval with a mixture of old and new features achieves lower accuracies than when we replace all the old features with new ones.…”

Section: Introductionmentioning

confidence: 99%

“…In this work, we focus on closing this performance gap. Further, some previous methods degrade the new model performance when trying to make it more compatible with the old model (Shen et al, 2020;Hu et al, 2022) or requiring availability of side-information, extra features from a separate self-supervised model, (Ramanujan et al, 2022) (which may not be available for an existing system). We relax both constraints in this work.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

FastFill: Efficient Compatible Model Update

Jaeckle¹,

Faghri²,

Farhadi³

et al. 2023

Preprint

View full text Add to dashboard Cite

In many retrieval systems the original high dimensional data (e.g., images) is mapped to a lower dimensional feature through a learned embedding model. The task of retrieving the most similar data from a gallery set to a given query data is performed through a similarity comparison on features. When the embedding model is updated, it might produce features that are not comparable/compatible with features already in the gallery computed with the old model. Subsequently, all features in the gallery need to be re-computed using the new embedding model -a computationally expensive process called backfilling. Recently, compatible representation learning methods have been proposed to avoid backfilling. Despite their relative success, there is an inherent trade-off between the new model performance and its compatibility with the old model. In this work, we introduce FastFill: a compatible model update process using feature alignment and policy based partial backfilling to promptly elevate retrieval performance. We show that previous backfilling strategies suffer from decreased performance and demonstrate the importance of both the training objective and the ordering in online partial backfilling. We propose a new training method for feature alignment between old and new embedding models using uncertainty estimation. Compared to previous works, we obtain significantly improved backfilling results on a variety of datasets: mAP on ImageNet (+4.4%), Places-365 (+2.7%), and VGG-Face2 (+1.3%). Further, we demonstrate that when updating a biased model with FastFill, the minority subgroup accuracy gap promptly vanishes with a small fraction of partial backfilling. 1

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

FastFill: Efficient Compatible Model Update

Jaeckle¹,

Faghri²,

Farhadi³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Cross-model Compatible Transformation. Compatible feature transformation (Chen et al 2019;Wang et al 2020;Ramanujan et al 2022) also targets the compatible retrieval among different embedding models. However, it concerns training transformation modules to map the features from different models to a common space.…”

Section: Related Workmentioning

confidence: 99%

Darwinian Model Upgrades: Model Evolving with Selective Compatibility

Zhang

et al. 2023

AAAI

View full text Add to dashboard Cite

The traditional model upgrading paradigm for retrieval requires recomputing all gallery embeddings before deploying the new model (dubbed as "backfilling"), which is quite expensive and time-consuming considering billions of instances in industrial applications. BCT presents the first step towards backward-compatible model upgrades to get rid of backfilling. It is workable but leaves the new model in a dilemma between new feature discriminativeness and new-to-old compatibility due to the undifferentiated compatibility constraints. In this work, we propose Darwinian Model Upgrades (DMU), which disentangle the inheritance and variation in the model evolving with selective backward compatibility and forward adaptation, respectively. The old-to-new heritable knowledge is measured by old feature discriminativeness, and the gallery features, especially those of poor quality, are evolved in a lightweight manner to become more adaptive in the new latent space. We demonstrate the superiority of DMU through comprehensive experiments on large-scale landmark retrieval and face recognition benchmarks. DMU effectively alleviates the new-to-new degradation at the same time improving new-to-old compatibility, rendering a more proper model upgrading paradigm in large-scale retrieval systems.Code: https://github.com/TencentARC/OpenCompatible.

show abstract

“…Without the task-identity, classifiers for different tasks fail to choose the corresponding features because these classifiers are not trained together. FACT [22] keeps an extra-embedding to capture irrelevant information from the old task yet may be useful for future tasks. It utilizes a linear layer to trans-form the extra-embedding and old feature into new feature.…”

Section: Introductionmentioning

confidence: 99%

Multi-level Prompt as Extra-Embedding for Incremental Learning

chaoyang,

zhouyang,

Cheng

et al. 2024

Preprint

View full text Add to dashboard Cite

Incremental learning has placed an emphasis on solving the issue of catastrophic forgetting. However, the intertask confusion is gradually becoming a non-negligible factor affecting incremental learning. It is observed that a model's learning performance deteriorates significantly when the task identity is unavailable. The primary reason for this is that the model’s features are not closely related to the task, and the ratios of classifier outputs for different tasks become erratic. In this paper, we propose a novel approach, Multi-level Prompt as Extra-Embedding(MPaEE) address the issue of inter-task confusion. MPaEE incorporates several task-independent prompt pools as additional embeddings for image features, enabling classifiers to distinguish inputs for various tasks. Exploiting the fact that the features of the pre-trained model are highly aggregated at the class-level, we assign these prompts to different categories. This approach alleviates the difficulty of selecting prompts for a particular task, as observed in previous work. Our experiments on four benchmark datasets have demonstrated the significant advantages of MPaEE and its effectiveness over the state-of-arts. The implementation of MPaEE can be found in Github: https://github.com/yanaCode0/MPaEE

show abstract

Forward Compatible Training for Large-Scale Embedding Retrieval Systems

Cited by 9 publications

References 15 publications

FastFill: Efficient Compatible Model Update

FastFill: Efficient Compatible Model Update

Darwinian Model Upgrades: Model Evolving with Selective Compatibility

Multi-level Prompt as Extra-Embedding for Incremental Learning

Contact Info

Product

Resources

About