Ranking Neural Checkpoints

Li, Yandong; Jia, Xuhui; Sang, Ruoxin; Zhu, Yukun; Green, Bradley; Wang, Liqiang; Gong, Boqing

doi:10.1109/cvpr46437.2021.00269

Cited by 20 publications

(16 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sec. 3.2 briefly describes how we setup the data for semantic segmentation instead of the image classification task considered in previous works on transferability [4,42,53,62,63,73]. In Sec.…”

Section: Methodsmentioning

confidence: 99%

“…LEEP [53] measures how well a classifier built on top of source model predictions performs on the target task. N LEEP [42] trains a Gaussian Mixture Model (GMM) on top of source model features. Then it measures how well a classifier built on top of these GMM predictions performs on the target task.…”

Section: Related Workmentioning

confidence: 99%

“…To put our work in context: (1) Instead of selecting a single source model [4,42,53,62,73], we do ensemble selection. (2) Instead of image classification [4,42,53,62,63,73], we address semantic segmentation. (3) We consider a larger variety of source datasets than previous works (17 vs at most 4 [62]).…”

Section: Related Workmentioning

confidence: 99%

“…Recently, transferability metrics were introduced to address this problem [4,42,53,62,63,73]. The general goal is to select a single source model which, after fine-tuning on the target training set, yields the best performance on the target test set.…”

Section: Introductionmentioning

confidence: 99%

“…1). In contrast, previous works typically focus on image classification [4,42,53,62,63,73], considers a narrower range of at most 4 source datasets [42,53,62,73], and often generate multiple datasets artificially by sampling different subsets of classes out of a single actual dataset [4,53,62,63].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Transferability Metrics for Selecting Source Model Ensembles

Agostinelli¹,

Uijlings²,

Mensink³

et al. 2021

Preprint

View full text Add to dashboard Cite

We address the problem of ensemble selection in transfer learning: Given a large pool of source models we want to select an ensemble of models which, after fine-tuning on the target training set, yields the best performance on the target test set. Since fine-tuning all possible ensembles is computationally prohibitive, we aim at predicting performance on the target dataset using a computationally efficient transferability metric. We propose several new transferability metrics designed for this task and evaluate them in a challenging and realistic transfer learning setup for semantic segmentation: we create a large and diverse pool of source models by considering 17 source datasets covering a wide variety of image domain, two different architectures, and two pre-training schemes. Given this pool, we then automatically select a subset to form an ensemble performing well on a given target dataset. We compare the ensemble selected by our method to two baselines which select a single source model, either (1) from the same pool as our method; or (2) from a pool containing large source models, each with similar capacity as an ensemble. Averaged over 17 target datasets, we outperform these baselines by 6.0% and 2.5% relative mean IoU, respectively.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%