Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space

How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey

“…Model Similarity-based Methods DSE (Vu et al, 2020) ϕ(x), ψ(x) ✓ ✗ ✗ DDS (Dwivedi et al, 2020) ϕ(x), ψ(x) ✓ ✗ ✗ Training-free Methods MSC (Meiseles and Rokach, 2020) ϕ(x), y ✗ ✗ ✓ kNN (Puigcerver et al, 2021) ϕ(x), y ✗ ✗ ✓ PARC (Bolya et al, 2021) ϕ(x), y ✗ ✗ ✓ GBC ϕ(x), y ✗ ✗ ✓ Logistic (Kumari et al, 2022) ϕ(x), y ✗ ✓ ✓ H-score (Bao et al, 2019) ϕ(x), y ✗ ✗ ✓ Reg. H-score (Ibrahim et al, 2022) ϕ(x), y ✗ ✗ ✓ N LEEP ϕ(x), y ✗ ✗ ✓ TransRate (Huang et al, 2022) ϕ(x), y ✗ ✗ ✓ LogME (You et al, 2021) ϕ(x), y ✗ ✓ ✓ SFDA (Shao et al, 2022) ϕ(x), y ✗ ✓ ✓ PACTran (Ding et al, 2022) ϕ(x), y ✗ ✓ ✓ target classes by LR's test accuracy.…”

“…There are also some metrics that involve fine-tuning dynamics. SFDA (Shao et al, 2022) simulates the dynamics by projecting the pre-trained features using Fisher Discriminant Analysis (FDA) to increase the class separability. Then, it approximates the log-likelihood by Bayes classification over projected features and also adds a self-challenging module to further measure the ability of the pre-trained models on hard samples.…”