Empirical Study on Optimizer Selection for Out-of-Distribution Generalization

Abstract

Modern deep learning systems are fragile and do not generalize well under distribution shifts. While much promising work has been accomplished to address these concerns, a systematic study of the role of optimizers and their out-of-distribution generalization performance has not been undertaken. In this study, we examine the performance of popular first-order optimizers for different classes of distributional shift under empirical risk minimization and invariant risk minimization. We address the problem settings for image and text classification using DomainBed, WILDS, and Backgrounds Challenge as out-of-distribution datasets for the exhaustive study. We search over a wide range of hyperparameters and examine the classification accuracy (in-distribution and out-of-distribution) for over 20,000 models. We arrive at the following findings: i) contrary to conventional wisdom, adaptive optimizers (e.g., Adam) perform worse than non-adaptive optimizers (e.g., SGD, momentum-based SGD), ii) in-distribution performance and out-of-distribution performance exhibit three types of behavior depending on the dataset -linear returns, increasing returns, and diminishing returns. We believe these findings can help practitioners choose the right optimizer and know what behavior to expect.

Exploring the Landscape: Analysis of Model Results on Various Convolutional Neural Network Architectures for iRESPOND System

“…Interestingly, the analysis also proves that the choice of optimization algorithm did not significantly impact the model's performance. Both Adam and RMSprop optimizers were used in training the architectures, but neither seemed to contribute significantly to the observed variations in accuracy or loss [52]. This finding implies that other factors, such as the architecture itself and the choice of learning rate, played a more substantial role in determining the model's performance.…”