Out of Model Zoo: Towards Selecting the Best Deep Neural Network with Cognitive Analysis

Abstract

Deep neural networks have achieved impressive performance on a huge amount of tasks and many model zoos are available that provide pre-trained models for free download. Despite the convenience, it is still difficult for the resource-constrained companies, especially startups, to select the best model from many candidates for their own tasks. Due to the lack of training data, the main challenges are to (1) evaluate the robustness of models with a test dataset and (2) optimize the prediction to maximize the usage of the model zoo. To address these challenges, we propose a robust model selector, Miss DL, which incorporates cognitive analysis and a series of criteria to select robust models from a given model zoo. We first propose a systemic method to simulate the distortions of new data points and calculate the dominant labels of the new data points. Then, we propose a metric to measure the robustness of models and select the best model. Motivated by the low confidence scores of the selected model on certain data points, we select a complementary model for each class for optimization. We evaluate the proposed Miss DL on popular deep neural networks and image datasets, including CIFAR-10, CIFAR-100, and ImageNet. Extensive experiments are performed to show the robustness and efficiency of Miss DL in different settings.