Multi-objective Asynchronous Successive Halving

Abstract

Hyperparameter optimization (HPO) is increasingly used to automatically tune the predictive performance (e.g., accuracy) of machine learning models. However, in a plethora of real-world applications, accuracy is only one of the multiple -often conflicting -performance criteria, necessitating the adoption of a multi-objective (MO) perspective. While the literature on MO optimization is rich, few prior studies have focused on HPO. In this paper, we propose algorithms that extend asynchronous successive halving (ASHA) to the MO setting. Considering multiple evaluation metrics, we assess the performance of these methods on three real world tasks: (i) Neural architecture search, (ii) algorithmic fairness and (iii) language model optimization. Our empirical analysis shows that MO ASHA enables to perform MO HPO at scale. Further, we observe that that taking the entire Pareto front into account for candidate selection consistently outperforms multi-fidelity HPO based on MO scalarization in terms of wall-clock time. Our algorithms (to be open-sourced) establish new baselines for future research in the area.

Towards Green Automated Machine Learning: Status Quo and Future Directions

“…Similarly, [75] suggests to iteratively enlarge a neural network through a splitting procedure which explicitly considers the increase in energy cost by splitting a certain neuron such that resulting networks are more energy-efficient than competitors. In this regard, it is also conceivable to obtain more energy-efficient pipelines as a result of the search process through the use of multi-objective AutoML methods such as [12,51,57], initialized with both performance and energy-efficiency as target measures.…”