Why Do Machine Learning Practitioners Still Use Manual Tuning? A Qualitative Study

Abstract

Current advanced hyperparameter optimization (HPO) methods, such as Bayesian optimization, have high sampling efficiency and facilitate replicability. Nonetheless, machine learning (ML) practitioners (e.g., engineers, scientists) mostly apply less advanced HPO methods, which can increase resource consumption during HPO or lead to underoptimized ML models. Therefore, we suspect that practitioners choose their HPO method to achieve different goals, such as decrease practitioner effort and target audience compliance. To develop HPO methods that align with such goals, the reasons why practitioners decide for specific HPO methods must be unveiled and thoroughly understood. Because qualitative research is most suitable to uncover such reasons and find potential explanations for them, we conducted semi-structured interviews to explain why practitioners choose different HPO methods. The interviews revealed six principal practitioner goals (e.g., increasing model comprehension), and eleven key factors that impact decisions for HPO methods (e.g., available computing resources). We deepen the understanding about why practitioners decide for different HPO methods and outline recommendations for improvements of HPO methods by aligning them with practitioner goals.

Nucleic Acid Quantification by Multi-Frequency Impedance Cytometry and Machine Learning

“…All the aforementioned techniques are dedicated to special cases; as an example, grid search is only reliable for low-dimensional input spaces [ 47 ]. On the other hand, it was shown that random search results in better sampling efficiency in high-dimensional search spaces compared to grid search [ 49 ]. Bayesian optimization might potentially trap the model at a local optimum.…”