No free theory choice from machine learning

Abstract

Ravit Dotan argues that a No Free Lunch theorem (NFL) from machine learning shows epistemic values are insufficient for deciding the truth of scientific hypotheses. She argues that NFL shows that the best case accuracy of scientific hypotheses is no more than chance. Since accuracy underpins every epistemic value, non-epistemic values are needed to assess the truth of scientific hypotheses. However, NFL cannot be coherently applied to the problem of theory choice. The NFL theorem Dotan’s argument relies upon is a member of a family of theorems in search, optimization, and machine learning. They all claim to show that if no assumptions are made about a search or optimization problem or learning situation, then the best case performance of an algorithm is that of random search or random guessing. A closer inspection shows that these theorems all rely upon assigning uniform probabilities over problems or learning situations, which is just the Principle of Indifference. A counterexample can be crafted that shows that NFL cannot be coherently applied across different descriptions of the same learning situation. To avoid this counterexample, Dotan needs to privilege some description of the learning situation faced by scientists. However, this means that NFL cannot be applied since an important assumption about the problem is being made. So Dotan faces a dilemma: either NFL leads to incoherent best-case partial beliefs or it is inapplicable to the problem of theory choice. This negative result has implications for the larger debate over theory choice.

Reliability in Machine Learning

“…Rather than providing theoretical guarantees, research in robustness proceeds experimentally, employing a variety of evaluation techniques and mitigation strategies to detect and preempt possible performance failures. Freiesleben and Grote (2023) develop a conceptual framework for robustness in machine learning, synthesizing different strands in a fragmented research landscape. They define robustness as a multi‐place concept, consisting of a robustness target (the machine learning model) and a robustness modifier (e.g., the deployment distribution).…”