Debugging Machine Learning Pipelines

Abstract

Machine learning tasks entail the use of complex computational pipelines to reach quantitative and qualitative conclusions. If some of the activities in a pipeline produce erroneous or uninformative outputs, the pipeline may fail or produce incorrect results. Inferring the root cause of failures and unexpected behavior is challenging, usually requiring much human thought, and is both time consuming and error prone. We propose a new approach that makes use of iteration and provenance to automatically infer the root causes and derive succinct explanations of failures. Through a detailed experimental evaluation, we assess the cost, precision, and recall of our approach compared to the state of the art. Our source code and experimental data will be available for reproducibility and enhancement.

BugDoc: A System for Debugging Computational Pipelines

“…In what follows, we give a brief overview of our debugging methodology. For a more detailed discussion, see [14,15]. splits it into training and test subsets, creates and executes an estimator, and computes the F-measure score using 10-fold cross-validation.…”

“…In previous work [14], we proposed and implemented new methods to debug machine learning pipelines that automatically and iteratively identify one or more minimal causes of failures, thereby avoiding the tedious and error-prone task of manually tuning and executing new pipeline instances to test and derive new hypotheses for the failures. We have extended this initial work and built BugDoc [15], a system that identifies root causes for errors in general computational pipelines (or workflows).…”

BugDoc: Algorithms to Debug Computational Processes

“…The first, called Shortcut, discovers definitive root causes (which we sometimes abbreviate to, simply, bugs) consisting of a single conjunction of parameter-value (formally, parameter-equality-value) pairs. The second, called Debugging Decision Trees and introduced in [36], discovers more complex definitive root causes involving inequalities (e.g., A takes a value between 5 and 13).…”

“…While the Shortcut and Stacked Shortcut algorithms can find a single minimal definitive root cause very efficiently, usually without truncation (as we will see in the experimental section), characterizing all minimal definitive root causes is challenging. For this purpose, we use an algorithm that is exponential (in the number of parameters) in the worst case, but can characterize inequalities as well as equalities and does well heuristically even with a small budget [36].…”

Provenance Supporting Hyperparameter Analysis in Deep Neural Networks

“…Basically, there are two approaches to make ML workflows provenance aware. The first is provenance provided for a specific ML platform [46,35,24,40,2,20,36,30] and the second is the provenance systems that are independent of the domain [32]. In the first approach, each ML platform provides provenance using its proprietary representation, which is difficult to interpret and compare with execution between different platforms.…”