Rethinking the Handling of Method Failure in Comparison Studies

Abstract

Comparison studies in methodological research are intended to compare methods in an evidence‐based manner to help data analysts select a suitable method for their application. To provide trustworthy evidence, they must be carefully designed, implemented, and reported, especially given the many decisions made in planning and running. A common challenge in comparison studies is to handle the “failure” of one or more methods to produce a result for some (real or simulated) data sets, such that their performances cannot be measured in those instances. Despite an increasing emphasis on this topic in recent literature (focusing on non‐convergence as a common manifestation), there is little guidance on proper handling and interpretation, and reporting of the chosen approach is often neglected. This paper aims to fill this gap and offers practical guidance on handling method failure in comparison studies. After exploring common handlings across various published comparison studies from classical statistics and predictive modeling, we show that the popular approaches of discarding data sets yielding failure (either for all or the failing methods only) and imputing are inappropriate in most cases. We then recommend a different perspective on method failure—viewing it as the result of a complex interplay of several factors rather than just its manifestation. Building on this, we provide recommendations on more adequate handling of method failure derived from realistic considerations. In particular, we propose considering fallback strategies that directly reflect the behavior of real‐world users. Finally, we illustrate our recommendations and the dangers of inadequate handling of method failure through two exemplary comparison studies.

A large-scale neutral comparison study of survival models on low-dimensional data

“…), the prediction of the KM estimator was used as a fallback result. This ensures a statistically sound evaluation, and is considered a reasonable compromise between either overpenalizing models by inserting some constant value which would depend on the given metric and dataset, or simply disregarding failed iterations during evaluation, which is overly optimistic ( Fischer et al 2024 , Wünsch et al 2025 ).…”

A large-scale neutral comparison study of survival models on low-dimensional data

“…), the prediction of the KM estimator was used as a fallback result. This ensures a statistically sound evaluation, and is considered a reasonable compromise between either overpenalizing models by inserting some constant value which would depend on the given metric and dataset, or simply disregarding failed iterations during evaluation, which is overly optimistic ( Fischer et al 2024 , Wünsch et al 2025 ).…”

Empirical Comparison of Win Ratio and Joint Frailty Models for Recurrent Event Endpoints With Applications in Oncology and Cardiology

“…All analyses were performed in (version 4.5.2) on a Dell Pro Max 16. If any issues arise during analyses, they will be reported following Wunsch et al's error handling workflow [ 26 ]. We fitted gamma‐joint frailty models using the function from the package [ 10 ] on version 3.8.0, under a parametric specification with a Weibull baseline hazard.…”

Living Synthetic Benchmarks: A Neutral and Cumulative Framework for Simulation Studies

“…This can lead to a suboptimal implementation of competitor methods, again favoring the method of choice. Due to computational issues or non-convergence of a method, simulation repetitions may not always lead to valid outputs (Pawel et al, 2025;Wünsch et al, 2025). Different ways of handling such issues can change the main results and implications of a simulation study.…”

“…For example, if a method does not converge in very challenging simulated datasets (e.g., with small sample sizes), excluding the missing repetitions from that method may bias performance in its favor. It is therefore helpful to also report convergence of methods and provide results for different missingness handling strategies (Pawel et al, 2025;Wünsch et al, 2025). In many cases, repetition-wise deletion (i.e., removal of all repetitions with at least one missing method)…”

A large-scale neutral comparison study of survival models on low-dimensional data

“…), the prediction of the KM estimator was used as a fallback result. This ensures a statistically sound evaluation, and is considered a reasonable compromise between either overpenalizing models by inserting some constant value which would depend on the given metric and dataset, or simply disregarding failed iterations during evaluation, which is overly optimistic ( Fischer et al 2024 , Wünsch et al 2025 ).…”

Empirical Comparison of Win Ratio and Joint Frailty Models for Recurrent Event Endpoints With Applications in Oncology and Cardiology

“…All analyses were performed in (version 4.5.2) on a Dell Pro Max 16. If any issues arise during analyses, they will be reported following Wunsch et al's error handling workflow [ 26 ]. We fitted gamma‐joint frailty models using the function from the package [ 10 ] on version 3.8.0, under a parametric specification with a Weibull baseline hazard.…”

Living Synthetic Benchmarks: A Neutral and Cumulative Framework for Simulation Studies

“…This can lead to a suboptimal implementation of competitor methods, again favoring the method of choice. Due to computational issues or non-convergence of a method, simulation repetitions may not always lead to valid outputs (Pawel et al, 2025;Wünsch et al, 2025). Different ways of handling such issues can change the main results and implications of a simulation study.…”

“…For example, if a method does not converge in very challenging simulated datasets (e.g., with small sample sizes), excluding the missing repetitions from that method may bias performance in its favor. It is therefore helpful to also report convergence of methods and provide results for different missingness handling strategies (Pawel et al, 2025;Wünsch et al, 2025). In many cases, repetition-wise deletion (i.e., removal of all repetitions with at least one missing method)…”

A large-scale neutral comparison study of survival models on low-dimensional data

“…), the prediction of the KM estimator was used as a fallback result. This ensures a statistically sound evaluation, and is considered a reasonable compromise between either overpenalizing models by inserting some constant value which would depend on the given metric and dataset, or simply disregarding failed iterations during evaluation, which is overly optimistic ( Fischer et al 2024 , Wünsch et al 2025 ).…”

Empirical Comparison of Win Ratio and Joint Frailty Models for Recurrent Event Endpoints With Applications in Oncology and Cardiology

“…All analyses were performed in (version 4.5.2) on a Dell Pro Max 16. If any issues arise during analyses, they will be reported following Wunsch et al's error handling workflow [ 26 ]. We fitted gamma‐joint frailty models using the function from the package [ 10 ] on version 3.8.0, under a parametric specification with a Weibull baseline hazard.…”

Living Synthetic Benchmarks: A Neutral and Cumulative Framework for Simulation Studies

“…This can lead to a suboptimal implementation of competitor methods, again favoring the method of choice. Due to computational issues or non-convergence of a method, simulation repetitions may not always lead to valid outputs (Pawel et al, 2025;Wünsch et al, 2025). Different ways of handling such issues can change the main results and implications of a simulation study.…”

“…For example, if a method does not converge in very challenging simulated datasets (e.g., with small sample sizes), excluding the missing repetitions from that method may bias performance in its favor. It is therefore helpful to also report convergence of methods and provide results for different missingness handling strategies (Pawel et al, 2025;Wünsch et al, 2025). In many cases, repetition-wise deletion (i.e., removal of all repetitions with at least one missing method)…”