Against the “one method fits all data sets” philosophy for comparison studies in methodological research

Strobl, Carolin; Leisch, Friedrich

doi:10.1002/bimj.202200104

Cited by 14 publications

(19 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Conditional answers imply that concepts such as global optimality are to be replaced by more shaded counterparts. If, for a given criterion, the nature of the optimal method further differs across data types, one ends up in a "one size does not fit al" type of scenario (Strobl & Leisch, 2022), which calls for a reporting in terms of proper conditional statements. A formally very similar scenario is well known in clinical trial research and occurs if there is no therapeutic alternative that outperforms all other alternatives for all (types of) patients; this further calls for a precision medicine approach.…”

Section: Possible Answers Resulting From Benchmarking Studiesmentioning

confidence: 99%

A white paper on good research practices in benchmarking: The case of cluster analysis

Mechelen

Boulesteix²,

Dangl

et al. 2023

WIREs Data Min & Knowl

Self Cite

View full text Add to dashboard Cite

To achieve scientific progress in terms of building a cumulative body of knowledge, careful attention to benchmarking is of the utmost importance, requiring that proposals of new methods are extensively and carefully compared with their best predecessors, and existing methods subjected to neutral comparison studies. Answers to benchmarking questions should be evidence‐based, with the relevant evidence being collected through well‐thought‐out procedures, in reproducible and replicable ways. In the present paper, we review good research practices in benchmarking from the perspective of the area of cluster analysis. Discussion is given to the theoretical, conceptual underpinnings of benchmarking based on simulated and empirical data in this context. Subsequently, the practicalities of how to address benchmarking questions in clustering are dealt with, and foundational recommendations are made based on existing literature.This article is categorized under: Fundamental Concepts of Data and Knowledge > Data Concepts Fundamental Concepts of Data and Knowledge > Key Design Issues in Data Mining Technologies > Structure Discovery and Clustering

show abstract

Section: Possible Answers Resulting From Benchmarking Studiesmentioning

confidence: 99%

A white paper on good research practices in benchmarking: The case of cluster analysis

Mechelen

Boulesteix²,

Dangl

et al. 2023

WIREs Data Min & Knowl

Self Cite

View full text Add to dashboard Cite

show abstract

“…The second data task we consider in our experiment is differential gene expression analysis, which aims at identifying genes that show differences in their expression levels between two or more conditions (Soneson & Delorenzi, 2013). Of the many methods that have been proposed for this task (Seyednasrollah et al, 2013), the more recent ones usually expect RNA-Seq data as input, which means that gene expression is measured as nonnegative counts (Rigaill et al, 2018).…”

Section: Data Analysis Task Ii: Differential Gene Expression Analysismentioning

confidence: 99%

Explaining the optimistic performance evaluation of newly proposed methods: A cross‐design validation experiment

et al. 2023

View full text Add to dashboard Cite

The constant development of new data analysis methods in many fields of research is accompanied by an increasing awareness that these new methods often perform better in their introductory paper than in subsequent comparison studies conducted by other researchers. We attempt to explain this discrepancy by conducting a systematic experiment that we call “cross‐design validation of methods”. In the experiment, we select two methods designed for the same data analysis task, reproduce the results shown in each paper, and then reevaluate each method based on the study design (i.e., datasets, competing methods, and evaluation criteria) that was used to show the abilities of the other method. We conduct the experiment for two data analysis tasks, namely cancer subtyping using multiomic data and differential gene expression analysis. Three of the four methods included in the experiment indeed perform worse when they are evaluated on the new study design, which is mainly caused by the different datasets. Apart from illustrating the many degrees of freedom existing in the assessment of a method and their effect on its performance, our experiment suggests that the performance discrepancies between original and subsequent papers may not only be caused by the nonneutrality of the authors proposing the new method but also by differences regarding the level of expertise and field of application. Authors of new methods should thus focus not only on a transparent and extensive evaluation but also on comprehensive method documentation that enables the correct use of their methods in subsequent studies.

show abstract

“…Therefore, we published a simulation study protocol, which includes details on the simulation setup, sample size, data-generating mechanism, and outcomes (https://osf.io/5wmg9). Our simulation study was designed, as suggested by Strobl et al [30], to have the complexity of real data and to reflect scenarios that are realistic but not idealistic [30]. This can be considered a strength of our study.…”

Section: Strengths and Limitationsmentioning

confidence: 99%

How to quantify between-study heterogeneity in single-group evidence synthesis? - It depends!

Iaquinto

Feldmann

Latal

et al. 2023

Preprint

View full text Add to dashboard Cite

Background: Random-effects meta-analysis models account for between-study heterogeneity by estimating and incorporating the heterogeneity variance parameter tau2. Numerous estimators for tau2 have been proposed, but no widely accepted guidance exists on when to best use which meta-analysis method. Especially in the context of observational studies, systematic evaluations and comparisons of the various meta-analysis methods are lacking. Since between-study heterogeneity is of crucial importance, particularly in meta-analysis of observational studies, where control groups typically do not exist, considerable attention should be paid to its estimation. This study aims to investigate the advantages of different meta-analysis methods for typical situations through comprehensive simulations in a neutral comparison study and with an empirical application. Methods: We compared seven methods for random-effects meta-analysis. The methods were selected with a focus on methodological diversity and availability and were evaluated both empirically and in a simulation study. We simulated typical meta-analysis scenarios for continuous and binary outcomes in a single-group meta-analysis setting. Results: Specific study characteristics, such as the number of studies included in a meta-analysis, the amount of heterogeneity in the data and binary outcome data with rare events, strongly affected the performance of the heterogeneity variance estimator. Moreover, we discovered that most heterogeneity variance estimators produce zero heterogeneity estimates under all simulated conditions, even though heterogeneity was present. The estimated overall effect was found to be relatively robust regarding different methods in the empirical application and in our simulation study. Conclusions: Although different meta-analysis methods produce substantially different heterogeneity variance estimates, too little attention is paid to selecting a suitable meta-analysis method in research applications. Based on our literature review, we conclude that the awareness about different heterogeneity variance estimators and their properties needs to be strengthened in practice. Our simulation study showed that all evaluated heterogeneity estimators were imprecise and often failed to estimate the true amount of heterogeneity. The estimation is particularly imprecise in situations where the meta-analysis contained few studies or when the binary outcomes included rare events. As it is rarely appropriate to rely on a single heterogeneity variance estimator, therefore we suggest careful consideration and evaluation of a wider range of plausible estimators in a sensitivity analysis before drawing a final conclusion about the meta-analysis results.

show abstract

Against the “one method fits all data sets” philosophy for comparison studies in methodological research

Cited by 14 publications

References 26 publications

A white paper on good research practices in benchmarking: The case of cluster analysis

A white paper on good research practices in benchmarking: The case of cluster analysis

Explaining the optimistic performance evaluation of newly proposed methods: A cross‐design validation experiment

How to quantify between-study heterogeneity in single-group evidence synthesis? - It depends!

Contact Info

Product

Resources

About