Meta-analytic approaches and effect sizes to account for ‘nuisance heterogeneity’ in comparative physiology

Abstract

Meta-analysis is a powerful tool used to generate quantitatively informed answers to pressing global challenges. By distilling data from broad sets of research designs and study systems into standardised effect sizes, meta-analyses provide physiologists with opportunities to estimate overall effect sizes and understand the drivers of effect variability. Despite this ambition, research designs in the field of comparative physiology can appear, at the outset, as being vastly different to each other because of ‘nuisance heterogeneity’ (e.g. different temperatures or treatment dosages used across studies). Methodological differences across studies have led many to believe that meta-analysis is an exercise in comparing ‘apples with oranges’. Here, we dispel this myth by showing how standardised effect sizes can be used in conjunction with multilevel meta-regression models to both account for the factors driving differences across studies and make them more comparable. We assess the prevalence of nuisance heterogeneity in the comparative physiology literature – showing it is common and often not accounted for in analyses. We then formalise effect size measures (e.g. the temperature coefficient, Q10) that provide comparative physiologists with a means to remove nuisance heterogeneity without the need to resort to more complex statistical models that may be harder to interpret. We also describe more general approaches that can be applied to a variety of different contexts to derive new effect sizes and sampling variances, opening up new possibilities for quantitative synthesis. By using effect sizes that account for components of effect heterogeneity, in combination with existing meta-analytic models, comparative physiologists can explore exciting new questions while making results from large-scale data sets more accessible, comparable and widely interpretable.

Best practices for building and curating databases for comparative analyses

“…Most published papers today provide useable data, but it is still relatively common for data to be shared in an unusable format. For example, when surveying the meta-analyses in comparative physiology identified by Noble et al (2022;this issue), only 70% (37) of the 53 papers that provided data did so in a useable format and 30% ( 16) did so with an unusable format. Similarly, around 64% of data archives in ecology and evolution are not usable (Roche et al, 2015).…”

“…studies of mass-corrected traits that do not present mass) might be problematic. In this sense, tools like the R packages Rphylopars (Goolsby et al, 2017) and mice (van Buuren and Groothuis- Noble et al (2022;this issue)] that clearly reported sample sizes, the percentage of primary studies included (orange) and discarded (blue) after the first (title and abstract) and second (full-text) screenings. The black line represents the minimum and maximum of the percentage of studies included across all 29 meta-analyses.…”

“…For comparison, the number of meta-analyses in the field published in 13 non-specialist journals found via systematic search in Noble et al (2022; this issue) is also plotted (continuous pink line). (B) Based on 29 meta-analyses [out of 62 fromNoble et al (2022; this issue)] that clearly reported sample sizes, the percentage of primary studies included (orange) and discarded (blue) after the first (title and abstract) and second (full-text) screenings. The black line represents the minimum and maximum of the percentage of studies included across all 29 meta-analyses.…”

“…For more information on the systematic search of meta-analyses (e.g. search terms, inclusion criteria, PRISMA), seeNoble et al (2022; this issue). Data were obtained with permission from D.W.A.…”

Looking to the future: Building New Paradigms in Comparative Physiology and Biomechanics

“…The authors provide practical approaches to maximising the efficiency and accessibility of comparative databases, which in turn helps to promote and ensure data transparency and integrity. Daniel Noble and co-workers (Noble et al, 2022) also provide a very useful guide to dealing with 'nuisance heterogeneity' in comparative physiology meta-analysis, offering a way forward to compare data across studies even when, for example, temperatures or dosages are different.…”