Common datastream permutations of animal social network data are not appropriate for hypothesis testing using regression models

Weiss, Michael N.; Franks, Daniel W.; Brent, Lauren J. N.; Ellis, Stephen; Silk, Matthew J.; Croft, Darren P.

doi:10.1111/2041-210x.13508

Cited by 51 publications

(45 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our simulations confirm that the double permutation method is robust to type I errors across a range of scenarios. In simulation 1 (Table 1), pre-network permutation tests suffer from elevated false positives (type I error rate of 26%), consistent with previous studies (Evans, Fisher & Silk 2020;Puga-Gonzalez, Sueur & Sosa 2021;Weiss et al 2021). We further show that the tendency for prenetwork permutation tests to generate type I errors is greater in smaller networks and when more data are collected (Figure S2, see also Evans, Fisher & Silk 2020).…”

Section: The Double Permutation Approach Is Robust To Type I and Type Ii Errorssupporting

confidence: 88%

“…In simulation 3, we simulate the process of testing for a link between pairwise kinship and association rate in a species that exhibits other social preferences not based on kinship. For each simulated dataset, we calculate pvalues using several tests: node permutations, node permutations in which the model controls for covariates (number of observations and, where possible, location), pre-network permutations, prenetwork permutations on the t-statistic (the equivalent of scaling the predictions, as proposed by Weiss et al 2021), pre-network permutations in which the model controls for covariates (number of observations and, where possible, location), and the double permutation method. We create the networks and conduct the permutation tests using the R package asnipe (Farine 2013) and use the package sna (Butts 2008) to calculate network metrics.…”

Section: Testing the Robustness Of The Double Permutation Approachmentioning

confidence: 99%

“…They are particularly useful when the standard assumptions of other statistical tests are violated, as is often the case with social network data (see Farine 2017 for a general introduction). However, several recent studies (Evans, Fisher & Silk 2020;Puga-Gonzalez, Sueur & Sosa 2021;Weiss et al 2021) have highlighted issues with using social network data to test null hypotheses about the relationship between a predictor and a response (i.e. conducting regressions using network data).…”

Section: Introductionmentioning

confidence: 99%

“…As a result, the permuted networks do not have socially realistic network-level properties of real animal societies, such as the natural variance in edge weights or distribution of degree values. Therefore, when a social network has other sources of social structure beyond the effect of interest, pre-network permutation tests will produce highly elevated false positives when testing the significance of a correlation or regression (Puga-Gonzalez, Sueur & Sosa 2021;Weiss et al 2021).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Permutation tests for hypothesis testing with animal social network data: problems and potential solutions

Farine

Carter

2020

Preprint

View full text Add to dashboard Cite

Generating insights about a null hypothesis requires not only a good dataset, but also statistical tests that are reliable and actually address the null hypothesis of interest. Recent studies have found that permutation tests, which are widely used to test hypotheses when working with animal social network data, can suffer from high rates of type I error (false positives) and type II error (false negatives). Here, we first outline why pre-network and node permutation tests have elevated type I and II error rates. We then propose a new procedure, the double permutation test, that addresses some of the limitations of existing approaches by combining pre-network and node permutations. We conduct a range of simulations, allowing us to estimate error rates under different scenarios, including errors caused by confounding effects of social or non-social structure in the raw data. We show that double permutation tests avoid elevated type I errors, while remaining sufficiently sensitive to avoid elevated type II errors. By contrast, the existing solutions we tested, including node permutations, pre-network permutations, and regression models with control variables, all exhibit elevated errors under at least one set of simulated conditions. Type I error rates from double permutation remain close to 5% in the same scenarios where type I error rates from pre-network permutation tests exceed 30%. The double permutation test provides a potential solution to issues arising from elevated type I and type II error rates when testing hypotheses with social network data. We also discuss other approaches, including restricted node permutations, testing multiple null hypotheses, and splitting large datasets to generate replicated networks, that can strengthen our ability to make robust inferences. Finally, we highlight ways that uncertainty can be explicitly considered during the analysis using permutation-based or Bayesian methods.

show abstract

Section: The Double Permutation Approach Is Robust To Type I and Type Ii Errorssupporting

confidence: 88%

Section: Testing the Robustness Of The Double Permutation Approachmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Permutation tests for hypothesis testing with animal social network data: problems and potential solutions

Farine

Carter

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Nodal regression uses a node-level social network metric such as node strength, eigenvector centrality, or closeness to quantify an individual's position in their social structure, and relates this social position to quantifiable traits such as age or sex (Farine and Whitehead, 2015). The relationship between network metric and trait is usually analysed using regression (Weiss et al, 2021b). The statistical power of a conventional regression depends on sample size, effect size, and significance level.…”

Section: Introductionmentioning

confidence: 99%

Accuracy and Power Analysis of Social Networks Built From Count Data

Hart

Franks

Brent

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Power analysis is used to estimate the probability of correctly rejecting a null hypothesis for a given statistical model and dataset. Conventional power analyses assume complete information, but the stochastic nature of behavioural sampling can mean that true and estimated networks are poorly correlated. Power analyses of animal social networks do not currently take the effect of sampling into account. This could lead to inaccurate estimates of statistical power, potentially yielding misleading results. Here we develop a method for computing how well an estimated social network correlates with its true network using a Gamma-Poisson model of interaction rates. We use simulations to assess how the level of correlation between true and estimated networks affects the power of nodal regression analyses. We also develop a generic method of power analysis applicable to any statistical test, based on the concept of diminishing returns. We demonstrate that our network correlation estimator is both accurate and moderately robust to its assumptions being broken. We show that social differentiation, mean interaction rate, and the harmonic mean of sampling times positively impacts the strength of correlation between true and estimated networks. We also show that the required level of correlation between true and estimated networks to achieve a given power level depends on many factors, but that 80% correlation usually corresponded to around 80% power for nodal regression. We provide guidelines for using our network correlation estimator to verify the accuracy of interaction networks, and to conduct power analysis. This can be used prior to data collection, in post hoc analyses, or even for subsetting networks for use in dynamic network analysis. We make our code available so that custom power analysis can be used in future studies.

show abstract

Social network analysis of white‐tailed deer scraping behavior: Implications for disease transmission

et al. 2023

View full text Add to dashboard Cite

Host contact structure affects pathogen transmission in host populations, but many measures of host contact do not distinguish contacts that are relevant to pathogen transmission from those that are not. Scrapes are sites for chemical communication by white‐tailed deer (Odocoileus virginianus) during the breeding season and potential sites of transmission of prions, the causative agent of chronic wasting disease (CWD). Scrape‐related behaviors vary in their probability of transmitting prions to or from the environment, suggesting that behavior be combined with contact structure to better reflect potential heterogeneity in prion transmission at scrapes. We recorded visits and behaviors by deer at scrapes throughout DeSoto National Wildlife Refuge, Nebraska in 2005 and 2006. We recorded 2013 interactions by 169 unique identifiable males and 75 females. Adult males performed the most scrape‐related behaviors and spent the most time at scrapes, especially smelling the overhanging branch (70%), smelling the scrape (59%), licking/grasping the overhanging branch (44%), and scraping (36%). We used social network analysis to test the effect of behavior on indirect contact networks among deer at scrapes. By weighting edges based on the frequency and duration of behaviors, we produced networks representing sources of variation in scrape use and compared these networks to evaluate the effects of behavior on network contact structure. Social networks based on scrape‐related behavior were highly connected and dependent upon the frequency, duration, and type of behavior exhibited at scrapes (e.g., scraping, interacting with a scrape or overhanging branch, rub‐urinating, grazing) as well as the age of the deer. Accounting for contact frequency produced networks with lower variation in contact, but higher ability to facilitate contact among disparate groups. Including behavior when defining edges did not preserve the network properties of simpler measures (i.e., unweighted networks) suggesting that heterogeneity in behaviors that affect transmission probability is important for inferring transmission networks from contact networks. High connectivity through indirect contacts suggests that scrapes may be effective targets for management. Adult male deer had the highest connectivity, suggesting that management strategies focused on reducing their interaction with scrapes through density reduction or behavioral modification could reduce the connectivity of indirect contact networks.

show abstract

Common datastream permutations of animal social network data are not appropriate for hypothesis testing using regression models

Abstract: This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

Cited by 51 publications

References 32 publications

Permutation tests for hypothesis testing with animal social network data: problems and potential solutions

Permutation tests for hypothesis testing with animal social network data: problems and potential solutions

Accuracy and Power Analysis of Social Networks Built From Count Data

Social network analysis of white‐tailed deer scraping behavior: Implications for disease transmission

Contact Info

Product

Resources

About