Guidelines for Improving the Power Values of Statistical Tests for Nonresponse Bias Assessment in OM Research

Abstract

The assessment of nonresponse bias in survey-based empirical studies plays an important role in establishing the credibility of research results. Statistical methods that involve the comparison of responses from two groups (e.g., early vs. late respondents) on multiple characteristics, which are relevant to the study, are frequently utilized in the assessment of nonresponse bias. We consider the concepts of individual and complete statistical power used for multiple testing and show their relevance for determining the number of statistical tests to perform when assessing nonresponse bias. Our analysis of factors that influence both individual and complete power levels, yielded recommendations that can be used by operations management (OM) empirical researchers to improve their assessment of nonresponse bias. A power analysis of 61 survey-based research papers published in three prestigious academic operations management journals, over the last decade, showed the occurrence of very low (<0.4) power levels in some of the statistical tests used for assessing nonresponse bias. Such low power levels can lead to erroneous conclusions about nonresponse bias, and are indicators of the need for more rigor in the assessment of nonresponse bias in OM research.

Concurrently Assessing Survey Mode and Sample Size in Off‐Site Angler Surveys

“…Nonresponse bias assessments involve the use of additional survey effort to contact sample members who have a low propensity to respond (Peress 2010) and statistical tests to determine whether there is a significant difference between respondents and nonrespondents with regard to the variables of interest (Clottey and Benton 2013). Nonresponse bias in angler surveys occurs when respondents are more avid anglers who have more successful fishing trips or expend more effort on the fishery than nonrespondents (Connelly et al 2000;Zarauz et al 2015).…”

Technical Note: Recommendations for Assessing Unit Nonresponse Bias in Dyadic Focused Empirical Supply Chain Management Research

“…It is easy to see that there is no clear agreement on the appropriate statistical method to be used in the assessment of potential unit nonresponse bias with dyadic data. Furthermore, in Table 1, the prevalent use of four or more tests in the assessment makes it likely that one or more of the tests results in an erroneous conclusion about potential nonresponse bias (Clottey & Benton, 2013). We believe that consideration of effect sizes (ESs), statistical power, and sample size requirements may help in providing a predominant approach to assessing for unit nonresponse bias in the presence of dyadic data.…”

“…For decades, in SCM research and other allied areas, the approach to assessing for potential unit nonresponse bias has been statistical methods that involve the comparison of responses from in‐sample versus out‐of‐sample respondents, on multiple characteristics, which are relevant to the study (Lambert & Harrington, 1990; Collier & Bienstock, 2007; Werner, Praxedes, & Kim, 2007; Wagner & Kemmerling, 2010; Clottey & Benton, 2013). If the responses of the two groups differ by very much, then it is good evidence that unit nonresponse cannot be ignored, and therefore, adjustments to the data may be required to accommodate for potential unit nonresponse bias (Whitehead, Groothuis, & Blomquist, 1993; Wagner & Kemmerling, 2010).…”

“…Usually, multiple statistical tests (e.g., t ‐tests or ANOVA) are performed, and unit nonresponse bias is considered not an issue if all the tests result in nonsignificance. For nondyadic data, Clottey and Benton (2013) noted that having multiple tests leads to the classical problem in statistics in which the more inferences are made, the more likely that erroneous inference has occurred. An argument can be made that dyadic data can further complicate nonresponse bias assessment due to the introduction of additional correlations (e.g., between the responses of members from the same dyad) in the analysis, along with an increase in the number of statistical tests performed (e.g., when separate tests are utilized for each member of the dyad).…”

“…An argument can be made that dyadic data can further complicate nonresponse bias assessment due to the introduction of additional correlations (e.g., between the responses of members from the same dyad) in the analysis, along with an increase in the number of statistical tests performed (e.g., when separate tests are utilized for each member of the dyad). Clottey and Benton (2013) recommend that researchers adopt methods, to assess for potential nonresponse bias, which minimizes the number of separate statistical tests needed to be performed. Our research expands on this suggestion by considering the use of a Multivariate Analysis of Variance (MANOVA) approach that allows for a single statistical test to be utilized for the assessment with dyadic data.…”