A Statistical Significance Test for Necessary Condition Analysis

Abstract

In this article, we present a statistical significance test for necessary conditions. This is an elaboration of necessary condition analysis (NCA), which is a data analysis approach that estimates the necessity effect size of a condition X for an outcome Y. NCA puts a ceiling on the data, representing the level of X that is necessary (but not sufficient) for a given level of Y. The empty space above the ceiling relative to the total empirical space characterizes the necessity effect size. We propose a statistical significance test that evaluates the evidence against the null hypothesis of an effect being due to chance. Such a randomness test helps protect researchers from making Type 1 errors and drawing false positive conclusions. The test is an "approximate permutation test." The test is available in NCA software for R. We provide suggestions for further statistical development of NCA.Keywords null hypothesis testing, permutation test, necessary condition analysis, statistical significance, p value Necessary condition analysis (NCA; Dul, 2016) is a tool for researchers to develop and test necessary but not sufficient conditions. A necessary condition enables the outcome when present and constrains the outcome when absent. NCA assumes that outcome Y is bound by condition X by drawing a ceiling line on top of the data in an XY scatter plot. The line defines the empty space in the upper left corner of the scatter plot.1 This empty space suggests that high values of Y are not possible with low values of X and indicates that X constrains Y. The size of the empty space relative to the total space with observations reflects the extent of the constraint that X poses on Y: The larger the empty space, the more X constrains Y. The necessity effect size (d) is the size of the empty space above the ceiling as a fraction of the total space where cases are observed or could be observed given by the minimum and maximum empirical or theoretical values of X and Y (scope 2 ). NCA's effect size d has values between 0 and 1.

Are individual-level factors necessary for creativity? A necessary condition analysis

“…Based on Dul's recommendations (2019), 0 < d ≤ 0.1 is considered a small effect, 0.1 ≤ d ≤ 0.3 a medium effect, 0.3 ≤ d ≤ 0.5 a large effect, and d ≥ 0.5 is considered a very large effect. A significance test is then performed, by comparing the effect size of the observed sample to the effect sizes of samples where X and Y are unrelated, using an approximate permutation test (for detailed explanations, see Dul et al, 2020).…”

Necessary Condition Analysis: Type I Error, Power, and Over-Interpretation of Test Results. A Reply to a Comment on NCA. Commentary: Predicting the Significance of Necessity

“…We reply to Sorjonen and Melin (2019) article “Predicting the significance of necessity” that is a comment on a recently proposed statistical test for Necessary Condition Analysis (Dul et al, in press). Necessary Condition Analysis (NCA) is a method that draws a ceiling line on top of the data in an XY scatter plot (Dul, 2016).…”

“…It is a permutation test 2 that produces an estimate of the p -value and “…is intended to answer the question: ‘Can the observed effect size be the result of random chance?' by responding: ‘Yes, but with probability smaller than p .”' (Dul et al, in press, p. 2). Dul et al (in press) show by simulations and by referring to a mathematical proof that the test is valid for identifying randomness, hence for helping researchers to avoid type I error (rejecting the null hypothesis when the null is true).…”

“…The statistical test for NCA is a valid and powerful “minimum statistical test” (Dul et al, in press, p. 8) that can test the randomness of an empty space in the upper left corner of a XY scatter plot: not more, not less. It may seem disappointing that a null hypothesis test like the one for NCA can only test whether a result is due to randomness or not, and cannot test for a specific alternative hypothesis.…”

Exploring necessary conditions in HRM research: Fundamental issues and methodological implications

“…Thus, we first elaborate the particularities of sufficiency and necessity logics. Then, we suggest using a new approach and analysis technique, that is, necessary condition analysis (NCA; Dul, 2016; Dul, van der Laan, & Kuik, 2018), which is able to identify necessary conditions in empirical data. We will demonstrate the differences between the sufficiency and necessity logics with an illustrative example based on data on the relationship between high‐performance work practices (HPWPs; Appelbaum, Bailey, Berg, & Kalleberg, 2000) and employee performance.…”