Often Trusted but Never (Properly) Tested: Evaluating Qualitative Comparative Analysis

Abstract

To date, hundreds of researchers have employed the method of Qualitative Comparative Analysis (QCA) for the purpose of causal inference. In a recent series of simulation studies, however, several authors have questioned the correctness of QCA in this connection. Some prominent representatives of the method have replied in turn that simulations with artificial data are unsuited for assessing QCA. We take issue with either position in this impasse. On the one hand, we argue that data-driven evaluations of the correctness of a procedure of causal inference require artificial data. On the other hand, we prove all previous attempts in this direction to have been defective. For the first time in the literature on configurational comparative methods, we lay out a set of formal criteria for an adequate evaluation of QCA before implementing a battery of inverse-search trials to test how this method performs in different recovery contexts according to these criteria. While our results indicate that QCA is correct when generating the parsimonious solution type, they also demonstrate that the method is incorrect when generating the conservative and intermediate solution type. In consequence, researchers using QCA for causal inference, particularly in human-sensitive areas such as public health and medicine, should immediately discontinue employing the method’s conservative and intermediate search strategies.

Electoral violence under different contexts. Evidence from the Western Balkans

“…In response to the recent methodological dispute on interpretation of solution terms, the paper roots its analysis in the work of Michael Baumgartner and Alrik Thiem and follows a recommendation to report parsimonious solution as the only type of the three QCA solution types that is not methodologically biased (Thiem 2016a). Since recent methodological work has demonstrated that both the conservative and intermediate solution types often infer (way) beyond the data and thus increase the risk of causal fallacies, it represents a safe approach for interpreting the results (Baumgartner and Thiem 2017b).…”

Strategy, operations, and profitability: the role of resource orchestration

“…Following Fiss (2011), +0.0001 is added to the calibrated variables in order to prevent the deletion of cases with membership scores of 0.5 in the outcome. Next, following the QCApro method, the parsimonious solution type is used to understand causal inference because the complex and intermediate solutions are demonstrably incorrect procedures of causal inference (Baumgartner and Thiem, 2017). For clarity, (1) These steps lead to performing the counterfactual analysis, which is possible due to the existence of limited diversity.…”

Declaring and Diagnosing Research Designs

“…Critics have highlighted that these algorithms are sensitive to measurement error (Hug 2013). Pointing to such sensitivity, some even go as far as to call for the rejection of QCA as a framework for inquiry (Lucas and Szatrowski 2014; for a nuanced response, see Baumgartner and Thiem 2017).…”

“…For both methods, we use the "parsimonious" solution and not the "conservative" or "intermediate" solutions that have been criticized inBaumgartner and Thiem (2017), though our declaration could easily be modified to check the performance of these alternative solutions.…”