A Robustness Test Protocol for Applied QCA: Theory and R Software Application

Abstract

The robustness of qualitative comparative analysis (QCA) results features high on the agenda of methodologists and practitioners. This article aims at advancing this debate on several fronts. First, in line with the extant literature, we take a comprehensive view on robustness arguing that decisions on calibration, consistency, and frequency thresholds should all be tested. Second, we introduce the notion of “sensitivity range” as the range of values for any of these parameters within which the solution formula remains unchanged. Third, we argue that interpreting robustness is more intricate than simply checking if solutions remain unchanged. Beyond sensitivity ranges, researchers should assess robustness by evaluating changes in parameters of fit and the classification of cases as robust, shaky, or possible. Fourth, we enable researchers to perform more than one robustness test at a time by proposing the notions of a “test set”: the overlap between conceptually plausible alternative solutions that can be generated; and of a “robust core”: that part of a QCA solution that withstands the robustness checks. Fifth, we present functionalities implemented in the R package SetMethods that enable researchers to put in practice our proposals. These advancements are integrated into a comprehensive QCA Robustness Test Protocol consisting of three main tests: sensitivity ranges, fit-oriented robustness, and case-oriented robustness. We illustrate the protocol’s implementation with an example on high life expectancy across the globe.

Identifying diverse paths toward successful policy performance in Organization for Economic Co‐operation and Development (OECD) and European Union countries: A Qualitative Comparative Analysis (QCA) exploitation of the Sustainable Governance Indicators (SGI) data

“…Performing a QCA analysis using an existing dataset involves a series of analytic decisions that could alter the results, particularly with respect to changes in calibration, changes in raw consistency, and changes in the frequency cutoff. To test the robustness of our results, we have applied systematic robustness tests to assess the consequences of changes in our analytic decisions (documented in the Appendix) (Oana & Schneider, 2021). Here, we focus our attention on alternative calibration strategies, as they entail the application of different conceptual criteria.…”

Gemeinwohlorientierte Infrastruktur und die Verminderung von Einsamkeit in Europa

“…This “configurational robustness testing” resulted in 17,280 unique theoretically valid combinations of analysis choices and 67,788 individual solution models. This new procedure is consistent with the state of the art in robustness testing (Oana and Schneider 2021 ), except that our approach advances the field by considering only conceptually valid choices and maps out the complete range of robustness choice combinations instead of each of the three robustness choices in isolation (Online Appendix 3).…”

The potential of consecutive qualitative comparative analysis as a systematic strategy for configurational theorizing

“…Researchers are expected to determine whether these differences arise from weak robustness in the earlier model, different scope conditions and contexts, or a different calibration of conditions or outcome. While researchers dealing with this challenge do not yet have clear guidelines at their disposal, they can rely on a growing literature body with various strategies to incorporate the time dimension into the analysis (Caren and Panofsky, 2005; Pagliarin and Gerrits, 2020; Ragin and Fiss, 2017; Verweij and Vis, 2021), robustness checks (Oana and Schneider, 2021), theory evaluation (Oana and Schneider, 2018), and the role of context and scope conditions (Falleti and Lynch, 2009). All these resources can help in this endeavor.…”

“…a growing literature body with various strategies to incorporate the time dimension into the analysis (Caren and Panofsky, 2005;Pagliarin and Gerrits, 2020;Ragin and Fiss, 2017;Verweij and Vis, 2021), robustness checks (Oana and Schneider, 2021), theory evaluation (Oana and Schneider, 2018), and the role of context and scope conditions (Falleti and Lynch, 2009). All these resources can help in this endeavor.…”