An iterative approach to case study analysis: insights from qualitative analysis of quantitative inconsistencies

Abstract

Large-N comparative studies have helped common pool resource scholars gain general insights into the factors that influence collective action and governance outcomes. However, these studies are often limited by missing data, and suffer from the methodological limitation that important information is lost when we reduce textual information to quantitative data. This study was motivated by nine case studies that appeared to be inconsistent with the expectation that the presence of Ostrom's Design Principles increases the like lihood of successful common pool resource governance. These cases highlight the limitations of coding and analysing large-N case studies. We examine two issues: 1) the challenge of missing data and 2) potential approaches that rely on context (which is often lost in the coding process) to address inconsistencies between empirical observations and theoretical predictions. For the latter, we conduct a post-hoc qualitative analysis of a large-N comparative study to explore 2 types of inconsistencies: 1) cases where evidence for nearly all design principles was found, but available evidence led to the assessment that the CPR system was unsuccessful and 2) cases where the CPR system was deemed successful despite finding limited or no evidence for design principles. We describe inherent challenges to large-N comparative analysis and to coding complex and dynamically changing common pool resource systems for the presence or absence of design principles and the determination of "success". Finally, we illustrate how, in some cases, our qualitative analysis revealed that the identity of absent design principles explained inconsistencies, and hence reconciled such apparent inconsistencies with theoretical predictions. This analysis demonstrates the value of combining quantitative An iterative approach to case study analysis 469 and qualitative analysis, and using mixed-methods approaches iteratively to build comprehensive methodological and theoretical approaches to understanding common pool resource governance in a dynamically changing context.

An Analysis of the Global Applicability of Ostrom’s Design Principles to Diagnose the Functionality of Common-Pool Resource Institutions

“…It also addressed the relative importance of DPs both within and across activities. Barnett et al [30] addressed the temporal issue of a certain CPR being successful at time 1 and unsuccessful at time 2, or a seemingly unsuccessful CPR succeeding in later years. Although these observations are true, the dynamics of relativity should not be overlooked.…”

“…Barnett et al [30] attributed inconsistent conclusions for CPR governance based on DPs to investigator's bias (missing data), procedural (coding process) and substantive (codebook development) errors. Whereas these factors are attributable to the scholars' studying cases, another important factor attributable to the CPR institution itself is the nominal presence of the DPs affected by poor implementation of CPR management.…”

Institutions and the performance of coupled infrastructure systems

“…The work of Baggio et al (2016) and Barnett et al (2016) in this special issue is a first step in attempting to understand how institutional configurations may produce different outcomes depending on the context in which they operate. As we might expect, just as we now know with genes, the Design Principles do not operate individually.…”

Introducing the "The Importance of Context, Scale, and Interdependencies in Understanding and Applying Ostrom's Design Principles for Successful Governance of the Commons"

“…More importantly, the legitimacy and credibility of the scientific enterprise is called into question with-out the ability and opportunity to examine and question the basis for scientifically produced claims. 4 The third manuscript, Barnett et al (2016) "An iterative approach to large-N studies: insights from qualitative analysis of quantitative inconsistencies" draws on the methods of Ratajczyk et al (2016) to build off of the findings of and illustrate the value of complementing quantitative analysis with qualitative analysis. Recall, found that successful cases of CPR governance exhibited 8 or more design principles.…”

“…Recall, found that successful cases of CPR governance exhibited 8 or more design principles. Barnett et al (2016), using qualitative methods, examine two sets of what appear to be anomalous cases. Type 1 anomalies are cases that exhibit most of the design principles but are not characterized by successful outcomes.…”