Quality of quality: A diagnostic approach to qualitative evaluation

Abstract

This article discusses the integration of a diagnostic lens in qualitative or mixed methods evaluations, arguing that this will improve quality: in particular, it will improve transparency, credibility and reliability of evaluation findings.We start by unpacking the notion of evaluation quality and pointing out the typical weaknesses of qualitative methods. We then introduce the basic notions of diagnostic approaches and how they relate to theory-based evaluation, process tracing, and Bayesian updating, arguing for the merits of taking a formal Bayesian approach, founded on the confusion matrix, which amongst else reduces confirmation bias and conservatism. This article draws parallels between the process tracing tests and elements of the confusion matrix.

Diagnostic evaluation and Bayesian Updating: Practical solutions to common problems

“…In Befani (2020) and throughout this article, we use as an example an influence theory stating that 'our' organisation (the one whose activity we are evaluating) has had an influence on another organisation's policy or strategy. In order for this statement to be testable, more conceptual precision is required, for example, focusing on specific forms of influence that can be clearly identified and tested empirically, like the following: 'this specific knowledge product has been used by this organisation in this way to produce this specific document, which has informed a process resulting in a specific section of their strategy', and so on.…”

“…Many of the benefits provided by diagnostic evaluation have been outlined in Befani (2020), including the minimisation of CB and conservatism. This article has shown how to apply the approach in practice, following the typical steps; addressing the many risks and types of biases arising along the way, and suggesting protective measures.…”

“…Process Tracing with Bayesian Updating is increasingly popular as a methodological option in the evaluator's toolkit. A companion article (Befani, 2020) has referred to it as a 'diagnostic' approach and linked it to the confusion matrix, discussing its benefits with regard to improving transparency, credibility and reliability of evaluation findings. However, the method presents a relatively steep learning curve and can run into a series of practical problems in some application settings.…”

“…This article provides suggestions on how to solve these problems, structuring the proposed solutions around three basic, iterative application steps: developing a testable theory; designing data collection; evaluating empirical data and updating confidence. All three steps come with perils in relation to confirmation bias (CB) and conservatism, which Bayesian Updating helps protect against (Befani, 2020).…”

Diagnostic evaluation with simulated probabilities

“…Evaluation has developed a range of methods to draw out causal relationships that are distinct from established tools like randomised controlled trials (RCTs) (Befani and O’Donnell, 2016; Stern et al, 2012), which is often seen as the gold standard but requires that the environment be controlled. Diagnostic Theory-Based Evaluation, which applies Bayesian Updating to Theory-Based Evaluation, has been proposed as a promising methodology to account for the uncertainty inherent in testing causal theories or mechanisms (Befani, 2020a). Historically, when Bayesian principles have been applied to theory-based evaluation, various labels have been used: Bayesian Process Tracing (Fairfield and Charman, 2017), Process Tracing with Bayesian Updating (Befani et al, 2016), and Contribution Tracing (Befani and Stedman-Bryce, 2017).…”