The Value in Validity

Abstract

House's classic Evaluating with Validity proposes three dimensions—truth, justice, and beauty—for evaluation validity. A challenge to achieving validity is balancing the priorities between these three dimensions when they conflict. This chapter examines the concept of validity and the values inherent in each of these dimensions and any choices between them. Our analysis of these inherent values and any prioritization between truth, justice, and beauty aims to help the evaluator confront the kinds of dilemmas faced when one's commitment to values, evaluation theories, or methodology comes up against conflicting realities for a particular evaluation. Striking an appropriate balance can be particularly challenging in contexts involving diverse cultures or even homogenous cultures of which the evaluator is not a part. We use two case examples to explore the issues in real‐life contexts.

How do we teach and learn with cases?

“…If your frame is that there is no difference, you might, for example, choose a case that helps students be able to use and think through the possible design options for addressing methodological threats to validity (Shadish et al., 2002). In this way, your goal might be to help students understand that there are always trade‐offs to be made, and how to reason through them is what is important for rigor, or if you believe there is a difference, you might instead choose a case that helps students understand the values in validity (Griffith & Montrosse‐Moorhead, 2014). Your goal then might be to help students move from a purely technical understanding of validity, to one that is grounded in argumentation and values.…”

Nonprofit Organizations and the Evaluation of Social Impact: A Research Program to Advance Theory and Practice

“…This work provides examples of how participants’ definition of, and pathway to, significant change does not always align with the stated outcomes of the organization. In exploring this line of inquiry, nonprofit researchers might examine related evaluation scholarship that centers on the experience of participants (Abma et al, 2020; Center for Evaluation Innovation, Institute for Foundation and Donor Learning, Dorothy A Johnson Center for Philanthropy, & Luminare Group, 2017; Cousins & Whitmmore, 1998; Fetterman, 2005; VanderPlaat, 1995), widens the understanding of validity, and provides more open and responsive methodologies that recognize diverse ways of knowing and alternative forms of evidence (e.g., Cavino, 2013; Griffith & Montrosse-Moorhead, 2014; House, 1980; Thomas & Campbell, 2020). Recent efforts around beneficiary feedback aim to better understand the experiences of beneficiaries, using methodologies such as “lean data” and “constituent voice” (Dichter et al, 2016; Twersky et al, 2013), but there has been little scholarly study of the effects of such methods, and the basis upon which they establish credible evidence.…”

Combining multiple approaches to valuing in the MUVA female economic empowerment programme

“…Irrespective of the approaches and tools used, evaluation is ‘not first and foremost about methods, but is about making sense of evidence and creating a coherent, logical, and, ultimately, if successful, persuasive argument about what the evidence shows’ (Patton, 2018a, p. 18). Understanding evaluation needs and context, recognising strengths and limitations of methods, selecting and defending contextually appropriate methods, and interpreting findings accordingly are critical to safe and effective evaluation (Griffith & Montrosse-Moorhead, 2014). Applying these principles to economic evaluation might in some cases lead to a decision not to include CBA (King, 2019a).…”