Unexpected but Most Welcome

Abstract

Although combining methods is nothing new, more contributions about why and how to mix methods for validation purposes are needed. This article presents a case of validating the inferences drawn from the Participatory Evaluation Measurement Instrument, an instrument that purports to measure stakeholder participation in evaluation. Although the process was intended to be almost exclusively quantitative, one of its components unexpectedly turned into a mixed methods study. This, in turn, spurred on a cycle of instrument revision and further quantitative validation. Whereas the validation evidence is modest and tentative, it suggests that the revised version of the Participatory Evaluation Measurement Instrument offers a better fit with the respondents' opinions regarding the participation level of selected evaluation cases. The article concludes with a brief discussion on the added value of mixed methods for validation purposes.

Objectives Driven Participatory Evaluation Model

“…The quantitative data was collected using two instruments, the Participatory Evaluation Measurement Instrument (PEMI) and the Evaluation Involvement Scale (EIS) (Daigneault & Jacob, 2009;Toal, 2009). The PEMI is a nonnormative instrument, which measures the evaluation participants perception of participation on the three dimensions of Participatory Inquiry outlined by Cousins and Whitmore (1998) (Daigneault & Jacob, 2009;Daigneault & Jacob, 2013). The instrument was validated using the Instrument Development and Construct Validation process developed by Onwuegbuzie, Bustamante, and Nelson (2010), a 10-phase mixedmethods validation process (Daigneault & Jacob, 2013).…”

“…The PEMI is a nonnormative instrument, which measures the evaluation participants perception of participation on the three dimensions of Participatory Inquiry outlined by Cousins and Whitmore (1998) (Daigneault & Jacob, 2009;Daigneault & Jacob, 2013). The instrument was validated using the Instrument Development and Construct Validation process developed by Onwuegbuzie, Bustamante, and Nelson (2010), a 10-phase mixedmethods validation process (Daigneault & Jacob, 2013). Each dimension in the instrument has the same scale with different intuitive labels for the respective domain, 0 = .00, 1 = .25, 2 = .50, 3 = .75, 4 = 1.00.…”

“…For an evaluation to be considered participatory it must receive at least an average of .25 on each domain. Finally, the scores from the three domains are averaged to measure overall participation (Daigneault & Jacob, 2013). Therefore, this instrument addresses the question of whether or not an evaluation can be considered participatory.…”

“…The threshold for whether or not an evaluation could be considered participatory, according to Daigneault and Jacob (2009), was the presence of all three fundamental attributes of participation, and each category must have at least a score of .25 or greater. Daigneault and Jacob (2013) revised the process by adding a final step of averaging the scores across the three dimensions as an indicator a measurement of overall participation. The Evaluation Involvement Scale (EIS) (Toal, 2009) was developed as a response to a call for instruments designed to measure the level of involvement of stakeholders in a participatory evaluation in multisite settings.…”

Moral distress and the Australian aged care worker: A mixed methods study

“…In this study key findings from the quantitative phase (Phase 1) were used to provide a focussed base for the qualitative phase (Phase 2). Also, as a primary component of Phase 1 of this study was to validate an amended measurement instrument, data collected during the Phase 2 telephone interviews were able to be used to support the validation of the instrument and the completeness of the instrument as a whole; a strategy supported by Daigneault and Jacob (2014).…”