Development of a Self-Rated Mixed Methods Skills Assessment: The National Institutes of Health Mixed Methods Research Training Program for the Health Sciences

Abstract

Introduction Demand for training in mixed methods is high, with little research on faculty development or assessment in mixed methods. We describe the development of a Self-Rated Mixed Methods Skills Assessment and provide validity evidence. The instrument taps six research domains: “Research question,” “Design/approach,” “Sampling,” “Data collection,” “Analysis,” and “Dissemination.” Respondents are asked to rate their ability to define or explain concepts of mixed methods under each domain, their ability to apply the concepts to problems, and the extent to which they need to improve. Methods We administered the questionnaire to 145 faculty and students using an internet survey. We analyzed descriptive statistics and performance characteristics of the questionnaire using Cronbach’s alpha to assess reliability and an ANOVA that compared a mixed methods experience index with assessment scores to assess criterion-relatedness. Results Internal consistency reliability was high for the total set of items (.95) and adequate (>=.71) for all but one subscale. Consistent with establishing criterion validity, respondents who had more professional experiences with mixed methods (e.g., published a mixed methods paper) rated themselves as more skilled, which was statistically significant across the research domains. Discussion This Self-Rated Mixed Methods Assessment instrument may be a useful tool to assess skills in mixed methods for training programs. It can be applied widely at the graduate and faculty level. For the learner, assessment may lead to enhanced motivation to learn and training focused on self-identified needs. For faculty, the assessment may improve curriculum and course content planning.

Skills Development and Academic Productivity of Scholars in the NIH Mixed Methods Research Training Program for the Health Sciences

“…As of the preparation of this manuscript, the first cohort entered the program two years ago and the second entered one year ago. We administered a self-rated mixed methods skills assessment questionnaire (Guetterman et al, 2017) to participating scholars before and after the retreat. Mixed methods skills were rated under six domains; namely, “Research question,” “Design/approach,” “Sampling,” “Data collection,” “Analysis,” and “Dissemination.” To assess academic productivity related to mixed methods, we sent a follow-up survey to scholars from both cohorts (i.e., at one year and two years since entry) to ask about grants, publications, and presentations involving mixed methods.…”

“…We distributed a Scholar Mixed Methods Skills Self-Assessment instrument (see Appendix A) designed for this program (Guetterman et al, 2017). We collected the baseline data upon entry to the program, approximately four months prior to the retreat and collected the post-retreat assessment two weeks after the retreat.…”

Mixed methods grant applications in the health sciences: An analysis of reviewer comments

“…Competitively selected scholars (primarily early-career faculty) participate in 1) a three-day in-person retreat with lectures and interactive discussions about their projects, 2) webinars on mixed methods topics, and 3) ongoing support from a mentor selected from a network of consultants created for the MMRTP who each have content and mixed methods expertise. [17–19]…”

Process Evaluation of a Retreat for Scholars in the First Cohort: The NIH Mixed Methods Research Training Program for the Health Sciences

“…We collected data through observations during the entire program retreat, mini-interviews with scholars, and the Mixed Methods Scholar Self-Assessment (the subject of other manuscripts: Guetterman et al, 2016) The program evaluator (TG), who is not a mentor in the program, collected observations throughout the retreat using an observational protocol. The reason for observations was to provide real-time documentation of topics discussed, to allow questions from scholars, to encourage interaction, and to eliminate the need to rely on recall of events after the retreat.…”