Quality and rigor of the concept mapping methodology: A pooled study analysis

Abstract

The use of concept mapping in research and evaluation has expanded dramatically over the past 20 years. Researchers in academic, organizational, and community-based settings have applied concept mapping successfully without the benefit of systematic analyses across studies to identify the features of a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept mapping studies to describe characteristics across study phases, generate specific indicators of validity and reliability, and examine the relationship between select study characteristics and quality indicators. Individual study characteristics and estimates were pooled and quantitatively summarized, describing the distribution, variation and parameters for each. In addition, variation in the concept mapping data collection in relation to characteristics and estimates was examined. Overall, results suggest concept mapping yields strong internal representational validity and very strong sorting and rating reliability estimates. Validity and reliability were consistently high despite variation in participation and task completion percentages across data collection modes. The implications of these findings as a practical reference to assess the quality and rigor for future concept mapping studies are discussed.

Concept mapping—An effective method for identifying diversity and congruity in cognitive style

“…The stress index for this project was 0.34, which is in the accepted range as suggested by the literature (Kane & Trochim, 2007;Rosas & Kane, 2012;Trochim, 1989).…”

“…****INSERT FIGURE 1 ABOUT HERE**** The stress value of the Adaptors' map was within the accepted range, and the stress value for the Innovators was slightly above the range found in previous meta-analytic studies of concept mapping (Rosas & Kane, 2012;Trochim, 1993). The configural similarity correlation between sort and distance matrices for Adaptors was 0.68 (p < 0.001) and 0.49 (p < 0.001) for Innovators, respectively.…”

Indicators to facilitate the early identification of patients with major depressive disorder in need of highly specialized care: A concept mapping study

“…Hence, the findings of this study cannot be generalized to nonclinical samples, children and adolescents. Third, although the number of subject‐matter experts that participated in one or more of the steps of the concept mapping process falls within the average range (20–649) of participants in concept mapping research (Rosas & Kane, 2012), it is unclear whether the participants’ conceptualization is representative of the larger population. In addition, although effort was made to include subject‐matter experts from a broad range of disciplines and countries, the majority of the participants were psychiatrists and worked as treating clinicians and/or researchers in the Netherlands.…”

Essential Public Health Competencies for Medical Students: Establishing a Consensus in Family Medicine

“…Group Concept Mapping generally consists of the following steps: (a) brainstorming, (b) idea synthesis, (c) sorting and rating of ideas, (d) multivariate analyses, and (e) interpretation. 17,[19][20][21][22] Figure 1 illustrates the process and participants at each step of the group concept mapping method and Table 1 specifies and describes the groups that participated in each step. Because not all participants shared identifying information and all responses were anonymized, exact group overlap is impossible to ascertain.…”