No abstract

Q methodology blends qualitative and quantitative, yet was only recently identified as a mixed method. Q has had a challenging 80-year history that can inform the broader but younger mixed methods community. This article introduces Q methodology and its position within mixed methods before discussing Q's struggles against dismissive voices and faulty assumptions. The benefits of research communities and mentoring are also discussed within the context of Q's history. Many of the struggles of mixed methods currently can be seen within Q's history. The purpose of this article is to use the experiences within the Q community to benefit and inform the practice of mixed methods researchers in ways that assist all of us to best study our world.

The assessment of learning has become a key component in program evaluation, grant proposals, and education research. Assessment requires valid and reliable instruments. The Force and Motion Conceptual Evaluation ͑FMCE͒ is one of several multiple-choice tests used to evaluate the learning of force and motion concepts. Although many physics education researchers accept its validity and reliability, validity and reliability estimates based on typical statistical analyses of data have not been established. This study used FMCE post-test results for approximately 150 students in a first-semester college physics course to estimate reliability and content validity. The results indicate that the FMCE is a valuable instrument for measuring student learning.

No abstract

Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting 'idiosyncratic rater variance' is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments have used open response formats to gather raters' comments and justifications. This design choice allows participants to use idiosyncratic response styles that could result in a distorted representation of the underlying rater cognition and skew subsequent analyses. In this study we explored rater variability using the structured response format of Q methodology. Physician raters viewed video-recorded clinical performances and provided Mini Clinical Evaluation Exercise (Mini-CEX) assessment ratings through a web-based system. They then shared their assessment impressions by sorting statements that described the most salient aspects of the clinical performance onto a forced quasi-normal distribution ranging from "most consistent with my impression" to "most contrary to my impression". Analysis of the resulting Q-sorts revealed distinct points of view for each performance shared by multiple physicians. The points of view corresponded with the ratings physicians assigned to the performance. Each point of view emphasized different aspects of the performance with either rapport-building and/or medical expertise skills being most salient. It was rare for the points of view to diverge based on disagreements regarding the interpretation of a specific aspect of the performance. As a result, physicians' divergent points of view on a given clinical performance cannot be easily reconciled into a single coherent assessment judgment that is impacted by measurement error. If inter-rater variability does not wholly reflect error of measurement, it is problematic for our current measurement models and poses challenges for how we are to adequately analyze performance assessment ratings.