Simulation for education and training in health-care professions has been widely applied. However, its value as an assessment tool for competence is not fully known. Logistical barriers of simulation-based assessments have led some health-care organizations to utilize computer-based case simulations (CCSs) for assessment. This article provides a review of the literature on the identification of psychometrically sound, CCS instruments designed to measure decision-making competence in health-care professionals. CINAHL, MEDLINE, and Ovid databases identified 84 potentially relevant articles published between January 2000 and May 2017. A total of 12 articles met criteria for inclusion in this review. Findings of these 12 articles indicate that summative assessment in health care using CCSs in the form of clinical scenarios is utilized to assess higher order performance aspects of competence in the form of decision-making. Psychometric strength was validated in eight articles and supported by four replication studies. Two of the eight articles reported evidence of construct validity and support the need for evidence based on a theoretical framework. This literature review offers implications for further research on the use of CCS tools as a method for assessment of competence in health-care professionals and the need for psychometric evidence to support it.

No abstract

It is known that the Rasch model is a special two‐level hierarchical generalized linear model (HGLM). This article demonstrates that the many‐faceted Rasch model (MFRM) is also a special case of the two‐level HGLM, with a random intercept representing examinee ability on a test, and fixed effects for the test items, judges, and possibly other facets. This perspective suggests useful modeling extensions of the MFRM. For example, in the HGLM framework it is possible to model random effects for items and judges in order to assess their stability across examinees. The MFRM can also be extended so that item difficulty and judge severity are modeled as functions of examinee characteristics (covariates), for the purposes of detecting differential item functioning and differential rater functioning. Practical illustrations of the HGLM are presented through the analysis of simulated and real judge‐mediated data sets involving ordinal responses.

No abstract

No abstract

This article presents results of the 2019 Role Delineation Study (RDS) for pharmacotherapy specialty certification, conducted by the Board of Pharmacy Specialties (BPS). This RDS used both qualitative (ie, focus group) and quantitative (ie, survey) methodologies to shape and empirically validate the knowledge, skills, and abilities characterized by the practice performance domain of the pharmacotherapy certification content outline. Over 1100 survey responses were collected from board‐certified pharmacotherapy specialists and then analyzed and reviewed by a representative panel of subject matter experts in pharmacotherapy and psychometric analyses. Using statistical summaries of rating scale data, the panelists recommended revisions to the certification content outline and examination blueprint. Descriptions of how the survey results were used to develop test specifications are also provided. This analysis provides validity evidence for the content scope for pharmacotherapy specialist certification and for the specifications (ie, domain weight percentages) of the high‐stakes examination. In particular, the study reaffirmed the pharmacotherapy specialist certification as a clinically relevant, patient‐focused credential, consistent with the mission of the BPS.

The Board Certified Psychiatric Pharmacist (BCPP) specialty certification was launched by the Board of Pharmacy Specialties in 1994. Candidates for the BCPP can qualify for the examination through 3 possible pathways: practice experience (4 years) in the specialty, completion of a PGY-1 residency plus an additional 2 years of practice experience, or completion of a PGY-2 specialty residency in psychiatric pharmacy. Recent fluctuations in the passing rate raised questions as to explanatory factors. This article represents the first published comprehensive study of candidate performance on the BCPP Examination. It describes a retrospective, observational study presenting (a) statistical trends of examination passing rates for biannual cohorts over the past 5 years, as well as (b) score distributions on the 3 performance domains of the certification. Pass-rate trend analyses suggest that variation in the proportion of eligibility pathway cohorts in the respective testing samples explains some of the fluctuation in passing rates. An analysis of variance of domain-level scores, using groups defined by eligibility pathway, yielded significant differences for nearly all group comparisons. Evaluation of the effect sizes suggest that the most disparate performance was observed on the core clinical domain, Patient-Centered Care. The results of this study are consistent with previously published research and will inform the upcoming role delineation study for the Psychiatric Pharmacy Certification.

The subjective aspect of standard‐setting is often criticized, yet data‐driven standard‐setting methods are rarely applied. Therefore, we applied a mixture Rasch model approach to setting performance standards across several testing programs of various sizes and compared the results to existing passing standards derived from traditional standard‐setting methods. We found that heterogeneity of the sample is clearly necessary for the mixture Rasch model approach to standard setting to be useful. While possibly not sufficient to determine passing standards on their own, there may be value in these data‐driven models for providing additional validity evidence to support decision‐making bodies entrusted with establishing cut scores. They may also provide a useful tool for evaluating existing cut scores and determining if they continue to be supported or if a new study is warranted.