Psychometric investigation of competency benchmarks.

Abstract

The 2009 special issue of Training and Education in Professional Psychology focused on competency areas and assessment-related issues, presenting 2 seminal articles: "Competency Benchmarks: A Model for Understanding and Measuring Competence in Professional Psychology Across Training Levels" ("Benchmarks";Fouad et al., 2009) and "Competency Assessment Toolkit for Professional Psychology" ("Toolkit"; Kaslow et al., 2009). The Benchmarks outlined and described competency areas encompassing both foundational and functional competencies, while the Toolkit (a partner to the Benchmarks) reviewed a variety of methods and considerations for competency assessment. However, as noted at the time (i.e.

Anatomy of psychology practitioner competence: Clustering approach reveals the hierarchical organization of competencies.

“…Bias stemming from grouping items into their domains can potentially be adjusted in statistical analyses (e.g., Price et al., 2017), but such methods are not able to be practically implemented at the individual trainee level at the point when evaluations are made. In contrast, randomizing items is a relatively straightforward and pragmatic strategy for reducing the effects of such biases made even easier with the availability of such functions in online survey packages.…”

“…With regard to how competencies should be assessed, there is a dearth of psychometrically valid instruments (Callahan et al., 2017; Kaslow et al., 2009; Price, Callahan, & Cox, 2017), and when available, measurement is dominated by the widespread use of Likert‐type scales to rate competence (Gonsalvez et al., 2013; Kaslow, 2004; Kaslow et al., 2009). These scales are easy to use, inexpensive to administer, and easily customized to measure a range of global and specific competencies.…”

“…Price et al. (2017) examined data from pre‐internship and external practicum data from 94 internship doctoral psychology trainees enrolled in clinical, clinical health, and counseling programs at a large university. In total, 270 evaluations from a 52‐item Practicum Evaluation Form were subjected to item analysis through Rasch methodology informed by item response theory.…”

“…While each of the three studies described above (Callahan et al., 2017; Gonsalvez & Freestone, 2007; Price et al., 2017) reported data from a single institution, a large, multisite study recently reported end‐placement competence assessments from five different institutions in Australia (Gonsalvez et al., 2015). The authors used a cluster‐analytic strategy to examine the stepwise unfolding of subclusters, clusters, and superclusters of competencies from 204 clinical psychology trainees.…”

“…Each of these studies describes their findings as preliminary, draws attention to limitations of their contributions, and calls for additional research to corroborate and extend their findings. Critical appraisal of the extant literature highlights the need for empirical data to further our understanding of what competencies should be measured, how they are ordered, and the nature of their hierarchical organization (Callahan et al., 2017; Klonoff, 2015; Price et al., 2017). These demands are strengthened by strong evidence that summative ratings by supervisors are systematically affected by biases, especially leniency and halo (Bogo et al., 2002; Gonsalvez et al., 2013, 2019; Gonsalvez & Freestone, 2007; Vinton & Wilke, 2011).…”