Psychometric properties of a standardized-patient checklist and rating-scale form used to assess interpersonal and communication skills

Abstract

The results show that the SP checklist scores and the SP ratings of interpersonal and communication skills have comparable psychometric properties. The reliabilities of the five-item rating form (.76) and the single global rating of patient satisfaction (.70) were slightly higher than the reliability of the 17-item checklist (.65); this finding is of particular significance, given the greater length of the checklist. Also, the checklist scores and ratings appear to be measuring the same underlying dimension, with correlations of the checklist with the five ratings and with the single global rating being .82 and .81, respectively. Van der Vleuten and associates, in two excellent articles, noted a recent shift away from the use of subjective measures of clinical competence, such as rating scales, toward the use of presumably more objective measures, such as SP checklists. Their concern was that these objective measures may focus on somewhat trivial and easily measured aspects of the clinical encounter, and that more subtle but critical factors in clinical performance may be overlooked or ignored. They referred to such measurement as "objectified" rather than objective. The shift is based on the presumption that objective or objectified measurement is superior to subjective measurement, such as ratings, with respect to psychometric properties such as reliability. On the basis of a survey of several studies, though, the authors concluded that "objectified methods do not inherently provide more reliable scores" and "may even provide unwanted outcomes, such as negative effects on study behavior and triviality of the content being measured." The results of the present study support this conclusion, showing somewhat higher reliabilities for subjective ratings than for the objective (or perhaps objectified) checklist. Also, the high uncorrected correlations suggest that the more reliable ratings are measuring the same underlying dimension as are the checklist scores. The present study also illustrates the application of a recently proposed method for constructing a valid SP checklist, which would consist of items that best reflect global ratings of performance. In this study, the ratings were provided by the SPs themselves, but ratings could be obtained from faculty-physician experts who observe student performance on the SP case. Thus, performance on individual checklist items would be correlated with expert ratings, to identify the items that best predict the ratings. The checklist, then, would be constructed of just those items that best predict the ratings, and the checklist could be used for future testing without the need for further faculty ratings (yet the checklist scores would reflect the faculty ratings). With this approach, it would seem possible to construct checklists for history-taking and physical-examination skills, as well as for interpersonal and communication skills. Thus, the faculty ratings would provide a basis for case development and refinement, including scoring and standard setting, and scores o...

An Investigation of the Standardised Patient Interview Rating Scale (SPIRS) for the Assessment of Speech Pathology Students in a Simulation Clinic

“…The less technical nature of these skills requires the tool to be sensitive to subtle differences in skill level, and to recognize the breadth of clinical competencies (Barry, Bradshaw, and Noonan 2013, Levine and Swartz 2008, Van Der Vleuten, Norman, and De Graaff 1991. In contrast, dichotomous checklists tend to be used in assessment of specific clinical examination techniques (Cohen et al 1996). A rating scale was chosen as appropriate for assessing and providing feedback on clinical skills in the current study and, importantly, offered the opportunity to provide students with specific feedback on their performance level.…”

“…Cox (1980) concluded that five response alternatives was ideal, highlighting that such a number allowed not only for expected variation between examinees but also was effective in reducing the overuse of the 'neutral' category and allowing for valid ratings of examinees "along a continuum representing a single attribute " (1980: 409). Similarly, Cohen et al (1996) suggested a 5 point scale for rating interpersonal and communication skills and used the terms: unacceptable; poor; average; good; and excellent. Understanding of each term can be facilitated by comprehensive instructions and labelling of the alternatives (Cox 1980) together with an acceptable level of training (Huber et al 2005).…”

Examen clínico objetivo y estructurado formativo tras inmersión clínica precoz empleando estudiantes de sexto curso como observadores y administradores de retroalimentación

“…Sin embargo, muchos expertos en desarrollo de los ECOE recomiendan usar también escalas globales para identificar áreas generales de competencia tales como la habilidad de organización y comunicación, y otros constructos independientes de la estación que no pueden captarse con los listados de ítems [15]. En cualquier caso, la fiabilidad y validez comparativas de listados y escalas globales, y en qué medida éstos pueden ser utilizados por diferentes tipos de examinadores de una manera no sesgada y válida, sigue siendo objeto de controversia [15][16][17][18][19][20][21][22][23][24][25][26][27]. Algunos estudios muestran que los listados parecen más fiables que las escalas globales [8,21,25], mientras que otros muestran que éstas son tan fiables como los listados cuando son usadas por los pacientes estandarizados [23], o incluso más fiables cuando son empleadas por profesores.…”

“…En nuestro ECOE, las puntuaciones medias sobre el nivel de competencia otorgadas por los pacientes estandarizados y los estudiantes observadores fueron similares, y existió una buena correlación entre las mediciones hechas con los listados y la escala global por ambos observadores. Por ello, y en línea con otros estudios [8,19,24], nuestra impresión preliminar es que estudiantes y pacientes adecuadamente entrenados son capaces de hacer distinciones globales acertadas sobre el nivel de entrenamiento clínico de los candidatos y transferirlas a una escala global de la competencia de manera fiable.…”

Role of Simulation in US Physician Licensure and Certification

“…A sizable literature has emerged that addresses the adequacy of measurements obtained using SPs as a means of assessment. 12,13 Today, SPs are used extensively for high-stakes assessments, such as the USMLE Step 2 Clinical Skills (CS) Examination (described in further detail later in this article).…”