Patient management problems Issues of validity

Abstract

Patient management problems (PMP) are being used in medical examinations with increasing frequency despite evidence which throws doubt on their validity as measures of clinical competence. This study investigated the construct validity of a PMP constructed in both written and interview formats. Each test was administered to groups of students of different seniorities and to two groups of doctors, interns and post-interns. The pattern of scores for the different groups was not that expected of a valid test of competence. The most competent groups (the post-interns) generally scored less well on the calculated indices than the senior students and interns. These findings were similar for both formats of the test so cueing was not thought to be the major factor. It appears that the scoring system is at fault. A comparison of performance on the written and interview (uncued) formats showed that many more options were chosen by all groups tested on the written PMP. It was concluded that written PMPs cannot yet be regarded as a valid simulation of clinical performance. Although content validity is high this does not appear to be so for construct validity or concurrent validity.

Validity of patient management problems re-examined

“…The present study reanalysed their data using analysis of variance followed by Scheffé a postiori comparisons of the mean scores for each group. Results suggest virtually no difference among the various groups’ performances on this particular PMP, in contrast to the assertion of Newble et al (1982) that ‘the most competent groups (the post‐interns) generally scored less well on the calculated indices than the senior students and interns’. While Newble et al (1982) maintained that their findings do not show the PMP to be a valid test of competence, the present analysis suggests that their study was not an adequate test of the construct validity of PMPs due to lack of power and insufficient sample size.…”

“… Summary Newble et al (1982) examined the construct validity of one Patient Management Problem (PMP) with groups of subjects with varied levels of medical competence, ranging from fourth‐year medical students to post‐intern medical registrars and consultant physicians. The present study reanalysed their data using analysis of variance followed by Scheffé a postiori comparisons of the mean scores for each group.…”

Critical care nurses’ decision‐making activities in the natural clinical setting

“…In both nursing ( Benner, 1984; Tanner et al ., 1987 ) and medicine ( Norman et al ., 1982 ), some subjects have been found to perform well on some problems and poorly on others, as a result of differing levels of knowledge and experience in certain areas. In addition, Newble et al . (1982 ) found that the availability of cues in simulated situations increased the number of cues used by clinicians.…”

Évaluation du raisonnement clinique en médecine d’urgence : les tests de concordance des scripts décèlent mieux l’expérience clinique que les questions à choix multiples à contexte riche

“…Les seniors ont obtenu des scores intermédiaires. Là encore, ces résultats étaient attendus, les QCM 13 , comme d'ailleurs les questions rédactionnelles 14 , pouvant ne pas distinguer les seniors des juniors (étudiants, résidents). À l'inverse, le TCS a pu distinguer les trois catégories de cliniciens, en fonction de leur degré d'expérience, les étudiants obtenant les scores les plus faibles, les seniors les scores les plus élevés et les résidents des scores intermédiaires.…”