Effect sizes of 0.8-1.0 are an unreasonable expectation from PBL because, firstly, the degree of changes that would be required of individuals would be excessive, secondly, leading up to medical school, students are groomed and selected for success in a traditional curriculum, expecting them to do better in a PBL curriculum than a traditional curriculum is an unreasonable expectation, and, thirdly, the average study reported in the literature and many commonly used and accepted medical procedures and therapies are based upon studies having lesser ESs. Information-processing theory, Cooperative learning, Self-determination theory and Control theory are suggested as providing better theoretical support for PBL than Contextual learning theory. Even if knowledge acquisition and clinical skills are not improved by PBL, the enhanced work environment for students and faculty that has been consistently found with PBL is a worthwhile goal.

The authors analyze the challenges to using academic measures (MCAT scores and GPAs) as thresholds for admissions and, for applicants exceeding the threshold, using personal qualities for admission decisions; review the literature on using the medical school interview and other admission data to assess personal qualities of applicants; identify challenges of developing better methods of assessing personal qualities; and propose a unified system for assessment. The authors discuss three challenges to using the threshold approach: institutional self-interest, inertia, and philosophical and historical factors. Institutional self-interest arises from the potential for admitting students with lower academic credentials, which could negatively influence indicators used to rank medical schools. Inertia can make introducing a new system complex. Philosophical and historical factors are those that tend to value maximizing academic measures. The literature identifies up to 87 different personal qualities relevant to the practice of medicine, and selecting the most salient of these that can be practically measured is a challenging task. The challenges to developing better personal quality measures include selecting and operationally defining the most important qualities, measuring the qualities in a cost-effective manner, and overcoming "cunning" adversaries who, with the incentive and resourcefulness, can potentially invalidate such measures. The authors discuss potential methods of measuring personal qualities and propose a unified system of assessment that would pool resources from certification and recertification efforts to develop competencies across the continuum with a dynamic, integrated approach to assessment.

The effects of problem-based learning (PBL) were examined by conducting a meta-analysis-type review of the English-language international literature from 1972 to 1992. Compared with conventional instruction, PBL, as suggested by the findings, is more nurturing and enjoyable; PBL graduates perform as well, and sometimes better, on clinical examinations and faculty evaluations; and they are more likely to enter family medicine. Further, faculty tend to enjoy teaching using PBL. However, PBL students in a few instances scored lower on basic sciences examinations and viewed themselves as less well prepared in the basic sciences than were their conventionally trained counterparts. PBL graduates tended to engage in backward reasoning rather than the forward reasoning experts engage in, and there appeared to be gaps in their cognitive knowledge base that could affect practice outcomes. The costs of PBL may slow its implementation in schools with class sizes larger than 100. While weaknesses in the criteria used to assess the outcomes of PBL and general weaknesses in study design limit the confidence one can give conclusions drawn from the literature, the authors recommend that caution be exercised in making comprehensive, curriculum-wide conversions to PBL until more is learned about (1) the extent to which faculty should direct students throughout medical training, (2) PBL methods that are less costly, (3) cognitive-processing weaknesses shown by PBL students, and (4) the apparent high resource utilization by PBL graduates.

Competency-based medical education is likely to be here for the foreseeable future. Whether or not these 5 criteria, or some variation of them, become the ultimate defining criteria for what constitutes a competency, they represent an essential step towards clearing the confusion that reigns.

Ensuring the reliability and validity of outcome measures used in clinical trials is essential to the success of the trial. The Trial of Org 10172 in Acute Stroke Treatment (TOAST) is a multicenter clinical trial that is recruiting patients with acute ischemic stroke seen at medical centers across the United States. This paper describes an approach to train physicians to use three clinical measures: the National Institutes of Health (NIH) Stroke Scale, a supplemental motor examination, and the Glasgow Outcome Scale. The program included education, certification, remediation when needed, monitoring, and reliability assessment. The goal was to ensure that interrater assessments were as equivalent to one another as possible. Of the first 95 clinicians who began the certification process, 75 passed during the first evaluation. Eighteen of the other physicians were able to complete the process after remediation. The intraclass correlations of both the NIH Stroke Scale and supplemental motor examination exceeded 0.95. The kappa values for the Glasgow Outcome Scale were 0.61 and 0.62 for the first and second ratings of the videotape, respectively. Our experience suggests that a program that includes educational and certification processes can be performed as part of the design of a multicenter clinical trial. The method of providing educational and testing videotapes to each site so that physicians can be trained and certified is an effective, inexpensive, and practical approach for enhancing and certifying the expertise of the large number of physicians involved in a multicenter study.

No abstract

Results from this pilot study suggest that self, faculty, and patient evaluations of resident performance constitutes a valid and reliable assessment of resident competence. Additional data are needed to determine whether the 360-degree assessment should be incorporated into residency programs and how frequently the assessment should be performed. Requiring only a specified number of assessments per rotation would make the process less burdensome for residents and faculty.

Changes in the healthcare environment are putting increasing pressure on medical schools to make faculty accountable and to document the quality of the medical education they provide. Faculty's ratings of students' performances and students' ratings of faculty's teaching are important elements in these efforts to document educational quality. This article discusses selected research related to factors affecting raters' judgements, analyses how changes in the health care environment are influencing such judgements, offers some suggestions to moderate some of the effects and links these influences to the system that upholds professional standards. Ratings are known to have a positive bias (generosity error), provide limited discrimination and often fail to document serious deficits. The potential sources of these problems relate to the mechanics of the rating task, the system used to obtain ratings and factors affecting rater judgement. As managed care demands reduce the time faculty have for teaching, as system-wide disincentives to provide negative ratings proliferate and as social engineering challenges, such as the Americans with Disabilities Act, impose differential standards for students, the natural tendency to avoid giving negative ratings becomes even harder to resist. Ultimately, these forces compromise the capability of faculty to uphold the standards of the profession. The author calls for a national effort to stem the erosion of those standards.