Feasibility and Validation of Real-Time Patient Evaluations of Internal Medicine Interns' Communication and Professionalism Skills

Abstract

Background Residents receive little information about how they interact with patients. Objective This pilot study assessed the feasibility and validity of a new 16-item tool developed to assess patients' perspectives of interns' communication skills and professionalism and the team's communication. Methods Feasibility was determined by the percentage of surveys completed, the average time for survey completion, the percentage of target interns evaluated, and the mean number of evaluations per intern. Generalizability was analyzed using an (evaluator:evaluatee) × item model. Simulated D studies estimated optimal numbers of items and evaluators. Factor analysis with varimax rotation was used to examine the structure of the items. Scores were correlated with other measures of communication and professionalism for validation. Results Most patients (225 of 305 [74%]) completed the evaluation. Each survey took approximately 6.3 minutes to complete. In 43 days over 18 weeks, 45 of 50 interns (90%) were evaluated an average of 4.6 times. Fifty evaluations would be required to reach a minimally acceptable coefficient (0.57). Two factor structures were identified. The evaluation did not correlate with faculty evaluations of resident communication but did correlate weakly (r = 0.140, P = .04) with standardized patient evaluations. Conclusions A large number of patient evaluations are needed to reliably assess intern and team communication skills. Evaluations by patients add a perspective in assessing these skills that is different from those of faculty evaluations. Future work will focus on whether this new information adds to existing evaluation systems and warrants the added effort.

How Navigating Uncertainty Motivates Trust in Medicine

“…Surveys abound sent by corporate-run health care systems that ask patients to evaluate a particular clinical visit, including questions about the cordiality and professionalism of staff [10]. Historically, professionalism was dictated by physicians themselves in the careful organization of training from the start of medical school through residency [11].…”

From Structured, Standardized Assessment to Unstructured Assessment in the Workplace

“…This principle is adhered to in many tests, and it is why in an objective structured clinical examination (OSCE) we add the performance on a chest examination station to those on resuscitation and communication stations to form a total score for ''skills,'' despite the counterintuitive nature of such an approach. Yet, collecting assessment information from patients, as performed by Dine et al, 1 is fundamentally different from the assessment modalities described above. One may question whether a construct-based approach is the most appropriate in this context or whether we need different approaches to determining the quality of patient-based assessments.…”

“…Dine et al, 1 who study patient assessment of internal medicine resident skills, acknowledge that is it not possible to obtain standardization or structuring in all facets of an assessment program, yet these very aspects are important. As often occurs when patient assessments are used, no attempts were made to train the patients providing the assessments; instead their ''raw'' information was collected and used.…”

“…1 The interactions between an intern and a patient may all have occurred at different points during the intern's learning periods. Even if these were taken into account, which is statistically possible, the researcher would still have to assume that learning takes place exactly in the same way in all trainees, and that is not a very plausible assumption.…”

Eliciting the impact of digital consulting for young people living with long term conditions (LYNC study): which Patient Reported Outcome Measure? (Preprint)

“…The aim of the PHBQ instrument development was to understand what physician humanistic behaviours, performed (or not) in the physician-patient interaction, were important to patients and to develop a measure to assess for these behaviours in different health care contexts (15). No further evaluations of the PHBQ full scale were identified although a number of investigators have used items from the PHBQ in new outcome measures or used the PHBQ as a bench mark for convergent validity when developing new measures (28)(29)(30). In summary, the PHBQ evidences satisfactory face, content and convergent validity but reports no reliability evidence.…”