Background

Background Although averaging across multiple examiners’ judgements reduces unwanted overall score variability in objective structured clinical examinations ( OSCE ), designs involving several parallel circuits of the OSCE require that different examiner cohorts collectively judge performances to the same standard in order to avoid bias. Prior research suggests the potential for important examiner‐cohort effects in distributed or national examinations that could compromise fairness or patient safety, but despite their importance, these effects are rarely investigated because fully nested assessment designs make them very difficult to study. We describe initial use of a new method to measure and adjust for examiner‐cohort effects on students’ scores. Methods We developed video‐based examiner score comparison and adjustment ( VESCA ): volunteer students were filmed ‘live’ on 10 out of 12 OSCE stations. Following the examination, examiners additionally scored station‐specific common‐comparator videos, producing partial crossing between examiner cohorts. Many‐facet Rasch modelling and linear mixed modelling were used to estimate and adjust for examiner‐cohort effects on students’ scores. Results After accounting for students’ ability, examiner cohorts differed substantially in their stringency or leniency (maximal global score difference of 0.47 out of 7.0 [Cohen's d = 0.96]; maximal total percentage score difference of 5.7% [Cohen's d = 1.06] for the same student ability by different examiner cohorts). Corresponding adjustment of students’ global and total percentage scores altered the theoretical classification of 6.0% of students for both measures (either pass to fail or fail to pass), whereas 8.6–9.5% students’ scores were altered by at least 0.5 standard deviations of student ability. Conclusions Despite typical reliability, the examiner cohort that students encountered had a potentially important influence on their score, emphasising the need for adequate sampling and examiner training. Development and validation of VESCA may offer a means to measure and adjust for potential systematic differences in scoring patterns that could exist between locations in distributed or national OSCE examinations, thereby ensuring equivalence and fairness.

No abstract

ContextGrades are commonly used in formative workplace-based assessment (WPBA) in medical education and training but may draw attention away from feedback about the task. The dilemma is that the selfregulatory focus of a trainee must include self-awareness relative to agreed standards, which implies grading.In this study we aimed to understand the meaning which medical students construct from WPBA feedback with and without grades, and what influences this. MethodsYear 3 students were invited to take part in a randomised crossover study in which each student was their own control. Each student had one WPBA with and one without grades, and then chose whether or not to have grades with their third WPBA. These preferences were explored via semi-structured interviews. A realist approach to analysis was used to gain understanding of student preferences and the impact of feedback with and without grades. Results and discussionOf students who had feedback with and without grades, 65 (78%) then chose to have feedback with grades and 18 (22%) without grades their third WPBA. 24 students were interviewed.Students described how grades locate their performance and calibrate their self-assessment. For some, low grades focused attention and effort. Satisfactory and high grades enhanced self-efficacy.Grades are also concrete, powerful and blunt, can be harmful and need explanation to help students create helpful meaning from them. Low grades risk reducing self-efficacy in some and may encourage others to focus on proving their ability rather than on improvement.A metaphor of the semi-permeable membrane is introduced to understand how students reduced potential negative effects and enhanced the positive effects of feedback with grades by selective filtering and pumping. ConclusionThis study illuminates the complexity of the processing of feedback by its recipients, and informs the use of grading in provision of more effective, tailored feedback.

Decision aids (DA) are evidence-based tools that support shared decision-making (SDM) implementation in practice; this study aimed to identify existing osteoporosis DAs and assess their quality and efficacy; and to gain feedback from a patient advisory group on findings and implications for further research. MethodsWe searched multiple bibliographic databases to identify research studies from 2000 to 2019 and undertook an environmental scan (search conducted February 2019, repeated in March 2020). A pair of reviewers, working independently selected studies for inclusion, extracted data, evaluated each trial's risk of bias, and conducted DA quality assessment using the International Patient Decision Aid Standards (IPDAS). Public contributors (patients and caregivers with experience of osteoporosis and fragility fractures) participated in discussion groups to review a sample of DAs, express preferences for a new DA and discuss plans for development of a new DA. ResultsWe identified 6 studies, with high or unclear risk of bias. Across included studies, use of an osteoporosis DA was reported to result in reduced decisional conflict compared with baseline, increased SDM and increased accuracy of patients' perceived fracture risk compared with controls.Eleven DAs were identified, of which none met the full set of IPDAS criteria for certification for minimization of bias. Public contributors expressed preferences for encounter DAs that are individualized to patients' own needs and risk. ConclusionsExisting DAs for informing patient decisions about osteoporosis treatment fail to comprehensively meet international quality standards and patient needs, underpinning the need for new DA development.

Four focus groups were conducted with members of the public to identify important areas for future osteoporosis research. Participants identified priorities to increase public awareness of osteoporosis, reduce delays in diagnosis, improve communication between healthcare providers and to improve follow-up and information provision about causes of osteoporosis, medication harms and prognosis. Purpose Patients and the public must be involved in setting research agendas to ensure relevant and impactful questions are prioritised. This study aimed to understand what people living with osteoporosis and fragility fractures felt was important to research, to inform the content of a national survey on research priorities in this area. Methods Focus groups were conducted with members of the public with experience of osteoporosis or fragility fractures. The topic guide was co-developed with a patient and public involvement research user group, and explored participants' experiences of osteoporosis including diagnosis, management and effect upon their lives, what aspects of their ongoing care was most important to them and what about their care or condition could be improved. Focus groups were audio-recorded, transcribed and analysed thematically. Results A total of twenty-three participants were recruited to four focus groups. Analysis identified two main themes: challenges in living with osteoporosis and healthcare services for osteoporosis. Information needs was a further cross-cutting theme. Participants called for increased public awareness of osteoporosis and wanted healthcare services to address conflicting messages about diet, exercise and medication. Participants described long delays in diagnosis, poor communication between primary and secondary care and the need for structured follow-up as important areas for future research to address. Conclusion The findings from this study provide an understanding of research priorities from the perspective of patients and the public, have informed the content of a national survey and have implications for patient education, health services research and policy.

Feedback is a key component of learning but effective feedback is a complex process with many aspects. One aspect may be a written summary which is passed to the learner but this may not be valued by learners. We examined the role of written feedback in the feedback process to determine whether it does more than provide a simple summary of the interaction. We conducted a secondary analysis of data gathered for a study of formative workplace based assessment. Interview data from 24 interviews with students and written summaries of workplace based assessments for 23 of them were reanalysed by two researchers who were already immersed in the data and examined all references to verbal, informal feedback and written, formal feedback or the assessment tool used. We found that students valued the verbal feedback discussion highly and that they often considered the written summaries superfluous. We also found that the act of preparing written feedback augmented the feedback discussion and tutors had adopted the language of the formal instrument in the verbal feedback and free text written feedback.What this study adds to existing research is evidence that there may be a secondary faculty development effect of requiring the preparation of written feedback which has served to enhance the educational content of feedback. Although this is not proof of causality (the requirement to provide written feedback alone producing the positive effects), we consider that the likelihood is sufficiently strong to continue the practice.

Patient information is important to help patients fully participate in their healthcare. Commonly accessed osteoporosis patient information resources were identified and assessed for readability, quality, accuracy and consistency. Resources contained inconsistencies and scored low when assessed for quality and readability. We recommend optimal language and identify information gaps to address. Introduction The purpose of this paper is to identify commonly accessed patient information resources about osteoporosis and osteoporosis drug treatment, appraise the quality and make recommendations for improvement. Methods Patient information resources were purposively sampled and text extracted. Data extracts underwent assessment of readability (Flesch Reading Ease and Flesch-Kincaid Grade Level) and quality (modified International Patient Decision Aid Standards (m-IPDAS)). A thematic analysis was conducted, and keywords and phrases were used to describe osteoporosis and its treatment identified. Findings were presented to a stakeholder group who identified inaccuracies and contradictions and discussed optimal language. Results Nine patient information resources were selected, including webpages, a video and booklets (available online), from government, charity and private healthcare providers. No resource met acceptable readability scores for both measures of osteoporosis information and drug information. Quality scores from the modified IPDAS ranged from 21 to 64% (7-21/33). Thematic analysis was informed by Leventhal's Common-Sense Model of Disease. Thirteen subthemes relating to the identity, causes, timeline, consequences and controllability of osteoporosis were identified. Phrases and words from 9 subthemes were presented to the stakeholder group who identified a predominance of medical technical language, misleading terms about osteoporotic bone and treatment benefits, and contradictions about symptoms. They recommended key descriptors for providers to use to describe osteoporosis and treatment benefits. Conclusions This study found that commonly accessed patient information resources about osteoporosis have highly variable quality, scored poorly on readability assessments and contained inconsistencies and inaccuracies. We produced practical recommendations for information providers to support improvements in understanding, relevance, balance and bias, and to address information gaps.