Changing estimates of leadership ability before a programme: retrospective self-assessments and response-shift bias

Abstract

BackgroundMost evaluations of clinical leadership development programmes rely on self-assessments. Self-assessments are vulnerable to response-shift bias. Using retrospective then-tests may help to avoid this bias.In this study, we investigate whether post-programme then-tests (retrospective self-assessments) are more sensitive to change in clinical leadership development programme participants than traditional pre-programme pre-tests when paired with post-test self-assessments.Methods17 healthcare professionals participated in an 8-month single-centre multidisciplinary leadership development programme. Participants completed prospective pre-test, retrospective then-test and traditional post-test self-assessments using the Primary Colours Questionnaire (PCQ) and Medical Leadership Competency Framework Self-Assessment Tool (MLCFQ). Pre–post pairs and then–post pairs were analysed for changes using Wilcoxon signed-rank tests and compared with a parallel multimethod evaluation organised by Kirkpatrick levels.ResultsA greater number of significant changes were detected using then-test pairs than pre-test pairs for both the PCQ (11 of 12 vs 4 of 12 items) and MLCFQ (7 of 7 vs 3 of 7 domains). The multimethods data showed positive outcomes at all Kirkpatrick levels.ConclusionsIn ideal circumstances, both pre-test and then-test evaluations should be conducted. We cautiously suggest that if only one post-programme evaluation can be conducted, then-tests may be appropriate means of detecting change.

Next Generation Leaders Programme: A Multi-Methods Evaluation of a Leadership Development Programme for Biomedical Researchers

“…This was most evident in the PCQ tasks of leadership, where retrospectively all five tasks showed a significant improvement, whereas only "Inspire" showed a significant prospective increase. It is possible the retrospective pairs were either more sensitive at detecting change or liable to overinflating change due to bias, as discussed in a previous paper 15 and by experts in retrospective assessments. 18 Of those who completed the programme, it should be noted one participant reported that they did not enjoy the programme.…”

“…This contrasts with a previous methodological study where there was broad concordance between the PCQ and MLCFQ outcomes. 15 The COVID-19 pandemic occurred partway through the programme and resulted in a quarter of participants needing to leave the programme. It also resulted in changes to clinical practice for many participants, programme workshops being moved online, and in many cases reduced the personal contact between research team members.…”

“…While the MLCFQ and PCQ instruments have been developed from theory and assessed by experts for face validity, they still rely on participant self-assessments and are thus vulnerable to bias, including “response-shift” bias, 20 bias from an implicit theory of change and error justification bias. 15 , 18 We employed both prospective and retrospective self-assessments in an attempt to partially mitigate these risks of bias. There remains no established objective method of evaluation of a healthcare or biomedical leadership development programme.…”

“…Participants completed these questionnaires pre-and post-programme, with a retrospective then-test (“what were you like back then”) added to each item at the post-programme evaluation. The two questionnaires used were the Primary Colours Questionnaire (PCQ) 15 and an adapted version of the Medical Leadership Competency Framework Self-Assessment Tool (MLCFQ) (see Supplementary Materials ), both of which have been assessed by experts for face validity and were developed from established leadership theories. 16 The Next Generation Leaders programme content and methods were mapped to the tasks and domains of leadership identified within the PCQ (see Supplementary Materials ).…”