Reliability and validity of the script concordance test for postgraduate students of general practice

Subra, J.; Chicoulaa, Bruno; Stillmunkès, André; Mesthé, Pierre; Oustric, Stéphane; Bugat, Marie-Ève Rougé

doi:10.1080/13814788.2017.1358709

Cited by 24 publications

(20 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It shows that a topperforming undergraduate student will reason out well at the post graduate's pro ciency level. Similar ndings of SCT scores differentiating the participants based on their expertise level have been noted in the literature (13)(14)(15).…”

Section: Relationship Of Consensus Index With Cronbach's Alphasupporting

confidence: 87%

Consensus Analysis for the Panel Responses and the Psychometrics Properties in Script Concordance Testing

Ganesan

Bhandary

Thulasingam

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: Clinical reasoning is an essential attribute in the teaching, learning, and assessment part of medical education for undergraduates. In using the Script Concordance Test (SCT) to foster clinical reasoning, expert panel members’ responses are initially created. There is no agreement in optimizing the panel members’ responses. Our study aimed to develop and validate an SCT and test the utility of the consensus index and panel response pattern. Methods: The methodology was an evolving pattern of constructing SCTs, administering them to the panel members, optimizing the panel with response pattern and consensus index. The SCT’s final items were chosen to be administered to the students. Item-total correlation and Cronbach’s alpha were calculated from the students’ scores. Results: Our study developed an SCT with 98 items and was administered to 20-panel members. The mean score of the panel members for these 98 items was 79.5 (+/- 4.4 SD). On optimizing with the panel responses, 14 items had a uniform response pattern, and 2 had bimodal response patterns. The consensus index calculated for the 98 item SCT ranged from 25.81 to 100. When the 16 items of bimodal and uniform response pattern were eliminated, the consensus index ranged from 58.65 to 100. We administered this 82 items SCT to 30 undergraduate and ten postgraduate students. The mean score of undergraduate students was 61.1 (+/-7.5 SD), and that of postgraduate students was 67.7 (+/- 6.3 SD), which was statistically significant using an independent t-test. Cronbach’s alpha for this 82 item SCT was 0.74. On analysing the item-total correlation, 22 items had a correlation of less than 0.05. Excluding these 22 poor items, the final SCT instrument of 60 items had a Cronbach’s alpha of 0.82. Conclusion: The consensus index can also be used to optimize the items for panel responses in SCT. Our study revealed that a consensus index of above 60 had a good item-total correlation with good internal consistency. Our study also revealed that the panel response clustering pattern could also be used to categorize the items though bimodal and uniform distribution patterns need further differentiation.

show abstract

Section: Relationship Of Consensus Index With Cronbach's Alphasupporting

confidence: 87%

Consensus Analysis for the Panel Responses and the Psychometrics Properties in Script Concordance Testing

Ganesan

Bhandary

Thulasingam

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This could partially be explained by the fact that significant improvement in clinical reasoning with clinical experience is a progressive process occurs over a significant period of time. Subra et al found that postgraduate students’ clinical reasoning skills take time to develop and the largest improvement occurs during the first 18 months of training in general practice 19 . The smaller effect size could also be due to a plausible confounder – i.e.…”

Section: Discussionmentioning

confidence: 99%

“…A further study used SCT scenarios to assess the reasoning skills of paediatric residents and neonatal-perinatal medicine fellows (qualified specialists), and reported a significant difference between all training levels from Post-graduate Year 1 (PGY-1) to PGY-3 and between PGY-3 and fellows, with improvement of scores observed for each progressive level of medical training 18 . More recently, Subra administered an SCT assessment to post-graduate students in general practice and showed progression of clinical reasoning throughout the 3 years of training pathway especially in the first 18-months 19 . However, there is an apparent gap in the literature, specifically in relation to empirical evidence of progression in clinical reasoning skills for medical students in undergraduate medical education.…”

Section: Introductionmentioning

confidence: 99%

Construct validity of script concordance testing: progression of scores from novices to experienced clinicians

2019

View full text Add to dashboard Cite

ObjectivesTo investigate the construct validity of Script Concordance Testing (SCT) scores as a measure of the clinical reasoning ability of medical students and practising General Practitioners with different levels of clinical experience.MethodsPart I involved a cross-sectional study, where 105 medical students, 19 junior registrars and 13 experienced General Practitioners completed the same set of SCT questions, and their mean scores were compared using one-way ANOVA. In Part II, pooled and matched SCT scores for 5 cohorts of students (2012 to 2017) in Year 3 (N=584) and Year 4 (N=598) were retrospectively analysed for evidence of significant progression. ResultsA significant main effect of clinical experience was observed [F(2, 136)=6.215, p=0.003]. The mean SCT score for General Practitioners (M=70.39, SD=4.41, N=13) was significantly higher (p=0.011) than that of students (M = 64.90, SD = 6.30, N=105). Year 4 students (M=68.90, SD= 7.79, N=584) scored a significantly higher mean score [t(552)=12.78, p<0.001] than Year 3 students (M = 64.03, SD=7.98, N=598). ConclusionsThe findings that candidate scores increased with increasing level of clinical experience add to current evidence in the international literature in support of the construct validity of Script Concordance Testing. Prospective longitudinal studies with larger sample sizes are recommended to further test and build confidence in the construct validity of SCT scores.

show abstract

“…We conducted a descriptive analysis of our data in which quantitative variables are described using the mean (+/- standard deviation) and/or median (+/- interquartile range) according to their distribution. With regard to SCT, the responses from the senior panel were used to score the assessment based on standard methods [ 36 , 38 ]. For each item, the credit refers to the number of panel members who chose that answer, divided by the modal value for the question.…”

Section: Methodsmentioning

confidence: 99%

Teaching emergency situations during a psychiatry residency programme using a blended learning approach: a pilot study

et al. 2021

View full text Add to dashboard Cite

Background Emergency psychiatry is an essential component in the training of psychiatry residents who are required to make patient-centred orientation decisions. This training calls for specific knowledge as well as skills and attitudes requiring experience. Kolb introduced a theory on experiential learning which suggested that effective learners should have four types of abilities: concrete experience, reflective observation, abstract conceptualisation and active experimentation. We aimed to evaluate a resident training programme that we designed for use in an emergency psychiatry setting based on the experimental learning theory. Methods We designed a four-step training programme for all first-year psychiatry residents: (i) theoretical teaching of psychiatric emergency knowledge, (ii) concrete experience of ability teaching involving an initial simulation session based on three scenarios corresponding to clinical situations frequently encountered in emergency psychiatry (suicidal crisis, hypomania and depressive episodes), (iii) reflective observation and abstract conceptualisation teaching based on videos and clinical interview commentary by a senior psychiatrist for the same three scenarios, (iv) active experimentation teaching during a second simulation session based on the same three frequently encountered clinical situations but with different scenarios. Training-related knowledge acquisition was assessed after the second simulation session based on a multiple-choice quiz (MCQ), short-answer questions and a script concordance test (SCT). The satisfaction questionnaire was assessed after the resident had completed his/her initial session in order to evaluate the relevance of teaching in clinical practice. The descriptive analyses were described using the mean (+/- standard deviation). The comparative analyses were conducted with the Wilcoxon or Student’s t tests depending on data distribution. Results The residents’ mean MCQ and short-answer question scores and SCT were 7.25/10 (SD = 1.2) 8.33/10 (SD = 1.4), 77.5/100 (SD = 15.8), respectively. The satisfaction questionnaire revealed that 67 % of residents found the teaching consistent. Conclusion We designed a blended learning programme that associated, classical theoretical learning to acquire the basic concepts, a learning with simulation training to experiment the clinical situations and a video support to improve learning of interview skills and memory recall. The residents indicate that this training was adequate to prepare them to be on duty. However, despite this encouraging point, this program needs further studies to attest of its efficiency.

show abstract

Reliability and validity of the script concordance test for postgraduate students of general practice

Cited by 24 publications

References 30 publications

Consensus Analysis for the Panel Responses and the Psychometrics Properties in Script Concordance Testing

Consensus Analysis for the Panel Responses and the Psychometrics Properties in Script Concordance Testing

Construct validity of script concordance testing: progression of scores from novices to experienced clinicians

Teaching emergency situations during a psychiatry residency programme using a blended learning approach: a pilot study

Contact Info

Product

Resources

About