Consensus Analysis for the Panel Responses and the Psychometrics Properties in Script Concordance Testing

Abstract

Background: Clinical reasoning is an essential attribute in the teaching, learning, and assessment part of medical education for undergraduates. In using the Script Concordance Test (SCT) to foster clinical reasoning, expert panel members’ responses are initially created. There is no agreement in optimizing the panel members’ responses. Our study aimed to develop and validate an SCT and test the utility of the consensus index and panel response pattern. Methods: The methodology was an evolving pattern of constructing SCTs, administering them to the panel members, optimizing the panel with response pattern and consensus index. The SCT’s final items were chosen to be administered to the students. Item-total correlation and Cronbach’s alpha were calculated from the students’ scores. Results: Our study developed an SCT with 98 items and was administered to 20-panel members. The mean score of the panel members for these 98 items was 79.5 (+/- 4.4 SD). On optimizing with the panel responses, 14 items had a uniform response pattern, and 2 had bimodal response patterns. The consensus index calculated for the 98 item SCT ranged from 25.81 to 100. When the 16 items of bimodal and uniform response pattern were eliminated, the consensus index ranged from 58.65 to 100. We administered this 82 items SCT to 30 undergraduate and ten postgraduate students. The mean score of undergraduate students was 61.1 (+/-7.5 SD), and that of postgraduate students was 67.7 (+/- 6.3 SD), which was statistically significant using an independent t-test. Cronbach’s alpha for this 82 item SCT was 0.74. On analysing the item-total correlation, 22 items had a correlation of less than 0.05. Excluding these 22 poor items, the final SCT instrument of 60 items had a Cronbach’s alpha of 0.82. Conclusion: The consensus index can also be used to optimize the items for panel responses in SCT. Our study revealed that a consensus index of above 60 had a good item-total correlation with good internal consistency. Our study also revealed that the panel response clustering pattern could also be used to categorize the items though bimodal and uniform distribution patterns need further differentiation.