Assessing clinical judgment using the Script Concordance test: the importance of using specialty-specific experts to develop the scoring key

Petrucci, Andrea; Nouh, Thamer; Boutros, Marylise; Gagnon, Robert; Meterissian, Sarkis

doi:10.1016/j.amjsurg.2012.09.002

Cited by 15 publications

(13 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With respect to the issue of recalling knowledge from long‐term memory, cognitive interviews with expert participants suggested that they lacked the knowledge to answer subspecialised items outside their area of practice. This finding supports the conclusions described by Petrucci et al 40 that suggest using specialty‐specific experts to develop the scoring key. Interestingly, the study by Petrucci et al used the same original SCT that was used for the Delphi in this study.…”

Section: Discussionsupporting

confidence: 92%

How do cognitive processes influence script concordance test responses?

et al. 2020

View full text Add to dashboard Cite

Introduction The script concordance test (SCT) is a test of clinical decision‐making (CDM) that compares the thought process of learners to that of experts to determine to what extent their cognitive ‘scripts’ align. Without understanding test‐takers’ cognitive process, however, it is unclear what influences their responses. The objective of this study was to gather response process validity evidence by studying the cognitive process of test‐takers to determine whether the SCT tests CDM and what cognitive processes may influence SCT responses. Methods Cases from an SCT used in a national validation study were administered and semi‐structured cognitive interviews were conducted with ten residents and five staff surgeons. A retrospective verbal probing technique was used. Data was independently analysed and coded by two analysts. Themes were identified as factors that influence SCT responses during the cognitive interview. Results Cognitive interviews demonstrated variability in CDM among test‐takers. Consistent with dual process theory, test‐takers relied on scripts formed through past experiences, when available, to make decisions and used conscious deliberation in the absence of experience. However, test‐takers’ response process was also influenced by their comprehension of specific terms, desire for additional information, disagreement with the planned management, underlying knowledge gaps and desire to demonstrate confidence or humility. Conclusion The rationale behind SCT answers may be influenced by comprehension, underlying knowledge and social desirability in addition to formed scripts and/or conscious deliberation. Having test‐takers verbalise their rationale for responses provides a depth of assessment that is otherwise lost in the SCT’s current format. With the improved ability to standardise CDM assessment using the SCT, consideration of test‐makers improving the SCT construction process and combining the SCT question format with verbal responses may improve the use of the SCT for CDM assessment.

show abstract

Section: Discussionsupporting

confidence: 92%

How do cognitive processes influence script concordance test responses?

et al. 2020

View full text Add to dashboard Cite

show abstract

“…These strategies helped nurses gain autonomy in clinical judgment. Petrucci et al ( 25 ) reported a significant correlation between residents’ skillfulness and their ability to make accurate clinical judgments. Our participating nurses tried to provide patient care based on nursing models and theories and clinical practice guidelines.…”

Section: Discussionmentioning

confidence: 99%

Nurses’ Clinical Judgment Development: A Qualitative Research in Iran

Seidi¹,

Alhani²,

Salsali³

2015

Iran Red Crescent Med J

View full text Add to dashboard Cite

Background:Clinical judgment development is necessary because it leads to appropriate nursing diagnoses, clinical decision-making and health promotion.Objectives:In this study we explored the process of Iranian nurses’ development in clinical judgment.Patients and Methods:This qualitative study was conducted in 2013 at hospitals of Kurdistan University of Medical Sciences, located in the Sanandaj city of Iran. The data were collected based on semi-structured interviews and the study included 24 participants. Data analysis was carried out concurrently with data collection using the grounded theory method.Results:The study participants’ main concern was ‘being non-professional in clinical judgment’. In response to this concern, they were struggling for gaining professional autonomy, striving for integrating clinical judgment skills, scrambling to make effective educational interventions and striving for professional and inter professional collaboration in clinical judgment. The core category was ‘struggling for becoming professional in clinical judgment development’. When nurses were supported professionally, they were able to develop their professional clinical judgment.Conclusions:The findings of this study provided critical information about nurses’ professionalization in clinical judgment. Accordingly, the participants adopted different strategies to develop their clinical judgment ability. Integrating these strategies into nursing theory and clinical education can improve nurses’ clinical judgment ability.

show abstract

“…Previous studies have investigated aspects of SCT such as comparing the answer keys obtained from panels with different levels of expertise [ 24 ], optimizing the answer keys [ 25 ], improving the development of the scoring key [ 26 ], investigating the effect of variability within the reference panel [ 27 ], and validating the test in different clinical fields [ 1 ]-[ 14 ]. However, to our knowledge, no study had challenged the reference standard of SCT by evidence before our study.…”

Section: Discussionmentioning

confidence: 99%

Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?

et al. 2014

View full text Add to dashboard Cite

BackgroundWe aimed to compare the clinical judgments of a reference panel of emergency medicine academic physicians against evidence-based likelihood ratios (LRs) regarding the diagnostic value of selected clinical and paraclinical findings in the context of a script concordance test (SCT).FindingsA SCT with six scenarios and five questions per scenario was developed. Subsequently, 15 emergency medicine attending physicians (reference panel) took the test and their judgments regarding the diagnostic value of those findings for given diseases were recorded. The LRs of the same findings for the same diseases were extracted from a series of published systematic reviews. Then, the reference panel judgments were compared to evidence-based LRs. To investigate the test-retest reliability, five participants took the test one month later, and the correlation of their first and second judgments were quantified using Spearman rank-order coefficient.In 22 out of 30 (73.3%) findings, the expert judgments were significantly different from the LRs. The differences included overestimation (30%), underestimation (30%), and judging the diagnostic value in an opposite direction (13.3%). Moreover, the score of a hypothetical test-taker was calculated to be 21.73 out of 30 if his/her answers were based on evidence-based LRs.The test showed an acceptable test-retest reliability coefficient (Spearman coefficient: 0.83).ConclusionsAlthough SCT is an interesting test to evaluate clinical decision-making in emergency medicine, our results raise concerns regarding whether the judgments of an expert panel are sufficiently valid as the reference standard for this test.

show abstract

Assessing clinical judgment using the Script Concordance test: the importance of using specialty-specific experts to develop the scoring key

Cited by 15 publications

References 17 publications

How do cognitive processes influence script concordance test responses?

How do cognitive processes influence script concordance test responses?

Nurses’ Clinical Judgment Development: A Qualitative Research in Iran

Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?

Contact Info

Product

Resources

About