Assessing Diagnostic Expertise of Counselors Using the Cochran–Weiss–Shanteau (CWS) Index

Abstract

Counseling studies have shown that increasing experience is not always associated with better judgments. However, in such studies performance is assessed against external criteria, which may lack validity. The authors applied the Cochran–Weiss–Shanteau (CWS) index, which assesses the ability to consistently discriminate. Results showed that novice counselors performed almost on the same level as very experienced counselors. The authors thus replicated earlier findings with a novel approach: applying an internal coherence criterion.

Expertise in Clinical Psychology. The Effects of University Training and Practical Experience on Expertise in Clinical Psychology

“…The experienced psychotherapists did not diagnose significantly more accurately than novices, which was a disappointing result. Similar negative effects were found regarding the diagnostic competence of experienced counselors (Witteman et al, 2012). The effects of years of experience on therapy outcome are unclear, effect sizes range from small negative effects to large positive effects (e.g., Beutler et al, 2004).…”

Using mutually validated memories of experts for case-based knowledge systems

“…In this model we have taken this effect into consideration, which could further be examined by the proposed knowledge consistency of the KC index. This index follows the spirit of the CWS index [81,69] and takes the measures of both individual and group consistencies into account, which can straightforwardly be used to evaluate and assign suitable weights to the experts in the aggregation process and help to obtain a more appropriate and realistic group consensus. As discussed above, the mutual validation between the set of the parameters of the membership functions associated with labels by experts and the evaluation of the experts' weights can be manifested in terms of the KC index.…”

“…In view of the shortcomings, Ou et al [55] proposed the validation techniques of decision tree classification and formal concept analysis, while and Herrera-Viedma et al [30] examined the group consistency with fuzzy preference relations. However, these models are not sound enough to validate knowledge quality [81,69], and thus a novel fuzzy MCGDM model, integrating the two aforementioned matrices of linguistic judgment and fuzzy performance, is proposed in this work to enhance the validation of knowledge quality. In this step, we validate an expert's knowledge consistency in terms of the matrices of fuzzy performance F e and linguistic judgment L e .…”

“…In this study approach for the evaluation of experts is proposed, in which a CWS (Cochran; Weiss; Shanteau) index combining discrimination and consistency measures provides insights into identifying experts [81,69].…”

The Conceptual Framework of Thematic Mapping in Case Conceptualization

“…After all, trained and experienced professionals are expected to be better equipped than the lay public to make accurate and informed clinical judgments. However, as a reminder, research shows that education, training, and clinical experience only modestly improves clinical judgment (Lichtenberg, ; Pilipis, ; Ridley et al., ; Spengler et al., ; Witteman, Weiss, & Metzmacher, ). In light of this state of affairs, accurate case formulation demands that clinicians adopt an attitude of humility along with their clinical judgment.…”