Intra-rater Kappa Accuracy of Prototype and ICD-10 Operational Criteria-Based Diagnoses for Mental Disorders: A Brief Report of a Cross-Sectional Study in an Outpatient Setting

Abstract

Background and ObjectivesThe use of “operational criteria” is a solution for low reliability, contrasting with a prototypical classification that is used in clinics. We aim to measure the reliability of prototypical and ICD-10 diagnoses.MethodsThis is a retrospective study, with a convenience sample of subjects treated in a university clinic. Residents reviewed their diagnosis using ICD-10 criteria, and Cohen's kappa statistic was performed on operational and prototype diagnoses.ResultsThree out of 30 residents participated, reviewing 146 subjects under their care. Diagnoses were grouped in eight classes: organic (diagnoses from F00 to F09), substance disorders (F10–F19), schizophrenia spectrum disorders (F20–F29), bipolar affective disorder (F30, F31, F34.0, F38.1), depression (F32, F33), anxiety-related disorders (F40–F49), personality disorders (F60–F69), and neurodevelopmental disorders (F70–F99). Overall, agreement was high [K = 0.77, 95% confidence interval (CI) = 0.69–0.85], with a lower agreement related to personality disorders (K = 0.58, 95% CI = 0.38–0.76) and higher with schizophrenia spectrum disorders (K = 0.91, 95% CI = 0.82–0.99).DiscussionUse of ICD-10 criteria did not significantly increase the number of diagnoses. It changed few diagnoses, implying that operational criteria were irrelevant to clinical opinion. This suggests that reliability among interviewers is more related to information gathering than diagnostic definitions. Also, it suggests an incorporation of diagnostic criteria according to training, which then became part of the clinician's prototypes. Residents should be trained in the use of diagnostic categories, but presence/absence checking is not needed to achieve operational compatible diagnoses.

Lost diagnoses? A multi-year trajectory of patients with childhood ADHD in the criminal justice system in Switzerland

“…Whether the diagnoses are lost because professionals arrive at different conclusions or whether it is related to the possible fluctuating nature of ADHD requires detailed investigation. Nonetheless, it is important to note that there are inherent limitations in the interrater reliability of diagnostic criteria, including those set forth in the ICD [57]. Achieving perfect interrater reliability is not feasible, and this reality must be factored into our considerations and analyses.…”

Operational criteria application does not change clinicians’ opinion on the diagnosis of mental disorder: a pre- and post-intervention validity study

“…However, it is not customary in daily practice to verify the presence or absence of diagnostic operational criteria, but rather to assign the code based on the identified prototype. IPUB’s outpatient functioning, Brazilian specialist training, and how prototypes are developed by residents were explained elsewhere ( 16 , 18 )⁠.…”

“…In the same way, clinicians were asked to review five patients' diagnoses, creating a ceiling effect for the number of possible differently observed and identified diagnoses by each subject and restraining the observable diagnostic variability. It is possible that if each clinician brought 10 reviewed diagnoses, the number of four-and three-digit diagnoses would have been greater, as it was in other studies (16), improving the ability to verify the checklist effect, although it is unlikely that the final list of large diagnostic groups would be different. The present paper also uses data generated during a psychiatry clinical course, implying some clinical inexperience from clinicians, and an unknown adherence to operational criteria application as requested.…”

“…However, SDIs are not practiced in clinical scenarios ( 14 )⁠, and operational criteria are neither observed as part of clinical practice ( 3 , 15 , 16 )⁠ nor described as how to be applied in the diagnostic process. Clinicians do not usually identify the disorder constructs described in operational criteria manuals, but rather disorder prototypes, which are nearer to clinicians’ reasoning ( 7 , 17 )⁠ and can be reasonably reliable between different clinicians ( 18 ).…”

“…A recent brief report tested the intra-rater kappa agreement of prototype and operational-based diagnosis and has shown high reliability ( 16 )⁠. However, that study had some limitations in checking the operational criteria power to change clinicians’ previous opinions, mainly related to the small number of clinician participants.…”