An intervention to improve the interrater reliability of clinical EEG interpretations

Abstract

Several studies have noted modest interrater reliability of clinical electroencephalogram (EEG) interpretations. Moreover, no study to date has investigated a means to improve the observed interrater agreement. The purpose of the present study was to examine (i) the interrater reliability of EEG interpretations among three raters (two psychiatrists and one pediatrician); and (ii) how to improve the reliability by establishing a consensus guideline for EEGinterpretation. Three raters, two psychiatrists and a pediatrician, interpreted 100 consecutive EEG recorded at Tajimi General Hospital. After discussing the results of the first trial, the raters established a consensus guideline for EEG interpretation. They then interpreted 50 consecutive EEG recorded at Nagoya City University Hospital following this guideline. Kappa for global judgment of EEG abnormality in three grades (abnormal/borderline/normal) was 0.42 on the first and 0.63 on the second trial. Kappa significantly improved by using the guideline ( P = 0.004). It is suggested that discussing and establishing the consensus guideline among the raters offers a feasible method to improve interrater reliability in clinical EEG interpretations.

Computer assisted interpretation of the human EEG

“…An important finding in (Halford et al, 2011) was the relatively large inter-rater variance (also reported by others, e.g. (Azuma et al, 2003;Benbadis et al, 2009)), which points to an even greater need to use a single, commonly accepted dataset for benchmarking purposes.…”

“…Although this approach still holds the highest specificity and sensitivity for finding abnormalities, it is time consuming and requires extensive training. Apart from this, visual analysis is susceptible to reviewer bias (Haut et al, 2002;Benbadis et al, 2009;Gerber et al, 2008;Azuma et al, 2003). Automated quantification techniques reduce the time required for reviewing and improve consistency in reporting.…”

“…The la er part can lead to a high degree of variability and inconsistency between reviewers (Haut et al, 2002;Benbadis et al, 2009;Gerber et al, 2008;Azuma et al, 2003), and given that the findings are not noted down using a set category of outcomes or that no commonly accepted guidelines exist for describing some properties, reports become di icult to query and compare to the findings of other clinicians. In recent work by Beniczky et al (2013), the authors describe a set of guidelines and definitions (including the reporting of common background properties) that is being constructed as part of a pan-European project with the goal of providing more consistency and structure for the reports in clinical EEG reviews (Beniczky et al, 2013;Aurlien et al, 2004Aurlien et al, , 2007.…”

“…Unfortunately, various studies have shown that a large inter-and intra-observer variability still exists between reviewers. Depending on the reported feature or decision outcome, the inter-rater agreement (Kappa coe icients) range from slight (0.09) to substantial (0.94) (Haut et al, 2002;Benbadis et al, 2009;Gerber et al, 2008;Azuma et al, 2003). One of the main reasons for this is a lack of consistency in describing the properties accurately.…”

“…One of the main reasons for this is a lack of consistency in describing the properties accurately. Azuma et al (2003) showed that by conforming to a set of general guidelines, inter-rater variability could be reduced significantly (Azuma et al, 2003). For many of the EEG properties mentioned however, formal guidelines do not exist or fall short of being used.…”