Interrater Reliability among Epilepsy Centers: Multicenter Study of Epilepsy Surgery

Abstract

Summary:Purpose: To measure the interrater reliability of presurgical testing and surgical decisions among epilepsy centers.Methods: Seven centers participating in an ongoing, prospective multicenter study of resective epilepsy surgery agreed to conform to a detailed protocol regarding presurgical evaluation and surgery. To assess quality assurance, each center independently reviewed 21 randomly selected surgical cases for preoperative study lateralization and localization, and surgical decisions. Interrater reliability was assessed by using intraclass correlation coefficients (ICCs), validated for use with multiple raters, and calculated in a two-way random model based on absolute agreement.Results: Agreement for ICC values: Ն0.75, excellent; 0.60-0.74, good; 0.40-0.59, fair; Յ0.39, poor. One center was excluded for missing data. Agreement was excellent for extracranial EEG lateralization (0.8039), magnetic resonance imaging (MRI) lateralization (0.9521) and localization (0.9130), Wada lateralization (0.9453), and intracranial EEG localization (0.7905). Agreement was good for extracranial EEG localization (0.7384) and neuropsychological testing lateralization (0.7178) and localization (0.6891). Consensus about the decision to perform intracranial monitoring was fair (0.5397), in part reflecting one center's tendency toward intracranial monitoring. Overall agreements on whether to perform surgery (0.8311) and specific surgery recommended (0.8164) were excellent.Conclusions: High interrater reliability among six epilepsy centers was present for interpretation of most components of presurgical testing. Although consensus for the decision to perform intracranial monitoring was only fair, agreements for the ultimate decision about resective surgery and specific choice of resection were excellent. We believe that this study demonstrates the feasibility of implementing multicenter protocols for neurologic management, especially those involving localization, as well as protocols combining study results with clinical decision making.

Interrater reliability of EEG-video monitoring

“…9 The coefficients for IRR can vary dramatically across different fields. As a reference point, one study revealed an IRR of 0.83 between epilepsy centers on whether to perform epilepsy surgery, 10 and the IRR between sleep centers for scoring 5 different sleep stages was 0.68. 11 It is well known that the range of values is constrained by the margins.…”

Neuropsychological Evaluation in Epilepsy Surgery – A Cross-Cultural Perspective

“…However, the demonstrated ability of the scale to relatively accurately differentiate between the LTL and RTL subgroups as reported above, may suggest that factors other than the psychometric properties of the scale are at work in influencing the performance of patients in this sample. On the other hand, the amount of agreement between the neuropsychological data and other evaluation modalities was at best 'fair' (Haut, et al, 2002) and limited to temporal lobe areas. Studies that compare and contrast the efficacy of different evaluation modalities in documenting dysfunction in epilepsy patients at the presurgical workup are quite rare (e.g., Akanuma, et al, 2003;Haut, et al, 2002).…”

“…On the other hand, the amount of agreement between the neuropsychological data and other evaluation modalities was at best 'fair' (Haut, et al, 2002) and limited to temporal lobe areas. Studies that compare and contrast the efficacy of different evaluation modalities in documenting dysfunction in epilepsy patients at the presurgical workup are quite rare (e.g., Akanuma, et al, 2003;Haut, et al, 2002). Such comparison studies can provide a gold standard against which neuropsychological evaluation tools can be calibrated.…”

Computer assisted interpretation of the human EEG

“…The la er part can lead to a high degree of variability and inconsistency between reviewers (Haut et al, 2002;Benbadis et al, 2009;Gerber et al, 2008;Azuma et al, 2003), and given that the findings are not noted down using a set category of outcomes or that no commonly accepted guidelines exist for describing some properties, reports become di icult to query and compare to the findings of other clinicians. In recent work by Beniczky et al (2013), the authors describe a set of guidelines and definitions (including the reporting of common background properties) that is being constructed as part of a pan-European project with the goal of providing more consistency and structure for the reports in clinical EEG reviews (Beniczky et al, 2013;Aurlien et al, 2004Aurlien et al, , 2007.…”

“…Unfortunately, various studies have shown that a large inter-and intra-observer variability still exists between reviewers. Depending on the reported feature or decision outcome, the inter-rater agreement (Kappa coe icients) range from slight (0.09) to substantial (0.94) (Haut et al, 2002;Benbadis et al, 2009;Gerber et al, 2008;Azuma et al, 2003). One of the main reasons for this is a lack of consistency in describing the properties accurately.…”