Validation of data submitted by the treating surgeon in the Victorian Audit of Surgical Mortality

Abstract

Background: The Victorian Audit of Surgical Mortality (VASM) seeks to peer review all deaths associated with surgical care in Victoria, Australia. The effectiveness of the VASM as an educational and quality improvement tool is dependent on the accuracy of source data it receives. We aimed to examine the accuracy and quality of source data provided by the treating surgeon for peer review, and the inter-rater concordance level between the external validator findings and the treating surgeon. Methods: Of the 629 cases that completed the VASM audit second-line peer review process over a 4-year period (from 1 July 2012 to 30 June 2016), a total of 32 (5%) were randomly selected for the external validation process. The blinded external validator was impartial to the VASM audit, and was provided only de-identified patient medical records. The analysis for the checked and validated data points and their concordance was determined using Gwet's agreement coefficient, which provides a stable inter-rater reliability coefficient not affected by prevalence and marginal probability. Results: The inter-rater concordance analysis suggested that there is a high level of agreement (82.9% overall) between the treating surgeon and external validator. The use of thromboembolism deterrent stockings was the only variable where agreement was poor (52.4%) with a Gwet score of 0.10 (−0.40 to 0.60). Conclusion: The inter-rater concordance analysis results support the validity of the VASM process, which is dependent on the accuracy of data submitted by the treating surgeon.

Systemic predictors of adverse events in a national surgical mortality audit: analysis of peer‐review data from Australia and New Zealand Audit of Surgical Mortality

“…We sought to minimize this by reserving analysis for the highest level of seriousness rather than possibly more contentious areas of consideration and areas of concern. A Victorian study also showed good concordance between the more serious ANZASM second‐line assessments, from which it could be inferred that there would be similarly low inter‐rater variability in FLA judgements.…”

Homogeneity score test of AC1 statistics and estimation of common AC1 in multiple or stratified inter-rater agreement studies

“…Gwet [24] and Shankar and Bangdiwala [28] compared some measures and showed that AC 1 has better properties than other kappa-type measures. In addition, AC 1 has been utilized in the field of medical research over the past decade [29][30][31][32][33][34][35]. However statistical inference procedures of AC 1 have not been discussed sufficiently.…”

“…Shanker and Bangdiwala compared Cohen's κ, Scott's π, Prevalence Adjusted Bias Adjusted Kappa (PABAK) [26], AC 1 and B-statistic [27], which is not a kappa-type chance-corrected measure, in the case of two raters and binary outcomes and showed that AC 1 has better properties than other kappa-type measures [28]. In addition, AC 1 has been utilized in the field of medical research over the past decade [29][30][31][32][33][34][35]. Therefore, in this study we have limited our discussion to AC 1 in the case of two raters with binary outcomes.…”

Assessment of accuracy of Australian health service death data: implications for the audits of surgical mortality

“…Staff without appropriate training may not understand either the actual data they are required to input or the significance of the data. The ANZASM have been collecting surgical death data for over a decade and regularly review their own data collection, management and transcription to ensure accuracy . The process is dependent on the initial notification of death which most commonly comes from hospitals and health services.…”

“…Under‐reported death data are not uncommon in healthcare settings; however, identification of under‐reporting to a quality assurance activity raises concerns about whether all deaths are being reported by all healthcare services. The individual audits of surgical deaths regularly review their own data and results, but the quality of hospital source notification data has never been assessed.…”