Incident reporting systems have been widely adopted to collect information about patient safety incidents. The value of incident reports often lies in the free-text section, but this is often inadequately processed. We developed a novel scoring system that uses machine learning to assess the severity of an incident using the semantic characteristics of text in incident reports. We retrospectively analyzed free-text data extracted from incident reports from between January 2012 and September 2021 at Nagoya University Hospital, Aichi, Japan. The sample was randomly allocated into a training dataset and validation dataset. Morphological analysis was used to segment terms using the training dataset. We calculated a severity term score, a severity report score and severity group score, by the report volume size. We then compared these with conventional severity classifications by patient safety experts and reporters. We allocated 96,082 incident reports into two groups. From the 48,041 reports in the training dataset, 1,802 severity term scores were calculated. There was a significant difference in severity report score between reports categorized as severe and not severe by experts (95% confidence interval [CI] − 0.83 to − 0.80, p < 0.001). Severity group scores were positively associated with severity ratings from experts and reporters, with correlation coefficients of 0.73 (95% CI 0.63–0.80, p < 0.001) and 0.79 (95% CI 0.71–0.85, p < 0.001) for all departments. Our severity scoring system using machine learning could contribute to improved organizational patient safety.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.