Similarity-Based Unsupervised Spelling Correction Using BioWordVec: Development and Usability Study of Bacterial Culture and Antimicrobial Susceptibility Reports (Preprint)

Abstract

BACKGROUND Existing bacterial culture test results for infectious diseases are written in unrefined text, resulting in many problems including typographical errors and stop words. Effective spelling correction processes are needed to ensure the accuracy and reliability of data for the study of infectious diseases, including medical terminology extraction. If a dictionary is established, spelling algorithms using edit distance are efficient. However, in the absence of dictionaries, traditional spelling correction algorithms that utilize only edit distances have limitations. OBJECTIVE In this research, we proposed a similarity-based spelling correction algorithm using pre-trained word embedding with the BioWordVec technique. This method uses a character-level N-grams-based distributed representation through unsupervised learning rather than the existing rule-based method. In other words, we propose a framework that detects and corrects typographical errors when a dictionary is not in place. METHODS For detected typographical errors not mapped to SNOMED clinical terms, a correction candidate group with high similarity considering the edit distance was generated using pre-trained word embedding from the clinical database. From the embedding matrix in which the vocabulary is arranged in descending order according to frequency, the grid search is used to search for candidate groups of similar words. Then, the correction candidate words are ranked in consideration of the frequency of the words, and the typos are finally corrected according to the ranking. RESULTS Bacteria identification words were extracted from 27,544 bacteria culture reports, and 16 types of 914 spelling errors were found. The similarity-based spelling correction algorithm using BioWordVec proposed in this research corrected 12 types of typographical errors and showed very high performance in correcting 99.45% of all spelling errors. CONCLUSIONS This tool corrected spelling errors effectively in the absence of a dictionary based on bacterial identification words in the bacteria culture reports. This method will help build a high-quality refined database of vast text data for electronic health records.