An efficient data preprocessing approach for large scale medical data mining

Abstract

BACKGROUND: The size of medical datasets is usually very large, which directly affects the computational cost of the data mining process. Instance selection is a data preprocessing step in the knowledge discovery process, which can be employed to reduce storage requirements while also maintaining the mining quality. This process aims to filter out outliers (or noisy data) from a given (training) dataset. However, when the dataset is very large in size, more time is required to accomplish the instance selection task. OBJECTIVE: In this paper, we introduce an efficient data preprocessing approach (EDP), which is composed of two steps. The first step is based on training a model over a small amount of training data after preforming instance selection. The model is then used to identify the rest of the large amount of training data. METHODS: Experiments are conducted based on two medical datasets for breast cancer and protein homology prediction problems that contain over 100000 data samples. In addition, three well-known instance selection algorithms are used, IB3, DROP3, and genetic algorithms. On the other hand, three popular classification techniques are used to construct the learning models for comparison, namely the CART decision tree, k-nearest neighbor (k-NN), and support vector machine (SVM). RESULTS:The results show that our proposed approach not only reduces the computational cost by nearly a factor of two or three over three other state-of-the-art algorithms, but also maintains the final classification accuracy. CONCLUSIONS: To perform instance selection over large scale medical datasets, it requires a large computational cost to directly execute existing instance selection algorithms. Our proposed EDP approach solves this problem by training a learning model to recognize good and noisy data. To consider both computational complexity and final classification accuracy, the proposed EDP has been demonstrated its efficiency and effectiveness in the large scale instance selection problem.

Ubiquitous and smart healthcare monitoring frameworks based on machine learning: A comprehensive review

“…Data preprocessing can also be employed for the purpose of reducing storage requirements and maintaining mining quality [97] . The authors [97] applied 3 instance selection algorithms for data preprocessing including genetic algorithms and evaluated using ML models: CART decision trees, K-NN, and SVM. Irrelevant or redundant features in health care data seriously affect subsequent model training and classification accuracy.…”

An online intelligent electronic medical record system via speech recognition

“…It is also an inevitable demand for the global medical and health industry information application in clinics. 4–7…”

“…1–4 In addition, the information inputs of EMR recordings can also take large time consumption, which affects the communications between doctors and patients for the detail information of patients and slows down the treatment process to some extent. 4–6…”

“…This homogenization of medical records has led to a large number of incorrect records, which seriously affects patient care. [1][2][3][4] In addition, the information inputs of EMR recordings can also take large time consumption, which affects the communications between doctors and patients for the detail information of patients and slows down the treatment process to some extent. [4][5][6] The two problems of EMR mentioned above are explained in detail below.…”

Big data and artificial intelligence (AI) methodologies for computer-aided drug design (CADD)

“…Successful applications in drug design, discovery and development can be achieved only when effective computational methods and tools are provided with accurate and reliable pre-processed data [ 8 , 9 ]. Hereafter, big data and artificial intelligence (AI) approaches to data pre-processing [ 10 ], modeling [ 11 , 12 ] and representative applications in drug design and discovery will be introduced.…”