Using statistical and machine learning to help institutions detect suspicious access to electronic health records

Abstract

ObjectiveTo determine whether statistical and machine-learning methods, when applied to electronic health record (EHR) access data, could help identify suspicious (ie, potentially inappropriate) access to EHRs.MethodsFrom EHR access logs and other organizational data collected over a 2-month period, the authors extracted 26 features likely to be useful in detecting suspicious accesses. Selected events were marked as either suspicious or appropriate by privacy officers, and served as the gold standard set for model evaluation. The authors trained logistic regression (LR) and support vector machine (SVM) models on 10-fold cross-validation sets of 1291 labeled events. The authors evaluated the sensitivity of final models on an external set of 58 events that were identified as truly inappropriate and investigated independently from this study using standard operating procedures.ResultsThe area under the receiver operating characteristic curve of the models on the whole data set of 1291 events was 0.91 for LR, and 0.95 for SVM. The sensitivity of the baseline model on this set was 0.8. When the final models were evaluated on the set of 58 investigated events, all of which were determined as truly inappropriate, the sensitivity was 0 for the baseline method, 0.76 for LR, and 0.79 for SVM.LimitationsThe LR and SVM models may not generalize because of interinstitutional differences in organizational structures, applications, and workflows. Nevertheless, our approach for constructing the models using statistical and machine-learning techniques can be generalized. An important limitation is the relatively small sample used for the training set due to the effort required for its construction.ConclusionThe results suggest that statistical and machine-learning methods can play an important role in helping privacy officers detect suspicious accesses to EHRs.

Security and Privacy in the Medical Internet of Things: A Review

“…Over the past decades, many auditing methods have been presented. Several supervised machine learning approaches, such as logistic regression and support vector machine, have been applied to detect suspicious access [21]. However, relying too much on expected judgments and predefined tags restricts their large-scale promotion.…”

Electronic Health Record Logs Indicate That Physicians Split Time Evenly Between Seeing Patients And Desktop Medicine

“…Previous research has used access logs to examine how physicians use EHRs, [13][14][15][16][17][18] to develop local access policies, 19 and to detect suspicious access. 20 We report here on an innovative use of the access log to study primary care physician work effort.…”

“…Previous research has used access logs to examine how physicians use EHRs, [13][14][15][16][17][18] to develop local access policies, 19 and to detect suspicious access. 20 We report here on an innovative use of the access log to study primary care physician work effort.Similar to a customs agent stamping an international traveler's passport with the time and place of entry and exit, the EpicCare EHR maintains an access log that tracks many discrete, time-stamped actions associated with patient care. The log records the user, time of access, device from which the EHR was accessed, and section of the EHR section that was accessed (for example, a medication list or lab results).…”

Decide Now or Decide Later?

“…In doing so, the AUC indicates the agility of a classifier, where the "best" classifier is the one that maximizes this value. The AUC has been invoked as a common approach for assessing various classification models for information security, such as intrusion detection systems (e.g., [12]), malware detection (e.g., [11]), and auditing techniques for EMRs (e.g., [1]). We recognize the relevance of machine learning (for which AUC is a popular evaluation measure), for information security has been questioned [19].…”

“…Much of the work on access control has focused on the prospective decision making, but it has often been pointed out [15,20] that retrospective decision making, in which users beg for forgiveness rather than permission, has some significant advantages. In many applications: (1) it is difficult to determine what access a user requires in advance, (2) denying access to a user with a legitimate need could result in significant inconvenience, expense, or loss, (3) most users are responsible and can be trusted to access resources for legitimate reasons, and (4) accountability (such as disciplinary action) is effective in deterring abuses. An iconic example of such a situation is access to patient records in Electronic Medical Record (EMR) systems, where (1) hospital workflows are complex and commonly involve emergencies and unexpected events, (2) lack of timely access could result in the loss of a patient's life, (3) most healthcare providers are highly trained and ethical professionals, and (4) there are strong penalties for abuse.…”