Evaluating current automatic de-identification methods with Veteran’s health administration clinical documents

Abstract

BackgroundThe increased use and adoption of Electronic Health Records (EHR) causes a tremendous growth in digital information useful for clinicians, researchers and many other operational purposes. However, this information is rich in Protected Health Information (PHI), which severely restricts its access and possible uses. A number of investigators have developed methods for automatically de-identifying EHR documents by removing PHI, as specified in the Health Insurance Portability and Accountability Act “Safe Harbor” method.This study focuses on the evaluation of existing automated text de-identification methods and tools, as applied to Veterans Health Administration (VHA) clinical documents, to assess which methods perform better with each category of PHI found in our clinical notes; and when new methods are needed to improve performance.MethodsWe installed and evaluated five text de-identification systems “out-of-the-box” using a corpus of VHA clinical documents. The systems based on machine learning methods were trained with the 2006 i2b2 de-identification corpora and evaluated with our VHA corpus, and also evaluated with a ten-fold cross-validation experiment using our VHA corpus. We counted exact, partial, and fully contained matches with reference annotations, considering each PHI type separately, or only one unique ‘PHI’ category. Performance of the systems was assessed using recall (equivalent to sensitivity) and precision (equivalent to positive predictive value) metrics, as well as the F2-measure.ResultsOverall, systems based on rules and pattern matching achieved better recall, and precision was always better with systems based on machine learning approaches. The highest “out-of-the-box” F2-measure was 67% for partial matches; the best precision and recall were 95% and 78%, respectively. Finally, the ten-fold cross validation experiment allowed for an increase of the F2-measure to 79% with partial matches.ConclusionsThe “out-of-the-box” evaluation of text de-identification systems provided us with compelling insight about the best methods for de-identification of VHA clinical documents. The errors analysis demonstrated an important need for customization to PHI formats specific to VHA documents. This study informed the planning and development of a “best-of-breed” automatic de-identification application for VHA clinical text.

An HL7 Data Pseudonymization Pipeline

“…As such, in our current evaluation we could not thoroughly evaluate our 'Text De-identifier' module. Due to the various complexities involved with de-identifying free text, this module may need some improvement when working with narrative, free-text reports; perhaps reusing some of the techniques discussed by other researchers [15,18] may help achieve a high degree of accuracy. A third limitation is that for zip codes, we retained only the first three characters and replaced the rest with zeros; however, for zip code de-identification to be fully compliant with HIPAA regulations, it needs to meet additional criteria where according to the current publicly available data from the Bureau of the Census, the geographic unit formed by combining all zip codes with the same three initial digits should contain more than 20,000 people.…”

“…al. [15] explored various data de-identification techniques for clinical documents within the VA, while Gupta et. al.…”

“…For our current work we developed our own HL7 Parser since we needed more control over the parsing process, although there are several APIs that can be used for this purpose, such as HAPI [13] and NHapi [14]. On the other hand, data deidentification techniques have been explored extensively by various researchers in the past [15][16][17]. For instance, Ferrández et.…”

Generating features for named entity recognition by learning prototypes in semantic space: The case of de-identifying health records

“…The ability to specify the optimization objective in terms of different F-scores is an advantage, as it gives some amount of control over the precision-recall trade-off. In the case of deidentification, for instance, recall is arguably of more importance than precision, as argued in [35], where F 2 -score was used to evaluate various deidentification systems.…”

Clinical Data Reuse or Secondary Use: Current Status and Potential Future Progress

“…Methods are usually based on pattern matching and dictionaries, or on machine learning algorithms. Some are more generalizable than others, and certain methods perform better with some types of PHI than others [71,72]. Recent examples such as MIST [73], BoB [74], Anonym [75], and several systems developed for the i2b2 NLP challenges [76,77], allow for good accuracy and very limited impact on clinical information.…”