Text Mining the History of Medicine

Abstract

Historical text archives constitute a rich and diverse source of information, which is becoming increasingly readily accessible, due to large-scale digitisation efforts. However, it can be difficult for researchers to explore and search such large volumes of data in an efficient manner. Text mining (TM) methods can help, through their ability to recognise various types of semantic information automatically, e.g., instances of concepts (places, medical conditions, drugs, etc.), synonyms/variant forms of concepts, and relationships holding between concepts (which drugs are used to treat which medical conditions, etc.). TM analysis allows search systems to incorporate functionality such as automatic suggestions of synonyms of user-entered query terms, exploration of different concepts mentioned within search results or isolation of documents in which concepts are related in specific ways. However, applying TM methods to historical text can be challenging, according to differences and evolutions in vocabulary, terminology, language structure and style, compared to more modern text. In this article, we present our efforts to overcome the various challenges faced in the semantic analysis of published historical medical text dating back to the mid 19th century. Firstly, we used evidence from diverse historical medical documents from different periods to develop new resources that provide accounts of the multiple, evolving ways in which concepts, their variants and relationships amongst them may be expressed. These resources were employed to support the development of a modular processing pipeline of TM tools for the robust detection of semantic information in historical medical documents with varying characteristics. We applied the pipeline to two large-scale medical document archives covering wide temporal ranges as the basis for the development of a publicly accessible semantically-oriented search system. The novel resources are available for research purposes, while the processing pipeline and its modules may be used and configured within the Argo TM platform.

A comprehensive and quantitative comparison of text-mining in 15 million full-text articles versus their corresponding abstracts

“…Some PDF files without texts are scans of the original article (point 1). We did not attempt to make an optical character recognition conversion (OCR) as the old typesetting fonts often are less compatible with present day OCR programs, and this can lead to text recognition errors [ 28 , 29 ]. For any discarded document, we still used the meta-data to calculate summary statistics.…”

An open-source tool for the transcription of paper-spreadsheet data: Code and supplemental materials available online: Https://github.com/deskool/images to spreadsheets

“…All data must be digital before it can be processed; but not all data that requires processing is in a usable digital format. While it is rare for data scientists to interact with non-digital data, many clinicians [1], historians [2], educators and field researchers [3] still regularly capture or must work with historical archives of paper-based spreadsheet data. For small datasets, manual transcription of these records is feasible, but as data requirements grow, researchers and professionals are required to invest considerable resources to transcribe their paper-based data into digital form [4], [5].…”

“…Data in the educational and healthcare domains, for instance, often contain sensitive personal information requiring specialized authorization to share with third parties. These constraints make transcription arduous and costly [1].…”

“…This may be due to the fact that many of the most effective machine learning algorithms are supervised, requiring large fully-annotated data-sets for training. To our knowledge, such a data-set for spreadsheets is not yet available within in the public domain [1].…”

Text mining resources for the life sciences

“…A recent study by Thompson et al (115) may serve as an example of combining all of the elements of text mining within a single project. The goal was to analyse medical vocabulary from a historical perspective, observing how certain terms and concepts appear, transform and wither across the years.…”