Mapping the plague through natural language processing

Abstract

Plague has caused three major pandemics with millions of casualties in the past centuries. There is a substantial amount of historical and modern primary and secondary literature about the spatial and temporal extent of epidemics, circumstances of transmission or symptoms and treatments. Many quantitative analyses rely on structured data, but the extraction of specific information such as the time and place of outbreaks is a tedious process. Machine learning algorithms for natural language processing (NLP) can potentially facilitate the establishment of datasets, but their use in plague research has not been explored much yet. We investigated the performance of five pre-trained NLP libraries (Google NLP, Stanford CoreNLP, spaCy, germaNER and Geoparser.io) for the extraction of location data from a German plague treatise published in 1908 compared to the gold standard of manual annotation. Of all tested algorithms, we found that Stanford CoreNLP had the best overall performance but spaCy showed the highest sensitivity. Moreover, we demonstrate how word associations can be extracted and displayed with simple text mining techniques in order to gain a quick insight into salient topics. Finally, we compared our newly digitised plague dataset to a re-digitised version of the famous Biraben plague list and update the spatio-temporal extent of the second pandemic plague mentions. We conclude that all NLP tools have their limitations, but they are potentially useful to accelerate the collection of data and the generation of a global plague outbreak database.

A Refined Phylochronology of the Second Plague Pandemic in Western Eurasia

“…The number of currently available genomes of the Second Pandemic surpasses a hundred (Table S3, S4, 31 ), but only a minority of them can be attributed to a specific epidemic event. In parallel, more existing datasets of documentarily reported plague outbreaks have been digitized [32][33][34] and historical reconstructions of plague outbreaks and waves, connecting them to hypothetical reservoirs in different regions of Eurasia, have been published by historians 3,[35][36][37][38][39][40][41] . However, despite these advances across fields of research, any attempts to associate ancient plague genomes or clades with documentary data remains significantly impaired by uncertainties pertaining to both palaeogenetics and history.…”