Mining Archive.org’s Twitter Stream Grab for Pharmacovigilance Research Gold

Abstract

In the last few years Twitter has become an important resource for the identification of Adverse Drug Reactions (ADRs), monitoring flu trends, and other pharmacovigilance and general research applications. Most researchers spend their time crawling Twitter, buying expensive pre-mined datasets, or tediously and slowly building datasets using the limited Twitter API. However, there are a large number of datasets that are publicly available to researchers which are underutilized or unused. In this work, we demonstrate how we mined over 9.4 billion Tweets from archive.org's Twitter stream grab using a drug-term dictionary and plenty of computing power. Knowing that not everything that shines is gold, we used pre-existing drug-related datasets to build machine learning models to filter our findings for relevance. In this work we present our methodology and the 3,346,758 identified tweets for public use in future research.

Social Media Mining Toolkit (SMMT)

“…https://doi.org/10.5808/GI.2020.18.2.e16 uses like Pharmacovigilance [11] among others. This initial version release of SMMT will continue growing with additional tools being developed for platforms like Reddit, Dark Web forums, and other social media data sources.…”

“…The tools part of SMMT allow users to simplify their research workflows and to focus on determining which data they want to use and the analyses they want to perform, rather than deciphering how to acquire the data. While most cutting-edge and near real-time research will be done pulling tweets from the Twitter API stream, there are countless datasets available for historical research, from large general purpose databases like the Internet Archive’s Twitter Stream Grab dataset [ 23 ], which consists of data from 2014 to 2019, to more specialized and pre-curated datasets for uses like Pharmacovigilance [ 11 ] among others. This initial version release of SMMT will continue growing with additional tools being developed for platforms like Reddit, Dark Web forums, and other social media data sources.…”

“…In an attempt to shift the biomedical community into better practices for research transparency and reproducibility, we introduce the Social Media Mining Toolkit (SMMT), a suite of tools aimed to encapsulate the cumbersome details of acquiring, preprocessing, annotating, and standardizing social media data. The need for a toolkit like SMMT arose from our work using Twitter data for the characterization of disease transmission during natural disasters [10] and mining large-scale repositories for drug usage related tweets for pharmacovigilance purposes [11]. We originally wanted to use other researcher's tools and surprisingly we found very little code available with the majority outdated and non-functioning.…”

Characterizing drug mentions in COVID-19 Twitter Chatter

“…We utilized the largest available Covid-19 dataset (Banda et al, 2020) curated using a Social Media Mining Toolkit (SMMT) (Tekumalla and Banda, 2020b). Version 15 of the Covid-19 dataset was utilized for our experiments since it was the latest released version at the time of experiments.…”

“…This dataset consists of tweets related to COVID-19 from January 1, 2020 to June 20, 2020. We automatically tagged ~424 million tweets using a drug dictionary compiled from RxNorm (National Library of Medicine, 2008) with 19,643 terms and validated in (Tekumalla et al, 2020) and (Tekumalla and Banda, 2020a).…”