Crowdsourcing Twitter annotations to identify first-hand experiences of prescription drug use

Abstract

Self-reported patient data has been shown to be a valuable knowledge source for post-market pharmacovigilance. In this paper we propose using the popular micro-blogging service Twitter to gather evidence about adverse drug reactions (ADRs) after firstly having identified micro-blog messages (also know as "tweets") that report first-hand experience. In order to achieve this goal we explore machine learning with data crowdsourced from laymen annotators. With the help of lay annotators recruited from CrowdFlower we manually annotated 1548 tweets containing keywords related to two kinds of drugs: SSRIs (eg. Paroxetine), and cognitive enhancers (eg. Ritalin). Our results show that inter-annotator agreement (Fleiss' kappa) for crowdsourcing ranks in moderate agreement with a pair of experienced annotators (Spearman's Rho=0.471). We utilized the gold standard annotations from CrowdFlower for automatically training a range of supervised machine learning models to recognize first-hand experience. F-Score values are reported for 6 of these techniques with the Bayesian Generalized Linear Model being the best (F-Score=0.64 and Informedness=0.43) when combined with a selected set of features obtained by using information gain criteria.

Detecting Personal Medication Intake in Twitter: An Annotated Corpus and Baseline Classification System

“…The study by Alvaro et al (2015) is perhaps the most closely related work to ours. The authors annotated 1,548 tweets for whether they contain "first-hand experiences" of adverse drug reactions (ADRs) to prescription medications, and they used this annotated data in a supervised classification framework aimed at automatically identifying tweets that report personal usage.…”

Research data management in the age of big data: Roles and opportunities for librarians

“…New types of data arising out of the internet, such as social media data, also have sometimes surprising research potential. Researchers have used social media sites like Twitter and Facebook for unexpected research purposes, like pharmacovigilance [1,4] and disease surveillance [5,16]. Not only do we live in an age of big data, but much of that data is freely and widely available.…”

Discovering explanatory models to identify relevant tweets on Zika

“…Multiple studies have used Twitter for exploration of public health issues [6], [7], [8]. One study focused on the spread of influenza from November 2008 to June 2010 and collected 300 million tweets [6].…”

“…Alvaro et al [7] obtained a random sample of tweets over a 12 month period to analyse first-hand experience with selective serotonin reuptake inhibitors or cognitive enhancers. The ground truth consisted of 100 annotated tweets for 15 categories which were then compared to crowd sourced annotators by calculating Kappa values for each of the categories.…”