Determining the Accuracy of Crowdsourced Tweet Verification for Auroral Research

Abstract

The Aurorasaurus project harnesses volunteer crowdsourcing to identify sightings of an aurora (the "northern/southern lights") posted by citizen scientists on Twitter. Previous studies have demonstrated that aurora sightings can be mined from Twitter with the caveat that there is a large background level of non-sighting tweets, especially during periods of low auroral activity. Aurorasaurus attempts to mitigate this, and thus increase the quality of its Twitter sighting data, by using volunteers to sift through a pre-filtered list of geolocated tweets to verify real-time aurora sightings. In this study, the current implementation of this crowdsourced verification system, including the process of geolocating tweets, is described and its accuracy (which, overall, is found to be 68.4%) is determined. The findings suggest that citizen science volunteers are able to accurately filter out unrelated, spam-like, Twitter data but struggle when filtering out somewhat related, yet undesired, data. The citizen scientists particularly struggle with determining the real-time nature of the sightings, so care must be taken when relying on crowdsourced identification.

Review of six Japanese online citizen science projects with main indicators to encourage citizen motivation

“…This project is similar to a preceding project called Aurorasaurus, which had hosted 950 auroral pictures posted on Twitter till 2015; these have been placed on its official website [46]. Whereas Aurorasaurus accommodates English and can be found throughout the world [46,47], the Aurora 4D Project accommodates only Japanese. In addition, aurora is not frequently seen in Japan [31,33,48].…”

Aurorasaurus Database of Real‐Time, Crowd‐Sourced Aurora Data for Space Weather Research

“…Tweets that are up‐voted to be real‐time auroral sightings are classified as positive verified tweets highlighted by the blue frame. The orange frame shows the number of negatively verified tweets indicating that they were not real‐time auroral sightings or not actual auroral sightings at all and therefore, down‐voted by the community (Case, MacDonald, McCloat, et al, ). The total number of negatively verified tweets for both years are significantly larger compared to positive verified tweets, reflecting the noise levels inherent in the Twitter data.…”

“…Aurorasaurus data are composed of direct reports submitted to the project via its website (aurorasaurus.org) and iOS and Android apps and tweets that are mined from Twitter via keyword searching and geotagging (Case, MacDonald, McCloat, et al, 2016). Direct reports can either be positive or negative, corresponding to whether or not the observer saw the aurora.…”

“…The details of manual tweet verification are discussed in an earlier study (Case, MacDonald, McCloat, et al, 2016) based on the analysis of tweets collected during March and April 2015 that includes the period of St. Patrick's Day storm (Case, MacDonald, & Patel, 2015). The raw positively verified tweets are sifted through one at a time and they are divided into two major categories, valid or invalid.…”

“…The distribution of these categories for 2015 and 2016 data is shown in Figure . The description of each category can be found in the work of Case, MacDonald, McCloat, et al (). True and false positives (TP and FP) refer to positively verified tweets that are valid and invalid, respectively.…”