A declarative query processing system for nowcasting

Abstract

Nowcasting is the practice of using social media data to quantify ongoing real-world phenomena. It has been used by researchers to measure flu activity, unemployment behavior, and more. However, the typical nowcasting workflow requires either slow and tedious manual searching of relevant social media messages or automated statistical approaches that are prone to spurious and low-quality results. In this paper, we propose a method for declaratively specifying a nowcasting model; this method involves processing a user query over a very large social media database, which can take hours. Due to the human-in-the-loop nature of constructing nowcasting models, slow runtimes place an extreme burden on the user. Thus we also propose a novel set of query optimization techniques, which allow users to quickly construct nowcasting models over very large datasets. Further, we propose a novel query quality alarm that helps users estimate phenomena even when historical ground truth data is not available. These contributions allow us to build a declarative nowcasting data management system, RaccoonDB, which yields high-quality results in interactive time. We evaluate RaccoonDB using 40 billion tweets collected over five years. We show that our automated system saves work over traditional manual approaches while improving result quality-57% more accurate in our user study-and that its query optimizations yield a 424x speedup, allowing it to process queries 123x faster than a 300-core Spark cluster, using only 10% of the computational resources.

Database Learning

“…In this section, we present the mathematical details we have omitted in the main body of this paper. First, we describe the analytics expression for the double-integrals in Equation (10). Second, we extend the result of Section 4 to categorical attributes.…”

“…In Figure 1, we visualize this idea using a real-world Twitter dataset [9,10]. Here, DBL learns a model for the number of occurrences of certain word patterns (known as n-grams, e.g., "bought a car") in tweets.…”

Constraint-based explanation and repair of filter-based transformations

“…User Constraints -This is a collection of constraints that define the user's desired output dataset at query cycle i: When specifying density estimation signal s, users have two options: for continuous values, users provide a signal. In past work [9], we provided a web-based tool for "drawing" signals; something similar could be used to assist users. For multinomial values, users provide a set of desired counts or percentages (e.g., gender : 60% male, 40% female).…”

“…Constraint-Based Querying -Constraint-based querying systems, such as ConQueSt [12] and our work with RaccoonDB [8,9,10], provide custom query languages for defining constraints, which are used to find relevant results. Our work and RaccoonDB both use code-and signal-based constraints to find a small subset of items in a large pool of candidates.…”