Dynamic memory allocation policies for postings in real-time Twitter search

Abstract

We explore a real-time Twitter search application where tweets are arriving at a rate of several thousands per second. Real-time search demands that they be indexed and searchable immediately, which leads to a number of implementation challenges. In this paper, we focus on one aspect: dynamic postings allocation policies for index structures that are completely held in main memory. The core issue can be characterized as a "Goldilocks Problem". Because memory remains today a scare resource, an allocation policy that is too aggressive leads to inefficient utilization, while a policy that is too conservative is slow and leads to fragmented postings lists. We present a dynamic postings allocation policy that allocates memory in increasingly-larger "slices" from a small number of large, fixed pools of memory. With an analytical model and experiments, we explore different settings that balance time (query evaluation speed) and space (memory utilization).

Fast candidate generation for real-time tweet search with bloom filter chains

“…Note that Earlybird's design can be extended to an arbitrary number of pools and pool sizes; we can create more memory pools or use different slice sizes-this is further explored in Asadi et al [2013] A particular instantiation of this general strategy can be described by P = {P 1 , P 2 , . .…”

The Feasibility of Brute Force Scans for Real-Time Tweet Search

“…In real-time search, there is another important advantage: users most often care only about the latest results. With an inverted index, it is desirable to traverse postings lists "backwards" (from most recent) and early exit when enough results have been accumulated [2]. Most systems are not designed this way, which foregoes optimization opportunities; adapting traditional query evaluation algorithms to operate in this manner is non-trivial.…”

The Performance Envelope of Inverted Indexing on Modern Hardware

“…Despite alternative approaches based on approximate nearest-neighbor search [2,9], inverted indexes-in combination with modern query evaluation algorithms such as block-max Wand [5]-remain the standard by which other retrieval techniques are judged. In this paper, we focus on inverted indexing applied to static document collections and explicitly leave aside the socalled real-time indexing problem, where high velocity document streams need to be ingested and made immediately searchable; such a scenario calls for different techniques than when working with static collections [3,1,12].…”