Indexing Boolean expressions

Abstract

We consider the problem of efficiently indexing Disjunctive Normal Form (DNF) and Conjunctive Normal Form (CNF) Boolean expressions over a high-dimensional multi-valued attribute space. The goal is to rapidly find the set of Boolean expressions that evaluate to true for a given assignment of values to attributes. A solution to this problem has applications in online advertising (where a Boolean expression represents an advertiser's user targeting requirements, and an assignment of values to attributes represents the characteristics of a user visiting an online page) and in general any publish/subscribe system (where a Boolean expression represents a subscription, and an assignment of values to attributes represents an event). All existing solutions that we are aware of can only index a specialized sub-set of conjunctive and/or disjunctive expressions, and cannot efficiently handle general DNF and CNF expressions (including NOTs) over multi-valued attributes.In this paper, we present a novel solution based on the inverted list data structure that enables us to index arbitrarily complex DNF and CNF Boolean expressions over multi-valued attributes. An interesting aspect of our solution is that, by virtue of leveraging inverted lists traditionally used for ranked information retrieval, we can efficiently return the top-N matching Boolean expressions. This capability enables emerging applications such as ranked publish/subscribe systems [16], where only the top subscriptions that match an event are desired. For example, in online advertising there is a limit on the number of advertisements that can be shown on a given page and only the "best" advertisements can be displayed. We have evaluated our proposed technique based on data from an online advertising application, and the results show a dramatic performance improvement over prior techniques.

Towards an extensible efficient event processing kernel

“…In user-centric processing applications, there are computational advertising [28,10], online job sites [17,28], and location-based services for emerging applications in the co-spaces [1,19]; common to all are patterns and specifications (e.g., advertising campaigns, job profiles, service descriptions) modeled as Boolean expressions, XPath expressions, or SQL queries and incoming user information (e.g., user profiles and preferences) modeled as events using attribute-value pairs, XML document, or relational tuples. In the real-time analysis domain, there are (complex) event processing [11,2,6,7,5], XML filtering [3,18,15], intrusion detection [27], and computational finance [23]; again, common among these applications are predefined set of patterns (e.g., investment strategies and attack specifications) modeled as subscriptions and streams of incoming data (e.g., XML documents, data packets, stock feeds) modeled as events.…”

“…Unique to user-centric processing and personalization are strict requirements to determine only the most relevant content (e.g., ads) that is both user-consumable and suitable for the often limited screen real estate of client devices [17,28,10]. In addition, the usercentric processing demands scaling to millions of patterns and specifications (e.g., advertising campaigns) for supporting large-scale enterprise-level user-services, processing latency constraints in the subsecond range for meeting an acceptable service-level agreement, and improve expression expressiveness for capturing interesting patterns and desired preferences.…”

“…In general, the prevalent space dimensionality in event processing applications is in hundreds (or thousands), namely, orders of magnitude larger than capabilities of existing multi-dimensional and high-dimensional structures developed in the database community (e.g., [12].) Second, the index must support predicates with an expressive set of operators over continuous and discrete domains, which is also not supported by prior matching approaches (e.g., [2,7,28].) Third, the index must enable dynamic insertion and deletion of expressions, often disregarded as a requirement in matching algorithm design (e.g., [2,17,28].)…”

“…Second, the index must support predicates with an expressive set of operators over continuous and discrete domains, which is also not supported by prior matching approaches (e.g., [2,7,28].) Third, the index must enable dynamic insertion and deletion of expressions, often disregarded as a requirement in matching algorithm design (e.g., [2,17,28].) Fourth, the index must employ a dynamic structure that adapts to changing workload distributions and expression schemata, also often disregarded in most matching algorithm designs (e.g., [2,17,28].)…”

“…Third, the index must enable dynamic insertion and deletion of expressions, often disregarded as a requirement in matching algorithm design (e.g., [2,17,28].) Fourth, the index must employ a dynamic structure that adapts to changing workload distributions and expression schemata, also often disregarded in most matching algorithm designs (e.g., [2,17,28].) Finally, the index structure must scale to millions of Boolean expressions and afford low-latency and high-throughput expression matching demanded by most user-centric and real-time data analysis applications.…”