No abstract

IntroductionOn-line analytical processing (OLAP) is a recent and important application of database systems. Typically, O L A P data is presented as a multidimensional "data cube." O L A P queries are complex and can take many hours or even days to run, if executed directly on the raw data. The most common method of reducing execution time is to precompute some of the queries into summary tables (subcubes of the data cube) and then to build indexes on these summary tables. In most coinmercaal OLAP systems today, the summary tables that are to be precomputed are picked first, followed b y the selection of the appropriate indexes on them.A trial-and-error approach is used to divide the space available between the summary tables and the indexes. This two-step process can perform very poorly. Since both summary tables and indexes consume the same resource -space -their selection should be done together for the most eficient use of space. In this payer, we give algorithms that automate the selection of sumrria'ry tables and zndexes. In particular, we present a family of algorathms of increasing time complexities, and pro~iie strong performance bounds for them. The algorithms with higher complexities have better performance borinds. However, the increase in the performance bound is diminishing, and we show that an algorithm of moderate complexity can perform fairly close to th,e optim,al.Decision-support systems are an increasingly iinportant application of databases. Corporations are beginning to use the accumulated operational data to help understand and run their business. Towards this purpose, data from the different operations of a corporation are reconciled and stored in a central database commonly called a "data warehouse." Analysts use the data warehouse to extract the business information that enables better decision making. This interactive decision-support process is called OLAP (On-line Analytical Processing) to distinguish it from conventional OLTP (On-line Transaction Processing) applications.OLAP applications require viewing the data from many different business perspectives (dimensions).Data cube [GBLP95] is a multidimensional view of a databases where a critical value, e.g., sales, is organized by several dimensions, for example, sales of automobiles organized by model, color, day of sale and so on. The metric of interest is called the measure attrzbute, which is sales in the example. It is generally accepted that OLAP systems need to present such a multidimensional view of the data to users. Each cell of the data cube corresponds to a unique set of values for the different dimensions and contains the value of the measure for this set of values. As mentioned in [GBLP95], the domain of each dimension is augmented with the special value "ALL." ln order to present this multidimensional view, the data is usually stored in the form of "summary tables" corresponding to the subcubes of the data cube.'In [HRU96] , the efficient implementation of "data *This work was supported by NSF grant IRI-92-23405, ARO gran...

Decision support applications involve complex queries on very large databases.

Many applications that process social data, such as tweets, must extract entities from tweets (e.g., "Obama" and "Hawaii" in "Obama went to Hawaii"), link them to entities in a knowledge base (e.g., Wikipedia), classify tweets into a set of predefined topics, and assign descriptive tags to tweets. Few solutions exist today to solve these problems for social data, and they are limited in important ways. Further, even though several industrial systems such as OpenCalais have been deployed to solve these problems for text data, little if any has been published about them, and it is unclear if any of the systems has been tailored for social media. In this paper we describe in depth an end-to-end industrial system that solves these problems for social data. The system has been developed and used heavily in the past three years, first at Kosmix, a startup, and later at Wal-martLabs. We show how our system uses a Wikipedia-based global "real-time" knowledge base that is well suited for social data, how we interleave the tasks in a synergistic fashion, how we generate and use contexts and social signals to improve task accuracy, and how we scale the system to the entire Twitter firehose. We describe experiments that show that our system outperforms current approaches. Finally we describe applications of the system at Kosmix and WalmartLabs, and lessons learned.

A knowledge base (KB) contains a set of concepts, instances, and relationships. Over the past decade, numerous KBs have been built, and used to power a growing array of applications. Despite this flurry of activities, however, surprisingly little has been published about the end-to-end process of building, maintaining, and using such KBs in industry. In this paper we describe such a process. In particular, we describe how we build, update, and curate a large KB at Kosmix, a Bay Area startup, and later at WalmartLabs, a development and research lab of Walmart. We discuss how we use this KB to power a range of applications, including query understanding, Deep Web search, in-context advertising, event monitoring in social media, product search, social gifting, and social mining. Finally, we discuss how the KB team is organized, and the lessons learned. Our goal with this paper is to provide a real-world case study, and to contribute to the emerging direction of building, maintaining, and using knowledge bases for data management applications.