Hive - a petabyte scale data warehouse using Hadoop

Thusoo, Ashish; Sarma, Joydeep Sen; Jain, Namit; Zheng, Shu; Chakka, Prasad; Zhang, Ning; Antony, Suresh J.; H, Liu; Murthy, Raghotham

doi:10.1109/icde.2010.5447738

Cited by 710 publications

(411 citation statements)

References 2 publications

Supporting

Mentioning

404

Contrasting

Unclassified

Order By: Relevance

“…In our work, we try to bring many of the existing DB features and incorporate them into Hadoop without utilising database systems as auxiliary tools to support them. There are many optimisation techniques that allow increasing the performance of MapReduce based solutions (Jiang et al, 2010;Thusoo et al, 2010). For instance, one way to improve Hadoop's performance is to design an optimised storage model that utilises various indexing solutions (Richter et al, 2014).…”

Section: Related Workmentioning

confidence: 99%

“…Hive is a good example of such an application that allows SQL-trained users to query data stored in the HDFS using HiveQL (Thusoo et al, 2010;Hive, 2014). Hive queries work in a way that is similar to traditional databases.…”

Section: 3mentioning

confidence: 99%

“…Equipped with different pruning strategies together with predicate push-down (PPD) optimisation (Thusoo et al, 2010), we can offer our version of an improved partition elimination algorithm. As a result of PPD optimisation, storage formats are exposed to expression trees containing a list of predicates specified by the user.…”

Section: Index-based Partition Eliminationmentioning

confidence: 99%

See 2 more Smart Citations

Efficient storage, retrieval and analysis of poker hands: An adaptive data framework

Gorawski

Lorek

2017

International Journal of Applied Mathematics and Computer Science

View full text Add to dashboard Cite

In online gambling, poker hands are one of the most popular and fundamental units of the game state and can be considered objects comprising all the events that pertain to the single hand played. In a situation where tens of millions of poker hands are produced daily and need to be stored and analysed quickly, the use of relational databases no longer provides high scalability and performance stability. The purpose of this paper is to present an efficient way of storing and retrieving poker hands in a big data environment. We propose a new, read-optimised storage model that offers significant data access improvements over traditional database systems as well as the existing Hadoop file formats such as ORC, RCFile or SequenceFile. Through index-oriented partition elimination, our file format allows reducing the number of file splits that needs to be accessed, and improves query response time up to three orders of magnitude in comparison with other approaches. In addition, our file format supports a range of new indexing structures to facilitate fast row retrieval at a split level. Both index types operate independently of the Hive execution context and allow other big data computational frameworks such as MapReduce or Spark to benefit from the optimized data access path to the hand information. Moreover, we present a detailed analysis of our storage model and its supporting index structures, and how they are organised in the overall data framework. We also describe in detail how predicate based expression trees are used to build effective file-level execution plans. Our experimental tests conducted on a production cluster, holding nearly 40 billion hands which span over 4000 partitions, show that multi-way partition pruning outperforms other existing file formats, resulting in faster query execution times and better cluster utilisation.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: 3mentioning

confidence: 99%

Section: Index-based Partition Eliminationmentioning

confidence: 99%

See 1 more Smart Citation

Efficient storage, retrieval and analysis of poker hands: An adaptive data framework

Gorawski

Lorek

2017

International Journal of Applied Mathematics and Computer Science

View full text Add to dashboard Cite

show abstract

“…One of the earliest proposals for a query language for MapReduce was from Hive [11], a data warehouse system with an SQL-like interface. Its evaluation engine represents the basic mechanism for converting a computational model based on query operators into that of MR.…”

Section: Related Workmentioning

confidence: 99%

Versatile XQuery Processing in MapReduce

Sauer

Bächle

Härder

2013

Advances in Databases and Information Systems

View full text Add to dashboard Cite

Abstract. The MapReduce (MR) framework has become a standard tool for performing large batch computations-usually of aggregative nature-in parallel over a cluster of commodity machines. A significant share of typical MR jobs involves standard database-style queries, where it becomes cumbersome to specify map and reduce functions from scratch. To overcome this burden, higher-level languages such as HiveQL, PigLatin, and JAQL have been proposed to allow the automatic generation of MR jobs from declarative queries. We identify two major problems of these existing solutions: (i) they introduce new query languages and implement systems from scratch for the sole purpose of expressing MR jobs; and (ii) despite solving some of the major limitations of SQL, they still lack the flexibility required by big data applications. We propose BrackitMR, an approach based on the XQuery language with extended JSON support. XQuery not only is an established query language, but also has a more expressive data model and more powerful language constructs, enabling a much greater degree of flexibility. From a system design perspective, we extend an existing single-node query processor, Brackit, adding MR as a distributed coordination layer. Such heavy reuse of the standard query processor not only provides performance, but also allows for a more elegant design which transparently integrates MR processing into a generic query engine.

show abstract

“…Frameworks such as Dryad [18], Pig [28], and Hive [45] use SQL and/or MapReduce constructs to describe data processing tasks out-of-core. While convenient for programming, such frameworks are limited to data that can be expressed as key-value pairs and to processing of loosely-coupled problems that are expressed as a global reduction of embarrassingly parallel components.…”

Section: Data-intensive Programming Modelsmentioning

confidence: 99%