Major technical advancements in apache hive

Huai, Yin; Chauhan, Ashutosh Singh; Gates, Alan; Hagleitner, Günther; Hanson, Eric N.; O'Malley, Owen; Pandey, Jitendra; Yuan, Yuan; Lee, Rubao; Zhang, Xiaodong

doi:10.1145/2588555.2595630

Cited by 124 publications

(51 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In big data query systems, column-wise compression [26] [27] [28] is commonly seen. [26] uses a method that simply compresses column data into a series of self-contained blocks.…”

Section: Data Compressionmentioning

confidence: 99%

“…Another scheme, Llama [27] allows groupedcolumn compression with different schemes best suitable for groups of records, and builds indices for compression blocks. Hive [28] is a data management system that allows column-wise compression. Data is organized like database tables in Optimized Record Columnar (ORC) format which is specific to Hive.…”

Section: Data Compressionmentioning

confidence: 99%

See 1 more Smart Citation

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Dong

Herbert

2018

IEEE Trans. Big Data

View full text Add to dashboard Cite

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Abstract-A substantial amount of information in companies and on the Internet is present in the form of text. The value of this semi-structured and unstructured data has been widely acknowledged, with consequent scientific and commercial exploitation. The ever-increasing data production, however, pushes data analytic platforms to their limit. Compression as an effective means to reduce data size has been employed by many emerging data analytic platforms, whom the main purpose of data compression is to save storage space and reduce data transmission cost over the network. Since general purpose compression methods endeavour to achieve higher compression ratios by leveraging data transformation techniques and contextual data, this context-dependency forces the access to the compressed data to be sequential. Processing such compressed data in parallel, such as desirable in a distributed environment, is extremely challenging. This work proposes techniques for more efficient textual big data analysis with an emphasis on content-aware compression schemes suitable for the Hadoop analytic platform. The compression schemes have been evaluated for a number of standard MapReduce analysis tasks using a collection of public and private real-world datasets. In comparison with existing solutions, they have shown substantial improvement in performance and significant reduction in system resource requirements.

show abstract

“…In big data query systems, column-wise compression [26] [27] [28] is commonly seen. [26] uses a method that simply compresses column data into a series of self-contained blocks.…”

Section: Data Compressionmentioning

confidence: 99%

Section: Data Compressionmentioning

confidence: 99%

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Dong

Herbert

2018

IEEE Trans. Big Data

View full text Add to dashboard Cite

show abstract

“…Its recent version has shifted from the classic MapReduce to Tez platform [40], which has made it practical for interactive analytics by accelerating it many times. ORC (Optimized Row Column) is used in Hive [41].…”

Section: Conceptual Big Data Architecture For the Oil And Gas Industrymentioning

confidence: 99%

Conceptual Big Data Architecture for the Oil and Gas Industry

Alıguliyev¹,

İmamverdiyev²

2017

JPIT

View full text Add to dashboard Cite

“…Apache provides several database oriented frameworks like Pig [23], Hive [24] and HBase [25]. Apache Pig consist of its own data flow language: PigLatin.…”

Section: Pig Hive and Hbasementioning

confidence: 99%

Rule Based Systems in a Distributed Environment: Survey

Bagavathi¹,

Tzacheva²

2017

World Congress on Electrical Engineering and Computer Systems and Science

View full text Add to dashboard Cite

-Over the past years, the internet has become faster, computer storage has become larger and the data from internet users and sensors is piling up to larger amount and is also spread around the globe. This requires more time and space for a single computer to cope with them. Ecosystems like Hadoop helps to store, process and retrieve back the data efficiently in a distributed fashion. Data mining finds substantial improvements over such distributed frameworks to process a large volume of data in a lesser time. Currently, there are many approaches to do data mining tasks such as classification and clustering in a distributed setup using Hadoop MapReduce, Spark, and other Cloud platforms. Actionable pattern mining is a rule based data mining approach for discovering knowledge from information systems in a form of Action Rules. An emphasis of traditional classification rules from a supervised Machine Learning is to predict class label of a data object. Whereas Action Rules produce actionable knowledge in the form of suggestions on how an object can change from one class value to another more desirable class value. This paper gives a brief survey of previous works on association and classification rule mining algorithms in a distributed environment, as well as action rule mining algorithms, and discusses Action Rule Mining in a distributed environment.

show abstract

Major technical advancements in apache hive

Cited by 124 publications

References 29 publications

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Conceptual Big Data Architecture for the Oil and Gas Industry

Rule Based Systems in a Distributed Environment: Survey

Contact Info

Product

Resources

About