Efficient Querying Distributed Big-XML Data using MapReduce

Song, Kunfang; Lu, Hongwei

doi:10.4018/ijghpc.2016070105

Cited by 7 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other studies also utilize XML documents to evaluate the processing time with HDFS and the Apache Spark engine; however, the files used for the tests contain simple types, with a few attributes, or do not present schemas ( Hai, Quix & Zhou, 2018 ; Kunfang, 2016 ; Zhang & Lu, 2021 ).…”

Section: Related Workmentioning

confidence: 99%

Efficient processing of complex XSD using Hive and Spark

Martinez-Mosquera

Navarrete

Luján‐Mora

2021

PeerJ Computer Science

View full text Add to dashboard Cite

The eXtensible Markup Language (XML) files are widely used by the industry due to their flexibility in representing numerous kinds of data. Multiple applications such as financial records, social networks, and mobile networks use complex XML schemas with nested types, contents, and/or extension bases on existing complex elements or large real-world files. A great number of these files are generated each day and this has influenced the development of Big Data tools for their parsing and reporting, such as Apache Hive and Apache Spark. For these reasons, multiple studies have proposed new techniques and evaluated the processing of XML files with Big Data systems. However, a more usual approach in such works involves the simplest XML schemas, even though, real data sets are composed of complex schemas. Therefore, to shed light on complex XML schema processing for real-life applications with Big Data tools, we present an approach that combines three techniques. This comprises three main methods for parsing XML files: cataloging, deserialization, and positional explode. For cataloging, the elements of the XML schema are mapped into root, arrays, structures, values, and attributes. Based on these elements, the deserialization and positional explode are straightforwardly implemented. To demonstrate the validity of our proposal, we develop a case study by implementing a test environment to illustrate the methods using real data sets provided from performance management of two mobile network vendors. Our main results state the validity of the proposed method for different versions of Apache Hive and Apache Spark, obtain the query execution times for Apache Hive internal and external tables and Apache Spark data frames, and compare the query performance in Apache Hive with that of Apache Spark. Another contribution made is a case study in which a novel solution is proposed for data analysis in the performance management systems of mobile networks.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient processing of complex XSD using Hive and Spark

Martinez-Mosquera

Navarrete

Luján‐Mora

2021

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Other studies also utilize XML documents to evaluate the processing time with HDFS and the Apache Spark engine; however, the files used for the tests contain simple types, with a few attributes, or do not present schemas (Hai et al, 2018;Kunfang, 2016;Zhang and Lu, 2021).…”

Section: Related Workmentioning

confidence: 99%