Parallel and distributed approach for processing large-scale XML datasets

Abstract

Abstract-An emerging trend is the use of XML as the data format for many distributed scientific applications, with the size of these documents ranging from tens of megabytes to hundreds of megabytes. Our earlier benchmarking results revealed that most of the widely available XML processing toolkits do not scale well for large sized XML data. A significant transformation is necessary in the design of XML processing for scientific applications so that the overall application turn-around time is not negatively affected. We present both a parallel and distributed approach to analyze how the scalability and performance requirements of large-scale XML-based data processing can be achieved. We have adapted the Hadoop implementation to determine the threshold data sizes and computation work required per node, for a distributed solution to be effective. We also present an analysis of parallelism using our PIXIMAL toolkit for processing large-scale XML datasets that utilizes the capabilities for parallelism that are available in the emerging multi-core architectures. Multi-core processors are expected to be widely available in research clusters and scientific desktops, and it is critical to harness the opportunities for parallelism in the middleware, instead of passing on the task to application programmers. Our parallelization approach for a multi-core node is to employ a DFA-based parser that recognizes a useful subset of the XML specification, and convert the DFA into an NFA that can be applied to an arbitrary subset of the input. Speculative NFAs are scheduled on available cores in a node to effectively utilize the processing capabilities and achieve overall performance gains. We evaluate the efficacy of this approach in terms of potential speedup that can be achieved for representative XML data sets.

Benchmarking MapReduce Implementations for Application Usage Scenarios

“…In fact, LEMO-MR and Twister have similar performance, even as Twister shows slightly faster runs than LEMO-MR. Hadoop, here proves to be slower in this CPU-intensive scenario. Hadoop uses a number of overheadprone operations such as data chunk replication, constant worker node pings along with speculative and redundant jobs, mainly for fault-tolerance reasons [20]. These measures can however impede performance in any application setting [3] However, in [4] this situation is shown to be further aggravated in CPU-intensive scenarios where the pre-dominance of CPU operations and the long processing nature of tasks make duplicating tasks and speculative jobs launched by Hadoop more costly than in data-intensive or memory-intensive cases.…”

DELMA: Dynamically ELastic MapReduce Framework for CPU-Intensive Applications

“…The ability to provide more nodes, and thus more power to applications while they are running, has thusfar not been explored in the MapReduce context. In a corollary condition, the inability to predict before making a job live, what the optimal cluster configuration should be for that particular job can create a condition where extraneous nodes hinder the performance produced by the optimal number of nodes required for a job [14]. A user in such straights would be unable to remove the extra nodes, and let the cluster run efficiently at its best, but would instead have to suffer the loss of performance, and wait to correct the condition in subsequent runs.…”

“…We not only test the impact of early and late node additions, but also the impact of diverse cluster sizes on DELMA, as well as that of progressive node addition. Our prior work with MapReduce applications [14] has shown that for application turn-around time to be efficient, the overhead introduced by additional processing units must dwarf the time gained by the work produced by the participating nodes. In a traditional MapReduce context, this condition is encountered when input processing sizes are small, or when the processing per input element within the input itself is insignificant.…”

“…In a traditional MapReduce context, this condition is encountered when input processing sizes are small, or when the processing per input element within the input itself is insignificant. In [14], we showed that a cluster of 5 Hadoop nodes could be outstaged by a single system for input sizes below a given threshold of data. In brief, a MapReduce job would yield poor performance if the time used to provide the nodes the input is significantly greater than the time spent by the collection of nodes processing that input.…”

XML Parsing on Multicore Processors and Data Representation in . NET Tree Control

“…They believed that a parallel XML processing model should be a cost-effective solution for the XML performance issue in the multicore era. [24] adapted the Hadoop implementation to determine the threshold data sizes and computation work required per node for a distributed solution to be effective. They also presented an analysis of parallelism using PIXIMAL toolkit for processing large-scale XML datasets that utilizes the capabilities for parallelism that were available in the emerging multi-core architectures.…”