SPARQL basic graph pattern processing with iterative MapReduce

Myung, Jaeseok; Yeon, Jongheum; Lee, Sang-goo

doi:10.1145/1779599.1779605

Cited by 50 publications

(29 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous work has shown the N-Triple serialisation of RDF to be the most suitable for processing via Hadoop, as it represents a complete triple via a single line of text [11].…”

Section: Hadoop Implementationmentioning

confidence: 99%

Data quality assessment and anomaly detection via map/reduce and linked data: A case study in the medical domain

Bonner

McGough

Kureshi

et al. 2015

2015 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

Use policyThe full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-pro t purposes provided that:• a full bibliographic reference is made to the original source • a link is made to the metadata record in DRO • the full-text is not changed in any way The full-text must not be sold in any format or medium without the formal permission of the copyright holders.Please consult the full DRO policy for further details. Abstract-Recent technological advances in modern healthcare have lead to the ability to collect a vast wealth of patient monitoring data. This data can be utilised for patient diagnosis but it also holds the potential for use within medical research. However, these datasets often contain errors which limit their value to medical research, with one study finding error rates ranging from 2.3% -26.9% in a selection of medical databases.Previous methods for automatically assessing data quality normally rely on threshold rules, which are often unable to correctly identify errors, as further complex domain knowledge is required. To combat this, a semantic web based framework has previously been developed to assess the quality of medical data. However, early work, based solely on traditional semantic web technologies, revealed they are either unable or inefficient at scaling to the vast volumes of medical data.In this paper we present a new method for storing and querying medical RDF datasets using Hadoop Map / Reduce. This approach exploits the inherent parallelism found within RDF datasets and queries, allowing us to scale with both dataset and system size. Unlike previous solutions, this framework uses highly optimised (SPARQL) joining strategies, intelligent data caching and the use of a super-query to enable the completion of eight distinct SPARQL lookups, comprising over eighty distinct joins, in only two Map / Reduce iterations. Results are presented comparing both the Jena and a previous Hadoop implementation demonstrating the superior performance of the new methodology. The new method is shown to be five times faster than Jena and twice as fast as the previous approach.

show abstract

“…Previous work has shown the N-Triple serialisation of RDF to be the most suitable for processing via Hadoop, as it represents a complete triple via a single line of text [11].…”

Section: Hadoop Implementationmentioning

confidence: 99%

Data quality assessment and anomaly detection via map/reduce and linked data: A case study in the medical domain

Bonner

McGough

Kureshi

et al. 2015

2015 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

show abstract

“…MapReduce is highly fault tolerant and scalable and can run on clusters with thousands of machines, facilitating its wide use as a cloud programming framework in bioinformatics (67). The projects (72)(73)(74)(75)(76) focused on developing large-scale RDF stores using the MapReduce paradigm. (82) is an open-source, distributed, column-oriented store model.…”

Section: Triple Store In the Cloudmentioning

confidence: 99%

Semantic Web technologies for the big data in life sciences

Yamaguchi

2014

BST

View full text Add to dashboard Cite

“…A query is usually made of a set of triple patterns and some filtering and returns only the triples matching all patterns and filters. We have implemented the query processing in Hadoop using the algorithm described in [11]. It is based on two phases: the selection phase and the join phase.…”

Section: B Rdf Querymentioning

confidence: 99%

“…It is based on two phases: the selection phase and the join phase. The selection phase filters the RDF data that satisfy at least one triple pattern and the join phase runs iteratively based on the number of join variables in the Basic Graph Pattern of the query [11].…”

Section: B Rdf Querymentioning

confidence: 99%

A Lightweight Continuous Jobs Mechanism for MapReduce Frameworks

Huet

2013

2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing

View full text Add to dashboard Cite

Abstract-MapReduce is a programming model which allows the processing of vast amounts of data in parallel, on a large number of machines. It is particularly well suited to static or slow changing set of data since the execution time of a job is usually high. However, in practice data-centers collect data at fast rates which makes it very difficult to maintain up-to-date results. To address this challenge, we propose in this paper a generic mechanism for dealing with dynamic data in MapReduce frameworks. Long-standing MapReduce jobs, called continuous Jobs, are automatically re-executed to process new incoming data at a minimum cost. We present a simple and clean API which integrates nicely with the standard MapReduce model. Furthermore, we describe cHadoop, an implementation of our approach based on Hadoop which does not require modifications to the source code of the original framework. Thus, cHadoop can quickly be ported to any new version of Hadoop. We evaluate our proposal with two standard MapReduce applications (WordCount and WordCount-N-Count), and one real world application (RDF Query) on real datasets. Our evaluations on clusters ranging from 5 to 40 nodes demonstrate the benefit of our approach in terms of execution time and ease of use.

show abstract

SPARQL basic graph pattern processing with iterative MapReduce

Cited by 50 publications

References 12 publications

Data quality assessment and anomaly detection via map/reduce and linked data: A case study in the medical domain

Data quality assessment and anomaly detection via map/reduce and linked data: A case study in the medical domain

Semantic Web technologies for the big data in life sciences

A Lightweight Continuous Jobs Mechanism for MapReduce Frameworks

Contact Info

Product

Resources

About