epiC: an extensible and scalable system for processing Big Data

Jiang, Dawei; Wu, Sai; Chen, Gang; Ooi, Beng Chin; Tan, Kian-Lee; Xu, Jun

doi:10.1007/s00778-015-0393-2

Cited by 25 publications

(21 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our proposed recovery scheme is an important component of our epiCG project: a scalable graph engine on top of epiC [12]. In this paper, we use Giraph [1] as the underlying graph engine for the experiments as epiCG was under development when our recovery method was being designed.…”

Section: Introductionmentioning

confidence: 99%

Fast failure recovery in distributed graph processing systems

et al. 2014

Self Cite

View full text Add to dashboard Cite

Distributed graph processing systems increasingly require many compute nodes to cope with the requirements imposed by contemporary graph-based Big Data applications. However, increasing the number of compute nodes increases the chance of node failures. Therefore, provisioning an efficient failure recovery strategy is critical for distributed graph processing systems. This paper proposes a novel recovery mechanism for distributed graph processing systems that parallelizes the recovery process. The key idea is to partition the part of the graph that is lost during a failure among a subset of the remaining nodes. To do so, we augment the existing checkpoint-based and log-based recovery schemes with a partitioning mechanism that is sensitive to the total computation and communication cost of the recovery process. Our implementation on top of the widely used Giraph system outperforms checkpointbased recovery by up to 30x on a cluster of 40 compute nodes.

show abstract

Section: Introductionmentioning

confidence: 99%

Fast failure recovery in distributed graph processing systems

et al. 2014

Self Cite

View full text Add to dashboard Cite

show abstract

“…After gathering the newest user features and system status, the recommendation module implements a series of machine learning methods to recommend a list of dating partners to each user. In this experiment, we adopt epiC [6], an extensible and scalable system as the integrated storage/computing platform(our recommendation system is also available on other big data platforms). It is designed to handle multistructured data under a united interface by decoupling the concurrent programming model and data processing model, which processes the data-intensive computing automatically in parallel efficiently.…”

Section: System Architecturementioning

confidence: 99%

A personalized recommendation system for NetEase dating site

et al. 2014

View full text Add to dashboard Cite

With the popularity of internet, more and more people try to find friends or dating partners on online dating web sites. Recommending appropriate partners from a large amount of candidates becomes an interesting and challenging problem in the field of recommendation system. Various types of recommendation techniques (e.g., content based recommendation, collaborative filtering and association rule mining) have be proposed to tackle this problem. However most of them ignore the personalization concerns that they (1) mainly consider the hot users or frequent items, (2) cover only a portion of users especially ignoring the long tails, (3) and cannot deal with the cold start problem properly. In this paper, we present a regression based hybrid recommendation system that makes use of matching degree, fancy degree, activity, sincerity, popularity and enthusiasm, to recommend appropriate partners. The experimental evaluation of our recommendation system on a real dating web site shows our strategy is more effective and efficient than its previous version which follows the principle of giving higher priority to the recent active users.

show abstract

“…Addressing this problem, Dryad [10] enables the user to handle a DAG of tasks, and Pregel [13] deals with vertex-centric programming for large graphs. Recently, epiC [11] provides a unified framework for a variety of data types, combining the merits of all afore-mentioned systems. We refer the reader to a comprehensive survey [12] for other related systems.…”

Section: Related Workmentioning

confidence: 99%

Design and implementation of a real-time interactive analytics system for large spatio-temporal data

et al. 2014

View full text Add to dashboard Cite

In real-time interactive data analytics, the user expects to receive the results of each query within a short time period such as seconds. This is especially challenging when the data is big (e.g., on the scale of petabytes), and the analytics system runs on top of cloud infrastructure (e.g., thousands of interconnected commodity servers). We have been building such a system, called OceanRT, for managing large spatio-temporal data such as call logs and mobile web browsing records collected by a telecommunication company. Although there already exist systems for querying big data in real time, OceanRT's performance stands out due to several novel designs and components that address key efficiency and scalability issues that were largely overlooked in existing systems. First, OceanRT makes extensive use of software RDMA one-sided operations, which reduce networking costs without requiring specialized hardware. Second, OceanRT exploits the parallel computing capabilities of each node in the cloud through a novel architecture consisting of Access-Query Engines (AQEs) connected with minimal overhead. Third, OceanRT contains a novel storage scheme that optimizes for queries with joins and multi-dimensional selections, which are common for large spatiotemporal data. Experiments using the TPC-DS benchmark show that OceanRT is usually more than an order of magnitude faster than the current state-of-the-art systems.

show abstract

epiC: an extensible and scalable system for processing Big Data

Cited by 25 publications

References 27 publications

Fast failure recovery in distributed graph processing systems

Fast failure recovery in distributed graph processing systems

A personalized recommendation system for NetEase dating site

Design and implementation of a real-time interactive analytics system for large spatio-temporal data

Contact Info

Product

Resources

About