Reef

Chun, Byung-Gon; Condie, Tyson; Curino, Carlo; Douglas, Chris; Matusevych, Sergiy; Myers, Brandon; Narayanamurthy, Shravan; Ramakrishnan, Raghu; Rao, Sriram; Rosen, Josh; Sears, Russell; Weimer, Markus

doi:10.14778/2536274.2536318

Cited by 8 publications

(2 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As with Orleans, SCOPE owns threads and schedules reducer code; thus, users embed Trill as a no-thread library within their reducers in order to perform temporal analytics [1] over search data such as clicks, impressions, and page views. Another such fabric used with Trill is REEF [18], which is built on YARN [17].…”

Section: Current Usage Scenariosmentioning

confidence: 99%

See 1 more Smart Citation

Trill

et al. 2014

View full text Add to dashboard Cite

This paper introduces Trill-a new query processor for analytics. Trill fulfills a combination of three requirements for a query processor to serve the diverse big data analytics space: (1) Query Model: Trill is based on a tempo-relational model that enables it to handle streaming and relational queries with early results, across the latency spectrum from real-time to offline; (2) Fabric and Language Integration: Trill is architected as a high-level language library that supports rich data-types and user libraries, and integrates well with existing distribution fabrics and applications; and (3) Performance: Trill's throughput is high across the latency spectrum. For streaming data, Trill's throughput is 2-4 orders of magnitude higher than comparable streaming engines. For offline relational queries, Trill's throughput is comparable to a major modern commercial columnar DBMS. Trill uses a streaming batched-columnar data representation with a new dynamic compilation-based system architecture that addresses all these requirements. In this paper, we describe Trill's new design and architecture, and report experimental results that demonstrate Trill's high performance across diverse analytics scenarios. We also describe how Trill's ability to support diverse analytics has resulted in its adoption across many usage scenarios at Microsoft.

show abstract

Section: Current Usage Scenariosmentioning

confidence: 99%

“…Phoenix++ [7] is a variant of map-reduce for in-memory analytics; unlike Trill, it is neither temporal nor streaming, and exposes a low-level key-value API. YARN [17] and REEF [18] generalize Hadoop to a distributed resource manager. Storm is a streaming analytics framework that can potentially embed Trill within its bolts.…”

Section: Related Workmentioning

confidence: 99%

Trill

et al. 2014

View full text Add to dashboard Cite

show abstract

Performance evaluation of job schedulers on Hadoop YARN

Lin

Lee

2016

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARYTo solve the limitation of Hadoop on scalability, resource sharing, and application support, the opensource community proposes the next generation of Hadoop's compute platform called Yet Another Resource Negotiator (YARN) by separating resource management functions from the programming model. This separation enables various application types to run on YARN in parallel. To achieve fair resource sharing and high resource utilization, YARN provides the capacity scheduler and the fair scheduler. However, the performance impacts of the two schedulers are not clear when mixed applications run on a YARN cluster. Therefore, in this paper, we study four scheduling-policy combinations (SPCs for short) derived from the two schedulers and then evaluate the four SPCs in extensive scenarios, which consider not only four application types, but also three different queue structures for organizing applications. The experimental results enable YARN managers to comprehend the influences of different SPCs and different queue structures on mixed applications. The results also help them to select a proper SPC and an appropriate queue structure to achieve better application execution performance.

show abstract

Blind men and an elephant coalescing open-source, academic, and industrial perspectives on BigData

Douglas

Curino

2015

2015 IEEE 31st International Conference on Data Engineering

Self Cite

View full text Add to dashboard Cite

This tutorial is organized in two parts. In the first half, we will present an overview of applications and services in the BigData ecosystem. We will use known distributed database and systems literature as landmarks to orient the attendees in this fast-evolving space. Throughout, we will contrast models of resource management, performance, and the constraints that shape the architectures of prominent systems. We will also discuss the role of academia and industry in the development of opensource infrastructure, with an emphasis on open problems and strategies for collaboration. We assume only basic familiarity with distributed systems.In the second half, we will delve into Apache Hadoop YARN. YARN (Yet Another Resource Negotiator) transformed Hadoop from a MapReduce engine to a general-purpose cluster scheduler. Since its introduction, it has been deployed in production and extended to support use cases beyond large-scale batch processing. The tutorial will present the active research and development supporting such heterogeneous workloads, with particular attention to multi-tenant scheduling. Topics include security, resource isolation, protocols, and preemption. This portion will be detailed, but accessible to anyone with a background in distributed systems and all attendees of the first half of the tutorial.978-1-4799-7964-6/15/$31.00

show abstract

Reef

Cited by 8 publications

References 33 publications

Trill

Trill

Performance evaluation of job schedulers on Hadoop YARN

Blind men and an elephant coalescing open-source, academic, and industrial perspectives on BigData

Contact Info

Product

Resources

About