On brewing fresh espresso

Abstract

Espresso is a document-oriented distributed data serving platform that has been built to address LinkedIn's requirements for a scalable, performant, source-of-truth primary store. It provides a hierarchical document model, transactional support for modifications to related documents, realtime secondary indexing, on-the-fly schema evolution and provides a timeline consistent change capture stream. This paper describes the motivation and design principles involved in building Espresso, the data model and capabilities exposed to clients, details of the replication and secondary indexing implementation and presents a set of experimental results that characterize the performance of the system along various dimensions.When we set out to build Espresso, we chose to apply best practices in industry, already published works in research and our own internal experience with different consistency models. Along the way, we built a novel generic distributed cluster management framework, a partition-aware changecapture pipeline and a high-performance inverted index implementation.

Mitigating Load Imbalance in Distributed Data Serving with Rack-Scale Memory Pooling

“…A workload with 20% of writes reduces the speedup from 6× for YCSB-B to 3.2×, and YCSB-A (50% writes) reduces it to 1.7×. However, workloads with atypically high fraction of writes are rare [7,10,15,62]. We observe a difference below 6% between the model and the platform at 1ms SLO.…”

“…Recent work has demonstrated the scalability benefits of the CREW model on Xeon-class servers [44,45]. As most workloads are read dominated [7,10,15,62], CREW offers a sweet spot in terms of scalable performance by keeping synchronization requirements to a minimum.…”

Apache Samza

“…At LinkedIn, Samza is commonly deployed with Databus inputs: Databus is a change data A Apache Samza, Fig. 1 The two operators of a streaming word-frequency counter using Samza's StreamTask API (Image source: Kleppmann andKreps 2015, © 2015 IEEE, reused with permission) capture technology that records the log of writes to a database and makes this log available for applications to consume (Das et al 2012;Qiao et al 2013). Processing the stream of writes to a database enables jobs to maintain external indexes or materialized views onto data in a database and is especially relevant in conjunction with Samza's support for local state (see section "Fault-Tolerant Local State") ( Fig.…”