Realtime Data Processing at Facebook

Chen, Guoqiang Jerry; Wiener, Joshua; Iyer, Shridhar Ganpathi; Jaiswal, Anshul; Lei, Ran; Simha, N. V. R.; Wang, Wei; Wilfong, Kevin; Williamson, Tim; Yılmaz, Serhat

doi:10.1145/2882903.2904441

Cited by 89 publications

(50 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It provides adaptors for commonly accessed data sources such as MySQL, S3, Kafka, and Salesforce. Other similar systems include Scribe [20], a messaging system as Facebook; Siphon [21], a messaging system in Microsoft Azure HDInsight that utilize Kafka. Marcu et al [12] developed KerA, a data ingestion framework that alleviate the limitations of Kafka and other ingestion systems.…”

Section: Dataflow Output and Performance Improvementmentioning

confidence: 99%

A Scalable and Robust Framework for Data Stream Ingestion

Isah

Zulkernine

2018

2018 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

An essential part of building a data-driven organization is the ability to handle and process continuous streams of data to discover actionable insights. The explosive growth of interconnected devices and the social Web has led to a large volume of data being generated on a continuous basis. Streaming data sources such as stock quotes, credit card transactions, trending news, traffic conditions, time-sensitive patient's data are not only very common but can rapidly depreciate if not processed quickly. The ever-increasing volume and highly irregular nature of data rates pose new challenges to data stream processing systems. One such challenging but important task is how to accurately ingest and integrate data streams from various sources and locations into an analytics platform. These challenges demand new strategies and systems that can offer the desired degree of scalability and robustness in handling failures. This paper investigates the fundamental requirements and the state of the art of existing data stream ingestion systems, propose a scalable and fault-tolerant data stream ingestion and integration framework that can serve as a reusable component across many feeds of structured and unstructured input data in a given platform, and demonstrate the utility of the framework in a real-world data stream processing case study that integrates Apache NiFi and Kafka for processing high velocity news articles from across the globe. The study also identifies best practices and gaps for future research in developing large-scale data stream processing infrastructure.

show abstract

Section: Dataflow Output and Performance Improvementmentioning

confidence: 99%

A Scalable and Robust Framework for Data Stream Ingestion

Isah

Zulkernine

2018

2018 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

show abstract

“…Streaming data represents a continuous flow of data that needs to be processed by systems equipped to ingest, process, store and analyze the data. Existing data stream processing systems (DSPS) [18][19][20][21] typically include the following main components: (i) streaming data sources, (ii) data ingestion systems, (iii) data stream processing engines (DSPE), (iv) storage systems, (v) resource management services, and (vi) data sink to channel the output to other DSPSs, storage or visualization tools. Fig.…”

Section: A Streaming Data Proccessing Systemsmentioning

confidence: 99%

“…The DSPS at Facebook [21] powers many use cases such as the real-time reporting of the aggregated voice of Facebook users, analytics for mobile applications, and insights for Facebook page administrators. It is made up of data sources such as mobile and web products; Scribe as a data distribution tool; stream processing systems such as Puma, Stylus, and Swift; and data stores such as Laser, Scuba, and Hive.…”

Section: B Dspsmentioning

confidence: 99%

Detecting Irregular Patterns in IoT Streaming Data for Fall Detection

Mahfuz

Isah

Zulkernine

et al. 2018

2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON)

View full text Add to dashboard Cite

Detecting patterns in real time streaming data has been an interesting and challenging data analytics problem. With the proliferation of a variety of sensor devices, real-time analytics of data from the Internet of Things (IoT) to learn regular and irregular patterns has become an important machine learning problem to enable predictive analytics for automated notification and decision support. In this work, we address the problem of learning an irregular human activity pattern, fall, from streaming IoT data from wearable sensors. We present a deep neural network model for detecting fall based on accelerometer data giving 98.75 percent accuracy using an online physical activity monitoring dataset called "MobiAct", which was published by Vavoulas et al. The initial model was developed using IBM Watson studio and then later transferred and deployed on IBM Cloud with the streaming analytics service supported by IBM Streams for monitoring real-time IoT data. We also present the systems architecture of the real-time fall detection framework that we intend to use with Mbientlab's wearable health monitoring sensors for real time patient monitoring at retirement homes or rehabilitation clinics.

show abstract

“…Distributed stream processing systems like Storm [7] and Heron [8] are frequently used for real-time analysis, online machine learning and continuous computing. As also identified by researchers at Facebook [9], the ease of use, performance, fault tolerance, scalability and correctness are five important design decisions for real-time data stream processing systems. Two architectural styles for real-time data processing are the Lambda architecture (http://lambda-architecture.net, by Nathan Marz, creator of Storm) [10] and the Kappa architecture (http://kappa-architecture.com, by Jay Kreps, from LinkedIn).…”

Section: Introductionmentioning

confidence: 99%

Access Control with Delegated Authorization Policy Evaluation for Data-Driven Microservice Workflows

Preuveneers

Joosen

2017

Future Internet

View full text Add to dashboard Cite

Microservices offer a compelling competitive advantage for building data flow systems as a choreography of self-contained data endpoints that each implement a specific data processing functionality. Such a 'single responsibility principle' design makes them well suited for constructing scalable and flexible data integration and real-time data flow applications. In this paper, we investigate microservice based data processing workflows from a security point of view, i.e., (1) how to constrain data processing workflows with respect to dynamic authorization policies granting or denying access to certain microservice results depending on the flow of the data; (2) how to let multiple microservices contribute to a collective data-driven authorization decision and (3) how to put adequate measures in place such that the data within each individual microservice is protected against illegitimate access from unauthorized users or other microservices. Due to this multifold objective, enforcing access control on the data endpoints to prevent information leakage or preserve one's privacy becomes far more challenging, as authorization policies can have dependencies and decision outcomes cross-cutting data in multiple microservices. To address this challenge, we present and evaluate a workflow-oriented authorization framework that enforces authorization policies in a decentralized manner and where the delegated policy evaluation leverages feature toggles that are managed at runtime by software circuit breakers to secure the distributed data processing workflows. The benefit of our solution is that, on the one hand, authorization policies restrict access to the data endpoints of the microservices, and on the other hand, microservices can safely rely on other data endpoints to collectively evaluate cross-cutting access control decisions without having to rely on a shared storage backend holding all the necessary information for the policy evaluation.

show abstract

Realtime Data Processing at Facebook

Cited by 89 publications

References 19 publications

A Scalable and Robust Framework for Data Stream Ingestion

A Scalable and Robust Framework for Data Stream Ingestion

Detecting Irregular Patterns in IoT Streaming Data for Fall Detection

Access Control with Delegated Authorization Policy Evaluation for Data-Driven Microservice Workflows

Contact Info

Product

Resources

About