Large-Scale Estimation in Cyberphysical Systems Using Streaming Data: A Case Study With Arterial Traffic Estimation

Abstract

Controlling and analyzing cyberphysical and robotics systems is increasingly becoming a Big Data challenge. Pushing this data to, and processing in the cloud is more efficient than on-board processing. However, current cloud-based solutions are not suitable for the latency requirements of these applications. We present a new concept, Discretized Streams or D-Streams, that enables massively scalable computations on streaming data with latencies as short as a second. We experiment with an implementation of D-Streams on top of the Spark computing framework. We demonstrate the usefulness of this concept with a novel algorithm to estimate vehicular traffic in urban networks. Our online EM algorithm can estimate traffic on a very large city network (the San Francisco Bay Area) by processing tens of thousands of observations per second, with a latency of a few seconds.Note to Practitioners This work was driven by the need to estimate vehicular traffic at a large scale, in an online setting, using commodity hardware. Machine Learning algorithms combined with streaming data are not new, but it still requires deep expertise both in Machine Learning and in Computer Systems to achieve large scale computations in a tractable manner. The Streaming Spark project aims at providing an interface that abstracts out all the technical details of the computation platform (cloud, HPC, workstation, etc.).As shown in this work, Streaming Spark is suitable for implementing and calibrating non-trivial algorithms on a large cluster, and provides an intuitive yet powerful programming interface. The readers are invited to refer to the source code referred in this article for more examples.This article presents algorithms to sample and compute densities for Gamma random variables restricted to a hyperplane (i.e. distributions of the form Ti| j αjTj = d with Tj independant Gamma distributions). It is common in this case to use Gaussian random variables because of closed form solutions to solve. If one considers positive valued distributions with heavy tails, our formulas using gamma distributions may be more suitable.

Modeling and analytics for cyber-physical systems in the age of big data

“…The data deluge caused by 24x7 sensing in CPS is making this situation worse: our ability to measure at finer spatial (e.g. component level) and temporal granularity keeps improving; however the CPS modeling methods lag behind (with a few notable exceptions [19]). …”

“…Hunter et al demonstrate the need to tackle massive amount of high-dimensional data with low information content collected by CPS sensors in real time [19]. With the rise of industrial big data [5], and smart infrastructure (cities, transportation, homes, etc.…”

“…Smart devices with GPS (e.g. smartphones or GPS on taxis) can provide good coverage but their data quality is low [19].…”

“…Traditionally, the design of the sensing component of a CPS focused on how to deploy a limited number of sophisticated, reliable, and expensive sensors to optimize coverage of an environment or physical phenomenon [19]. However, advances in sensing and communication technologies over the past 10-15 years have disrupted the traditional way.…”

“…Today, low cost, ubiquitous sensing is driving a paradigm shift away from resource constrained sensing towards using big data analytics to extract information and actionable intelligence from massive amount of sensor data [8,19]. In this paper argue that the availability of big monitoring data on CPS creates challenges for traditional CPS modeling by breaking some of the assumptions, but also provides opportunities to simplify the CPS model identification task, achieve proactive maintenance, and build novel applications by combining CPS modeling with data driven learning and mining techniques.…”