Many "big data" applications need to act on data arriving in real time. However, current programming models for distributed stream processing are relatively low-level, often leaving the user to worry about consistency of state across the system and fault recovery. Furthermore, the models that provide fault recovery do so in an expensive manner, requiring either hot replication or long recovery times. We propose a new programming model, discretized streams (D-Streams), that offers a high-level functional API, strong consistency, and efficient fault recovery. D-Streams support a new recovery mechanism that improves efficiency over the traditional replication and upstream backup schemes in streaming databasesparallel recovery of lost state-and unlike previous systems, also mitigate stragglers. We implement D-Streams as an extension to the Spark cluster computing engine that lets users seamlessly intermix streaming, batch and interactive queries. Our system can process over 60 million records/second at sub-second latency on 100 nodes.

In this paper, we combine the most complete record of daily mobility, based on large-scale mobile phone data, with detailed Geographic Information System (GIS) data, uncovering previously hidden patterns in urban road usage. We find that the major usage of each road segment can be traced to its own -surprisingly few -driver sources. Based on this finding we propose a network of road usage by defining a bipartite network framework, demonstrating that in contrast to traditional approaches, which define road importance solely by topological measures, the role of a road segment depends on both: its betweeness and its degree in the road usage network. Moreover, our ability to pinpoint the few driver sources contributing to the major traffic flow allows us to create a strategy that achieves a significant reduction of the travel time across the entire road system, compared to a benchmark approach.I n an era of unprecedented global urbanization, society faces a rapidly accelerating demand for mobility, placing immense pressure on urban road networks 1,2 . This demand manifests in the form of severe traffic congestion 3,4 , which decreases the roads' level of service, while at the same time increasing both fuel consumption 5 and traffic-related air pollution 6 . In 2007 alone, congestion forced Americans living in urban areas to travel 4.2 billion hours more, purchase an additional 2.8 billion gallons of fuel, at a total cost of $87.2 billion 3 . To mitigate congestion in urban roads, urban planning 1 , traffic prediction 7-9 and the study of complex networks [10][11][12][13][14][15] have been widely investigated potential influencing factors. However, without comprehensive knowledge of how roads are used dynamically, these studies are conventionally based on expensive and quickly outdated travel surveys or segmented information on traffic flow and travel time 7-9 , which fail to support the researchers with the information needed to cope with modern mobility demand. Up to now our understanding of the origins of the drivers in each road remained limited and not quantitatively solved.In this work we validate for the first time a methodology, which employs comprehensive mobile phone data to detect patterns of road usage and the origins of the drivers. Thus, providing a basis for better informed transportation planning, including targeted strategies to mitigate congestion 3,4 . We formalize the problem by counting the observed number of individuals moving from one location to another, which we put forward as the transient origin destination (t-OD) matrix (Fig. S5, Fig. S11 and Supplementary Information (SI) section II.A). Traditionally, ODs are costly and difficult to obtain, because they are at best based on travel diaries made every few years, which quickly become obsolete and strongly rely on provided reports 7,8 . In contrast, the rapidly increasing penetration rates and massive usage of mobile phones, with towers densely located in urban areas, can provide the most detailed information on daily human mobility [16][17][...

No abstract

Abstract-We consider the problem of reconstructing vehicle trajectories from sparse sequences of GPS points, for which the sampling interval is between 10 seconds and 2 minutes. We introduce a new class of algorithms, called altogether path inference filter (PIF), that maps GPS data in real time, for a variety of trade-offs and scenarios, and with a high throughput. Numerous prior approaches in map-matching can be shown to be special cases of the path inference filter presented in this article. We present an efficient procedure for automatically training the filter on new data, with or without ground truth observations. The framework is evaluated on a large San Francisco taxi dataset and is shown to improve upon the current state of the art. This filter also provides insights about driving patterns of drivers. The path inference filter has been deployed at an industrial scale inside the Mobile Millennium traffic information system, and is used to map fleets of data in San Francisco, Sacramento, Stockholm and Porto.

No abstract

We consider the problem of reconstructing vehicle trajectories from sparse sequences of GPS points, for which the sampling interval is between 10 seconds and 2 minutes. We introduce a new class of algorithms, called altogether path inference filter (PIF), that maps GPS data in real time, for a variety of trade-offs and scenarios, and with a high throughput.Numerous prior approaches in map-matching can be shown to be special cases of the path inference filter presented in this article. We present an efficient procedure for automatically training the filter on new data, with or without ground truth observations. The framework is evaluated on a large San Francisco taxi dataset and is shown to improve upon the current state of the art. This filter also provides insights about driving patterns of drivers. The path inference filter has been deployed at an industrial scale inside the Mobile Millennium traffic information system, and is used to map fleets of data in San Francisco, Sacramento, Stockholm and Porto.

No abstract

Many "big data" applications must act on data in real time. Running these applications at ever-larger scales requires parallel platforms that automatically handle faults and stragglers. Unfortunately, current distributed stream processing models provide fault recovery in an expensive manner, requiring hot replication or long recovery times, and do not handle stragglers. We propose a new processing model, discretized streams (D-Streams), that overcomes these challenges. D-Streams enable a parallel recovery mechanism that improves efficiency over traditional replication and backup schemes, and tolerates stragglers. We show that they support a rich set of operators while attaining high per-node throughput similar to single-node systems, linear scaling to 100 nodes, subsecond latency, and sub-second fault recovery. Finally, D-Streams can easily be composed with batch and interactive query models like MapReduce, enabling rich applications that combine these modes. We implement D-Streams in a system called Spark Streaming.