Monitoring Streams — A New Class of Data Management Applications

Carney, Don; Çetintemel, Uǧur; Cherniack, Mitch; Convey, Christian; Lee, Sang−Don; Seidman, Greg; Stonebraker, Michael; Tatbul, Nesime; Zdonik, Stanley B.

doi:10.1016/b978-155860869-6/50027-5

Cited by 591 publications

(487 citation statements)

References 7 publications

Supporting

Mentioning

483

Contrasting

Unclassified

Order By: Relevance

“…It is functionally similar to a continuous data stream query processor [5,7,12,15,16], a publish/subscribe system [25], or an event processing system [3,13,14,38,44]. However, there are significant differences.…”

Section: Techniques For the Subscription Matchermentioning

confidence: 99%

“…It supports on-line extraction of relevant information from streams in real-time, and ... Hence the CKB will ultimately combine the functionality of data warehouses [34] with that of data stream management systems (STREAM [5,7], Aurora [12], TelegraphCQ [15], NiagaraCQ [16]) and sensor database systems [9,37]. In the following sections we describe the components of the Cornell Knowledge Broker in more detail.…”

Section: The Cornell Knowledge Brokermentioning

confidence: 99%

See 1 more Smart Citation

The Architecture of the Cornell Knowledge Broker

Demers

Gehrke

Riedewald

2004

Intelligence and Security Informatics

View full text Add to dashboard Cite

Abstract.Intelligence applications have to process massive amounts of data in order to extract relevant information. This includes archived historical data as well as continuously arriving new data. We propose a novel architecture that addresses this problem -the Cornell Knowledge Broker. It will not only support knowledge discovery, but also security, privacy, information exchange, and collaboration.

show abstract

Section: Techniques For the Subscription Matchermentioning

confidence: 99%

Section: The Cornell Knowledge Brokermentioning

confidence: 99%

The Architecture of the Cornell Knowledge Broker

Demers

Gehrke

Riedewald

2004

Intelligence and Security Informatics

View full text Add to dashboard Cite

show abstract

“…It does not rely on any specific processing model. There are different singlenode processing models that are currently under development such as TelegraphCQ [6], Aurora [4] and STREAM [1]. Our system is not restricted to any processing model because it separates the stream processing engine in each node from the distributed processing details.…”

Section: Problem Analysismentioning

confidence: 99%

Efficient Dynamic Operator Placement in a Locally Distributed Continuous Query System

Zhou

Ooi

Tan

et al. 2006

On the Move to Meaningful Internet Systems 2006: CoopIS, DOA, GADA, and ODBASE

View full text Add to dashboard Cite

Abstract. In a distributed processing environment, the static placement of query operators may result in unsatisfactory system performance due to unpredictable factors such as changes of servers' load, data arrival rates, etc. The problem is exacerbated for continuous (and long running) monitoring queries over data streams as any suboptimal placement will affect the system for a very long time. In this paper, we formalize and analyze the operator placement problem in the context of a locally distributed continuous query system. We also propose a solution, that is asynchronous and local, to dynamically manage the load across the system nodes. Essentially, during runtime, we migrate query operators/fragments from overloaded nodes to lightly loaded ones to achieve better performance. Heuristics are also proposed to maintain good data flow locality. Results of a performance study shows the effectiveness of our technique.

show abstract

“…There has been a lot of work with managing the data transmission to work around bottlenecks and to annotate and select meaningful data from the stream. Many of these [15,7,4] use a centralized site for applying selection or annotation operations. More scalable solutions, for example [16,14], use in-network aggregation and distributed deployment of analysis routines.…”

Section: Related Workmentioning

confidence: 99%

Advanced I/O for large-scale scientific applications.

Klasky

Schwan

Oldfield³

et al. 2010

View full text Add to dashboard Cite

As scientific simulations scale to use petascale machines and beyond, the data volumes generated pose a dual problem. First, with increasing machine sizes, the careful tuning of IO routines becomes more and more important to keep the time spent in IO acceptable. It is not uncommon, for instance, to have 20% of an application's runtime spent performing IO in a 'tuned' system. Careful management of the IO routines can move that to 5% or even less in some cases. Second, the data volumes are so large, on the order of 10s to 100s of TB, that trying to discover the scientifically valid contributions requires assistance at runtime to both organize and annotate the data. Waiting for offline processing is not feasible due both to the impact on the IO system and the time required. To reduce this load and improve the ability of scientists to use the large amounts of data being produced, new techniques for data management are required. First, there is a need for techniques for efficient movement of data from the 3 compute space to storage. These techniques should understand the underlying system infrastructure and adapt to changing system conditions. Technologies include aggregation networks, data staging nodes for a closer parity to the IO subsystem, and autonomic IO routines that can detect system bottlenecks and choose different approaches, such as splitting the output into multiple targets, staggering output processes. Such methods must be end-to-end, meaning that even with properly managed asynchronous techniques, it is still essential to properly manage the later synchronous interaction with the storage system to maintain acceptable performance. Second, for the data being generated, annotations and other metadata must be incorporated to help the scientist understand output data for the simulation run as a whole, to select data and data features without concern for what files or other storage technologies were employed. All of these features should be attained while maintaining a simple deployment for the science code and eliminating the need for allocation of additional computational resources.4 Acknowledgment

show abstract

Monitoring Streams — A New Class of Data Management Applications

Cited by 591 publications

References 7 publications

The Architecture of the Cornell Knowledge Broker

The Architecture of the Cornell Knowledge Broker

Efficient Dynamic Operator Placement in a Locally Distributed Continuous Query System

Advanced I/O for large-scale scientific applications.

Contact Info

Product

Resources

About