Towards High Performance Processing of Streaming Data in Large Data Centers

Kamburugamuve, Supun; Ekanayake, Saliya; Pathirage, Milinda; Fox, Geoffrey

doi:10.1109/ipdpsw.2016.103

Cited by 17 publications

(10 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These challenges include the development of new, scalable and efficient streaming algorithms, programming models, languages and runtime systems for this type of applications. This requires considerable investment in the DaSP software stack to accomodate the increasing volume of streaming data [66].…”

Section: Parallel Pattern Interface Extensions For Distributed Architmentioning

confidence: 99%

Exploring stream parallel patterns in distributed MPI environments

et al. 2019

View full text Add to dashboard Cite

En primer lugar, a mis padres. Os debo todo lo que soy y seré. Gracias por el cuidado y educación recibidos, por los buenos (y malos) momentos, por aquellos días de piscina en Playa de Madrid, por las barbacoas en la Fundación del Lesionado Medular/ASPAYM Madrid, por las comidas en "La Bodeguita" y en el Centro Municipal de Mayores Ascao. Gracias a vosotros aprendí a ser fuerte. A mamá, porque me enseñaste lo más importante, a sonreir a pesar de la adversidad, y otras cosas de menor importancia, como mantener un hogar. A papá, por enseñarme gran parte de lo que sé, el equilibrio, e iluminarme el camino; aunque fuera de forma accidental, sin tí, mi vocación sería otra. Nunca os lo podré agradecer tanto como lo merecisteis. D.E.P.En segundo lugar, a mis compañeros de trabajo: Manu, Javi Doc, y David. Por todos los momentos compartidos en el laboratorio y fuera de él; por ayudarme y presionarme para que trabajara. Sin vosotros, dudo que estuviera escribiendo estas líneas ahora. A mis compañeros de laboratorio: Guille, Mario, Javi Prieto, y José Cabrero, por las charlas y momentos de risa. A Alejandro Calderón, que siempre ha estado ahí para escucharme y echar una mano. A José Daniel, por darme la oportunadad de trabajar en el grupo y por estar ahí en los momentos difíciles que últimamente han acontecido. También a mis compañeros de trabajo del CERN: Enric, Stephan, Guilherme, Bertrand, Javi, Emmanouil "Manos", y especialmente a Axel, Jakob y Pere, por acogerme en ROOT.No voy a terminar sin mencionar a las personas que siempre estuvieron ahí (no, no me olvido de vosotros): a mi hermano, a Úrsula, a mi tía Paloma y mi tío Matías, a mi primo "el niño".

show abstract

Section: Parallel Pattern Interface Extensions For Distributed Architmentioning

confidence: 99%

Exploring stream parallel patterns in distributed MPI environments

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Optimizing Streaming on HPC: The ability to leverage HPC hardware and software capabilities to optimize Big Data frameworks has been extensively explored. Kamburugamuve et al [25] propose the usage of optimized HPC algorithms for low-latency communication (e. g. trees) and scheduling of tasks to enhance distributed stream processing in the Apache Storm framework [18]. In [26] they investigate the usage of HPC network technology, such as Infiniband and Omnipath, to optimize the interprocess communication system of Heron [19], the successor of Storm.…”

Section: B Related Workmentioning

confidence: 99%

Pilot-Streaming: A Stream Processing Framework for High-Performance Computing

Luckow¹,

Chantzialexiou²,

Jha³

2018

Preprint

View full text Add to dashboard Cite

An increasing number of scientific applications utilize stream processing to analyze data feeds of scientific instruments, sensors, and simulations. In this paper, we study the streaming and data processing requirements of light source experiments, which are projected to generate data at 20 GB/sec in the near future. As beamtimes available to users are typically short, it is essential that processing and analysis can be conducted in a streaming mode. The development and deployment of streaming applications is a complex task and requires the integration of heterogeneous, distributed infrastructure, frameworks, middleware and application components written in different languages and abstractions. Streaming applications may be extremely dynamic due to factors, such as variable data rates, network congestions, and application-specific characteristics, such as adaptive sampling techniques and the different processing techniques. Consequently, streaming system are often subject to back-pressures and instabilities requiring additional infrastructure to mitigate these issues. We propose Pilot-Streaming, a framework for supporting streaming applications and their resource management needs on HPC infrastructure. Underlying Pilot-Streaming is a unifying architecture that decouples important concerns and functions, such as message brokering, transport and communication, and processing. Pilot-Streaming simplifies the deployment of stream processing frameworks, such as Kafka and Spark Streaming, while providing a high-level abstraction for managing streaming infrastructure, e. g. adding/removing resources as required by the application at runtime. This capability is critical for balancing complex streaming pipelines. To address the complexity in the development of streaming applications, we present the Streaming Mini-Apps, which supports different plug-able algorithms for data generation and processing, e. g., for reconstructing light source images using different techniques. We use the streaming Mini-Apps to evaluate the Pilot-Streaming framework demonstrating its suitability for different use cases and workloads.

show abstract

“…The high performance stream processing paper [13] describes a novel approach of using shared memory maps within Apache Storm. This is similar to SPIDAL Java's intra-node implementation except it implements a custom memory mapped based queue to coordinate between workers within a node.…”

Section: Related Workmentioning

confidence: 99%

SPIDAL Java: High Performance Data Analytics with Java and MPI on Large Multicore HPC Clusters

Ekanayake

Kamburugamuve

Fox

2016

24th High Performance Computing Symposium

View full text Add to dashboard Cite

Within the last few years, there have been significant contributions to Java-based big data frameworks and libraries such as Apache Hadoop, Spark, and Storm. While these systems are rich in interoperability and features, developing high performance big data analytic applications is challenging. Also, the study of performance characteristics and high performance optimizations is lacking in the literature for these applications. By contrast, these features are well documented in the High Performance Computing (HPC) domain and some of the techniques have potential performance benefits in the big data domain as well. This paper presents the implementation of a high performance big data analytics library-SPIDAL Java-with a comprehensive discussion on five performance challenges, solutions, and speedup results. SPIDAL Java captures a class of global machine learning applications with significant computation and communication that can serve as a yardstick in studying performance bottlenecks with Java big data analytics. The five challenges present here are the cost of intra-node messaging, inefficient cache utilization, performance costs with threads, overhead of garbage collection, and the costs of heap allocated objects. SPIDAL Java presents its solutions to these and demonstrates significant performance gains and scalability when running on up to 3072 cores in one of the latest Intel Haswell-based multicore clusters.

show abstract

Towards High Performance Processing of Streaming Data in Large Data Centers

Cited by 17 publications

References 20 publications

Exploring stream parallel patterns in distributed MPI environments

Exploring stream parallel patterns in distributed MPI environments

Pilot-Streaming: A Stream Processing Framework for High-Performance Computing

SPIDAL Java: High Performance Data Analytics with Java and MPI on Large Multicore HPC Clusters

Contact Info

Product

Resources

About