A Parallel Workflow for Real-time Correlation and Clustering of High-Frequency Stock Market Data

Rostoker, Camilo; Wagner, Alan; Hoos, Holger H.

doi:10.1109/ipdps.2007.370216

Cited by 9 publications

(4 citation statements)

References 18 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The need to continuously adapt to the evolution of the graph, without the overhead of re-loading an updated snapshot of the graph or re-partitioning from scratch, has been recently reported in practical [31,22,12] and more theoretical [24] studies. However, the previously mentioned techniques cannot handle structural graph changes, either degrading partition quality, or fully triggering the partition process.…”

Section: Related Workmentioning

confidence: 99%

Adaptive Partitioning for Large-Scale Dynamic Graphs

Vaquero

Cuadrado

Logothetis

et al. 2014

2014 IEEE 34th International Conference on Distributed Computing Systems

View full text Add to dashboard Cite

Mining large graphs is critical for several real-world systems such as social networks. Such graphs typically consist of hundreds of millions of vertices and edges and are highly dynamic, with their structure continuously evolving over time. The current data deluge is making this problem even harder: bigger graphs must be processed even faster than before. Appropriate graph partitioning is crucial to scale to large graphs and reduce processing time. However, partitioning large dynamic graphs is challenging: current techniques either do not scale well as they require global graph knowledge or do not handle changes in the graph.In this work, we introduce a general purpose graph partitioning algorithm that adapts to dynamic structural changes in massive graphs. The algorithm works by iteratively adapting the partitions using only local information. We show how the application of our algorithm to three real-world scenarios can speed up the execution of graph analyses by over 50%, while adapting the partitioning to a high volume of changes to the graph.

show abstract

Section: Related Workmentioning

confidence: 99%

Adaptive Partitioning for Large-Scale Dynamic Graphs

Vaquero

Cuadrado

Logothetis

et al. 2014

2014 IEEE 34th International Conference on Distributed Computing Systems

View full text Add to dashboard Cite

show abstract

“…Beyond these fixed techniques for static graphs, the need to continuously adapt to changes to the graph structure without the overhead of re-loading an updated snapshot of the graph or re-partitioning from scratch, has been recently reported in practical [33,23,12] and more theoretical [26] studies. Few systems can cope with run time changes in structure [24,45,9].…”

Section: Changing Graph Structure At Runtimementioning

confidence: 99%

xDGP: A Dynamic Graph Processing System with Adaptive Partitioning

Vaquero,

Cuadrado,

Logothetis

et al. 2013

Preprint

View full text Add to dashboard Cite

Many real-world systems, such as social networks, rely on mining efficiently large graphs, with hundreds of millions of vertices and edges. This volume of information requires partitioning the graph across multiple nodes in a distributed system. This has a deep effect on performance, as traversing edges cut between partitions incurs a significant performance penalty due to the cost of communication. Thus, several systems in the literature have attempted to improve computational performance by enhancing graph partitioning, but they do not support another characteristic of realworld graphs: graphs are inherently dynamic, their topology evolves continuously, and subsequently the optimum partitioning also changes over time.In this work, we present the first system that dynamically repartitions massive graphs to adapt to structural changes. The system optimises graph partitioning to prevent performance degradation without using data replication. The system adopts an iterative vertex migration algorithm that relies on local information only, making complex coordination unnecessary. We show how the improvement in graph partitioning reduces execution time by over 50%, while adapting the partitioning to a large number of changes to the graph in three real-world scenarios.

show abstract

“…Rostoker et al [10] investigate multivariate analysis with the computation of clusters of correlated stocks in a market. They creates the complete correlation structure and then identify cliques in the graph based on correlation strength.…”

Section: Related Workmentioning

confidence: 99%

Multi-variate finance kernels in the Blue Gene supercomputer

Daly

Ryu

Moreira

2008

2008 Workshop on High Performance Computational Finance

View full text Add to dashboard Cite

Computational finance is an important application area for high-performance computing today. Large computational resources are used for a variety of operations related to securities and asset portfolios. For online operations, the focus has been both on reducing latency and improving the quality of the algorithms. This focus on latency has forced a predominance of univariate analysis simply from a feasibility perspective. In this paper, we demonstrate that current supercomputers, and in particular the Blue Gene family of supercomputers, enables the move to online multivariate analysis of entire markets. We use a simple but representative example of multivariate analysis, namely the computation of the correlation matrix, to explore that space. We show how the computation can be parallelized and run as an online real-time operation at the scale of thousands of securities and millions of events per second.

show abstract

A Parallel Workflow for Real-time Correlation and Clustering of High-Frequency Stock Market Data

Cited by 9 publications

References 18 publications

Adaptive Partitioning for Large-Scale Dynamic Graphs

Adaptive Partitioning for Large-Scale Dynamic Graphs

xDGP: A Dynamic Graph Processing System with Adaptive Partitioning

Multi-variate finance kernels in the Blue Gene supercomputer

Contact Info

Product

Resources

About