Faster FAST: multicore acceleration of streaming financial data

Abstract

By 2010, the global options and equity markets will average over 128 billion messages per day, amounting to trillions of dollars in trades. Trading systems, the backbone of the low-latency high-frequency business, need fundamental research and innovation to overcome their current processing bottlenecks. With market data rates rapidly growing, the financial community is demanding solutions that are extremely fast, flexible, adaptive, and easy to manage. This paper explores multiple avenues to deal with the decoding and normalization of Option Price Reporting Authority (OPRA) stock market data feeds encoded with FIX Adapted for Streaming (FAST) representation, on commodity multicore platforms, and describes a novel solution that encodes the OPRA protocol with a high-level description language. Our algorithm achieves a processing rate of 15 million messages per second in the fastest single socket configuration on an Intel Xeon E5472, which is an order of magnitude higher than the current needs of the financial systems. We also present an in-depth performance evaluation that exposes important properties of our OPRA parsing algorithm on a collection of multicore processors.

Streaming, low-latency communication in on-line trading systems

“…The proposed system relies on previous work [2] and integrates the OPRA decoder in a fully functional system that includes the network communication between processing nodes. In fact, the network and the network stack play an important role, and are increasingly becoming the bottleneck in systems that can handle the OPRA protocol operating at the sub-microsecond level.…”

“…Thanks to our high-level compilative approach based on DotStar our system can be easily retargeted to dynamically changing OPRA formats, as well as other market data feeds [2].…”

Ultra low latency market data feed on IBM PowerENTM

“…The implementation uses only a fraction of the resources available in PowerEN TM and "leaves" enough room for trading analytics and/or algorithms. This software leverages the high efficient protocol parsing and decoding techniques described in [2] as well as several unique features of PowerEN TM . The results show an average processing latency of 6.6 µs for OPRA updates and a throughput capability of more than 16 million updates per second with minimal jitter.…”

“…Each row represents a message field and columns contain each individual message in a packet. The solution handles data decoding using the techniques previously described in [2,7]. For example, it avoids all branches and category specific routines and employs a single function that uses a category bitmap that identifies required fields.…”

Scalable XML query processing using parallel pushdown transducers

“…For example, the full "Firehose" stream from Twitter produces XML data at a rate of tens of megabytes per second [18] and is likely to grow significantly in the future. In other domains, such as web analytics, financial data processing, cellular network operations or real-time telematics, stream data rates of 10s or 100s of millions of items per second are not uncommon [1]. Even with static XML datasets, the advent of "big data" means that a single-pass stream processing model becomes the only viable choice when faced with processing 10s of terabytes or petabytes of data generated, for example, by community-driven websites such as Twitter or Wikipedia [32].…”