Alchemy: Distributed financial quantitative analysis system with high‐level programming model

Abstract

Nowadays, the financial securities investment and transaction are processed digitally. The securities companies have accumulated a large amount of financial data. Given that, quantitative analysis has become a common method for securities investors. However, as the scale of data continues to grow, data storing and processing have increasingly become a big challenge to financial quantitative analysts. First, most existing stand‐alone quantitative analysis systems are hard to accelerate data processing in a distributed way. Second, existing distributed computing frameworks such as Apache Spark demands financial quantitative analysts with expertise knowledge on distributed computing. Third, transition from traditional stand‐alone financial quantitative analysis (FQA) system to such distributed computing system often introduces learning efforts and development overheads. To solve these problems, we propose Alchemy, a distributed processing platform tailored for FQA with high‐level programming interface. Alchemy offers distributed computing service and provides a pipeline‐based programming model which shares many similarities with the traditional stand‐alone systems, reducing learning efforts for financial quantitative analysts. The performance of Alchemy is evaluated in both experimental environment and a real‐world production environment in Huatai Securites which is a leading financial company in China. The results show that Alchemy is able to achieve up to 300 times speedup compared against existing financial quantitative analysis systems.