A tick database can be the foundation for improving trading algorithms and supporting better informed trades. But timeliness of data and keeping up with market feeds can be a challenge and are essential to deriving value from the database.
We recently were engaged by a proprietary trading company to build such a tick database based on the Paradigm4 data processing and analytics stack. SciDB’s scale-out architecture met this challenge, improving the timeliness of data used to fine-tune trading algorithms and increasing the amount of data used to drive trade decisions.
The deployment ingests 1TB of option and equity data (~800,000 financial instruments, 12 exchanges, at millisecond resolution) per 7.5 hour trading day. The average ingest rate is a modest 20-40 MB/second but most of the data arrives in the first and last few minutes of the trading day—making the peak ingest rate over 1 GB/second. The system keeps up with the peak ingest while responding to queries interactively.
After ingesting, the data has to be indexed in order to support a variety of queries—e.g., selecting portions of the day quickly, and aggregating the data into different time granularities. The trading company wanted to provide information to:
- Quants to feed back-testing processes, which are used to improve trading algorithms;
- Traders—in near-real time (a few seconds to a minute)—to inform trading decisions.
In addition to these benefits, this use case is particularly interesting because the customer wanted to replace an industry-leading quant finance platform with a faster one that could be programmed by their traders and quants rather than by hard to find programming talent. SciDB delivered with its MAC™ technology, massively parallel processing architecture, and analyst-friendly programming languages enabling their:
- IT staff to scale out and make things run as fast as needed—without the stress of right-sizing up front
- Quants and traders to self-service their data management needs—without hiring specialized programmers