Fast order book building and consolidation with SciDB

Quants, system developers, traders, and compliance professionals use SciDB to help solve lots of difficult problems. All these roles benefit from SciDB’s ability to quickly build an order book or consolidate order books across multiple exchanges, symbols, and time periods of interest.

An order book is a key tool for traders trying to profile market liquidity by time of day or symbol to help optimize their execution strategies or understand how their trades have affected the market.

Changes in the state of the order books are the key source of valuation information for quants and algo traders. They also need to test and validate trading strategies based on an understanding of how their algos will affect price and liquidity. The ability to quickly simulate trading strategy effects on the consolidated order book using various time periods and subsets of symbols is imperative.

Compliance departments need to demonstrate best execution as mandated by Reg NMS. This requires building an order book weeks or months after a particular trade was executed. SciDB’s native array data model with in-database analytics, ACID guarantees, and blindingly fast windowed aggregates is an ideal fit for this problem. But to prove our claim, we ran some benchmarks using the Paradigm4 Order Book Aggregator.

Unlike blackbox systems, you can download the open source Community Edition of SciDB and reproduce these results on-premise or in the cloud. Once you’ve installed SciDB, download the Order Book Aggregator on our GitHub page and see how fast SciDB produces a consolidated order book for requested symbols, exchanges and time periods.


Building an order book

We downloaded a single day of ARCA order data containing 212,746,543 orders and 8,000 symbols, and ran our orderbook-example to generate a book. To give you a feel for SciDB scalability we ran this using both 8 and 16 SciDB instances on a single Linux server with 16 cores.

We built order books to depths of one, five, ten, and 100 orders; and found no material difference in runtime. You get essentially unlimited depth with SciDB without sacrificing performance.


Consolidating multiple order books

SciDB may also be used to consolidate individual order books across multiple exchanges. We consolidated nine exchanges; 358,000,000 events and 8,000 symbols using an order book depth of ten. Using the same Linux server, we created the consolidated order book in a matter of minutes—beating a well known competing solution by a matter of hours.

EDITOR COMMENT: We have been getting a lot of requests for more specific comparisons. We ran our first-cut code (unoptimized) in 21 minutes on our cluster. That’s the blue bar below on the right. This is compared to what brand K ran at a customer in 8 hours (4 hours after optimization). The brand K hardware was 4 CPUs,  20 cores per CPU, 2.4GHz, 512GB memory, NAS disk. Our hardware specs are below.

Order book benchmark chart ~300X200


We ran these bencmarks on a low-cost commodity Linux server running Ubuntu 12.04, with sixteen Intel Xeon 2.0 GHz processors (CPU E5-2650 0), and 128GB total RAM. Each SciDB instance had a dedicated spinning disk.

Want to learn more? Check out our white paper on how SciDB can help with your computational finance problems.