How we architected our backtesting engine to validate strategies faster than ever before.
The Speed Imperative
Strategy development is an iterative process. You hypothesize, implement, test, analyze, and refine — often hundreds of times before arriving at a production-ready strategy. If each backtest takes hours, you can only test a handful of ideas per day. But if each backtest takes seconds, you can explore thousands of variations and find optimal parameters with unprecedented thoroughness.
Architecture Decisions
Memory-Mapped Data Files
Instead of reading market data from a database or CSV files, we store our historical data in a custom binary format that can be memory-mapped directly into the process address space. This eliminates all I/O overhead and allows the operating system to handle caching automatically.
Columnar Storage
Following the design principles of modern analytical databases, we store each data field (open, high, low, close, volume) in separate contiguous arrays. This maximizes CPU cache utilization when computing indicators that only need a subset of fields.
Lock-Free Event Processing
Our event processing pipeline uses lock-free ring buffers inspired by the LMAX Disruptor pattern. Strategy callbacks receive market events without any mutex contention, enabling true parallel execution across cores.
Realistic Simulation
Speed means nothing if the simulation isn't realistic. Our engine includes sophisticated models for slippage based on historical order book depth, variable exchange fees by tier, network latency simulation, and partial fill modeling based on available liquidity at each price level.
Parameter Optimization
With 100k+ bars per second throughput, we can exhaustively search large parameter spaces. A strategy with 5 parameters, each with 20 possible values, creates 3.2 million combinations. At our processing speed, we can evaluate all of them in under 30 minutes on a 16-core machine.
Looking Forward
We're currently adding support for multi-asset portfolio backtesting with cross-asset correlation modeling, which will enable testing of our more sophisticated multi-agent strategies in a realistic portfolio context.