Design a Financial Trading Backtest System

Problem statement

Design a financial trading backtest platform that loads historical price ticks or full order book data, replays them in exact chronological order, applies strategy rules against that replayed stream, tracks positions, P&L, drawdowns and risk metrics, optionally compares many strategies, and produces evaluation reports. The platform must serve quant researchers who iterate quickly, data engineers who own petabyte-scale tick archives, and portfolio managers who compare strategy candidates before allocating capital.

A backtest is not a query and not a training job. It is a deterministic simulation of a hypothetical trading firm operating inside a recorded history. The engine must feed the strategy only what was knowable at each historical instant, execute its orders through a realistic model of the market, and account for every share, contract, fee and borrow cost. Get one of those wrong and the output is a confident lie: strategies that look profitable in simulation and lose money on day one of live trading.

Why this is a Big Data problem

Consider US equities alone. The consolidated feeds (CTS/UTP SIP) are reported to publish on the order of 10 billion quote and trade messages per trading day, and NASDAQ reports that its TotalView-ITCH feed carries tens of billions of messages per day across its product family. There are about 252 trading days per year and a regular session of 390 minutes. A platform that keeps 10 years of consolidated trades and quotes is therefore reasoning about roughly 25 trillion events and hundreds of terabytes after compression. Every backtest run is a scan over a slice of that archive, every parameter sweep multiplies that scan by hundreds or thousands, and every researcher wants results in minutes, not overnight.

That produces five structural pressures:

1. Scan bandwidth: reading symbol-and-date slices at hundreds of MB/s to GB/s per worker with column pruning and predicate pushdown.
2. Ordering fidelity: events must reach the strategy in true exchange-time order per symbol, with microsecond or nanosecond precision, even when the underlying storage is distributed.
3. Simulation correctness: no look-ahead, no survivorship bias, corporate actions applied point-in-time, fills modeled against the book that actually existed.
4. Massive parallelism: thousands of concurrent backtests, parameter sweeps, and walk-forward windows with fair scheduling and cost control.
5. Reproducibility: the same code, config, and data version must reproduce the same result to the last basis point, months later, for compliance review.

Vectorized versus event-driven

A weak answer jumps straight to pandas. Vectorized bar-level backtests (NumPy over daily or minute bars) are great for coarse signal research, but they cannot honestly model queue position, partial fills, order latency, intraday risk limits, or strategies whose actions change the prices they would have received. An event-driven engine processes a chronologically ordered event stream - market data, order submissions, fills, timers, corporate actions - and is the only architecture that supports the brief's tick replay requirement. Production platforms do both: vectorized research to narrow the hypothesis space, event-driven backtests to validate it. This design focuses on the event-driven core with a vectorized fast path as a secondary capability.

The four architectural planes

1. Data plane: ingestion, normalization, adjustment, versioned storage of ticks, bars, order books, reference data and corporate actions.
2. Simulation plane: deterministic event replay, portfolio accounting, execution modeling, metric accumulation.
3. Orchestration plane: run lifecycle, task fan-out for sweeps, scheduling, caching, worker fleet management.
4. Analytics plane: metric aggregation, cross-strategy comparison, reports, notebooks, and governance evidence.

A strong interview answer keeps these planes separate: the simulation plane must be deterministic and re-runnable, the data plane immutable and versioned, the orchestration plane allowed to fail and retry, and the analytics plane free to be eventually consistent.