A Backtesting Checklist for Price-Forecasting Models

Every few weeks I evaluate a new forecasting model for the crop futures project, and every time I re-learn the same lesson: the interesting bugs are in the backtest, not the model. These are the checks I run before believing a Sharpe ratio. This note doubles as a demo post for the LaTeX-to-blog converter.

The one-line trap

A Sharpe ratio is easy to compute and easy to fool yourself with:

$$S = \frac{\mu - r_f}{\sigma}\sqrt{252},$$

where $\mu$ is the mean daily return of the strategy and $\sigma$ its standard deviation. The $\sqrt{252}$ annualizes from daily data --- but only if your returns are actually daily. Resample before computing, not after.

The annualization factor for a different sampling frequency $k$ is $\sqrt{k}$, and the attention layer I use for combining feature streams is

$$\begin{aligned} \text{Att}(Q, K, V) &= \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \\ \text{ctx}_t &= \sum_i \alpha_{t,i}\, v_i, \qquad \alpha_{t,i} \ge 0,\ \sum_i \alpha_{t,i} = 1. \end{aligned}$$

The pipeline under test

The system under test looks like this --- three data streams, one shared feature layer, and a forecasting head:

The checklist

Data hygiene

• No lookahead. Every feature at time $t$ must use data available at time $t$. NOAA revisions are the classic silent killer.
• Scale with train-fold statistics only --- normalizing with full-dataset means leaks the future into the past.
• Write the splits down. If you can't state the train/val/test dates from memory, the backtest is already suspect.

Evaluation honesty

1. Report the out-of-sample period length --- 2 years minimum before quoting a Sharpe with a straight face.
2. Compare against a naive baseline first:

• last-value carry
• seasonal mean
• linear regression on the same features

1. Re-run with three seeds; a strategy that only works with one initialization is a random number generator with good PR.

Feature cadence

A note on compute

The fine-tuning budget rule I use: if a full sweep doesn't fit in the weekend, the search space is too big. Quoting the greats: garbage in, garbage out --- and see the repo for the current setup.

Verdict

If the model survives all of the above, it earns the right to be wrong in production. That's the deal.

$ python backtest.py --model attn_lstm --oos 2024-01-01:2026-01-01
sharpe: 5.9 (baseline patchtst: 5.1)