Strategy validation checklist: 7 checks before you trust a backtest
Run seven checks in a fixed order, cheapest first: data quality, costs, look-ahead bias, a time-ordered in-sample/out-of-sample split, walk-forward efficiency, Monte Carlo and parameter stress tests, and a minimum trade count. Write every pass/fail number before the first run, and stop a strategy at the first check it fails.
By Soren Huang · Published · updated · Editorial policy
Contents
- What is on a strategy validation checklist for a futures strategy?
- Checks 1 to 3: are the data, the costs and the execution order right before you read any result?
- Check 4: how should you split in-sample and out-of-sample data?
- Check 5: what walk-forward efficiency counts as a pass?
- Check 6: what do Monte Carlo and parameter stress tests add that one equity curve cannot?
- Check 7: how many trades does a backtest need, and which thresholds do you set beforehand?
- Why does the best backtest in a search still need all seven checks?
- Frequently asked questions
- References
What is on a strategy validation checklist for a futures strategy?
A strategy validation checklist for a futures strategy is seven checks run in a fixed order: data quality, costs, look-ahead bias, an in-sample/out-of-sample split, walk-forward analysis, Monte Carlo and parameter stress tests, and a minimum trade count with pass/fail thresholds written down before the first run. A strategy that fails one check stops there.
A strategy validation checklist is a fixed list of tests, applied in the same order to every candidate, with the pass line for each test written before the first result is read. Think of seven gates at an airport: a passenger who fails the passport check is not sent on to the baggage scanner to see whether that goes better. The order matters because the first three checks fix the inputs, and every later check reads results built on those inputs. The claim of this article is that the order and the pre-written numbers do more for you than any single test, including the slow, impressive ones.
| Check | What it catches | Time it takes |
|---|---|---|
| 1 Data quality | Unadjusted rolls, gaps, bad ticks, wrong time zone | Minutes |
| 2 Costs in | An edge smaller than the round-trip cost | Minutes |
| 3 Look-ahead check | Fills at prices the rule could not have had | Minutes |
| 4 In-sample / out-of-sample | Rules that describe one stretch of history | One extra run |
| 5 Walk-forward | Settings that stop working when re-chosen on new data | Many runs |
| 6 Monte Carlo and parameter stress | A lucky trade order; a setting with no good neighbours | Many runs |
| 7 Trade count and thresholds | A result built on too few trades; goalposts moved after the fact | One reading |
AlphaPrime (Valynox) works this way inside strategy validation: the thresholds are yours, set before the search, and a candidate that fails an early check does not go on to the slower ones.
Checks 1 to 3: are the data, the costs and the execution order right before you read any result?
Run the cheap checks first, because a bad roll, a missing round-trip cost or a rule that reads the close of the bar it trades on can turn a losing rule into a winning backtest, and no later check can repair that. These three checks take minutes and remove most false winners before any statistics are involved.
- Check 1, data quality. A continuous futures series is built by joining several contract months, and the price jump at each roll has to be adjusted, or a trend rule will trade a gap that nobody could trade. Look at the roll dates, the overnight gaps, the time zone of the bars and the holiday calendar. Pass when every roll is adjusted the same way, no bar sits far outside its neighbours, and the session hours match the hours you would trade.
- Check 2, costs. Put commission, exchange fees and slippage into every run from the start, per round trip (entry plus exit). Then compare the average trade after costs with the round-trip cost. A rule whose gross average trade is close to the cost of taking it has no edge, and it is usually the rule with the most trades.
- Check 3, look-ahead bias. Look-ahead bias is a backtest using information that was not available at the moment the rule decided. The most common leak is a rule that decides on the close of a bar and fills at that same close. The fix is next-bar execution: decide on the close, fill at the next open. The one-bar check: run the strategy with fills one bar later than the signal; if the result collapses, the rule was reading the future.
Whether your software shows roll adjustments, takes a per-side slippage and can force next-bar fills is worth knowing before any result; the six checks in backtesting software compared cover what to look for.
Check 4: how should you split in-sample and out-of-sample data?
Split by time, tune only on the earlier part, run the rules unchanged on the later part, and keep a final block of recent bars that nothing touches until one last test. In-sample data is the part of the history used to choose rules and settings. Out-of-sample data is the part set aside and used only to run the finished rules and read the result. Study material and exam paper have to be different pages.
- Pick the split dates before the first run and write them down. A common shape is about 70 percent in-sample, about 20 percent out-of-sample, and the most recent 10 percent reserved for one final test.
- Check that both sides contain a trend, a range and a volatile stretch. If the out-of-sample years are one long rise, a rule that is simply long passes for the wrong reason.
- Count the out-of-sample trades, not the years; check 7 explains why.
- Keep every setting frozen after you read the out-of-sample result. A changed setting makes that data part of the tuning, and the test is spent.
How much of the in-sample result should survive, and what to do with a result that is weaker but not broken, is covered in in-sample vs out-of-sample: how to split and how much is enough.
Check 5: what walk-forward efficiency counts as a pass?
Walk-forward efficiency is the out-of-sample profit per unit of time divided by the in-sample profit per unit of time, and a figure around half or better, read across all windows rather than the best one, is the common pass line that you write down before the run. Walk-forward analysis is a test that tunes settings on one window of history, runs them unchanged on the window that follows, moves both windows forward and repeats, until the joined out-of-sample pieces form one result. It turns one split into many and produces more out-of-sample trades from the same data.
Dividing by time is what makes the two sides comparable: a three-year in-sample window and a one-year out-of-sample window cannot be compared on total profit. Pardo (2008) describes the ratio; many systematic traders use about 50 percent as the floor. Read the number pooled across all windows. One weak window is normal, because markets change between windows. A pooled figure well under the line, a sign that flips from window to window, or a result that lives in one window is a fail.
Hypothetical example
Five rolling windows. Each in-sample window earns about 100 units per year. The out-of-sample windows earn 60, 45, 70, 20 and 55 units per year, so the efficiencies are 60, 45, 70, 20 and 55 percent. Pooled, 250 of 500 units survive, which is 50 percent and meets a pass line of 50 written before the run. Window four alone would fail; the pooled figure passes.
Window lengths, how many windows you need and how to read the joined curve are in walk-forward analysis: the one backtest number you could have traded.
Check 6: what do Monte Carlo and parameter stress tests add that one equity curve cannot?
One equity curve is one ordering of trades and one point on the parameter grid; shuffling the trade order shows how deep the drawdown could have been, and moving each setting up and down shows whether the result lives on a plateau or on a spike. The curve you see is one deal of the cards. The same 200 trades dealt in another order give the same net profit and a different worst drawdown.
A Monte Carlo test is a test that reorders or resamples the trade list many times and records the maximum drawdown of each new sequence, giving a range instead of a single value. You can run one on any trade list with the free Monte Carlo drawdown simulator, and the reason the historical drawdown usually sits on the kind side of that range is set out in Monte Carlo backtest: your real max drawdown is probably worse. A workable pass rule, written in advance: the drawdown that 95 percent of the shuffles stay under must fit the account you will trade.
A parameter stress test is a test that moves each setting a few steps up and down from the chosen value and reruns the strategy. A lookback of 20 that works while 18 and 22 are near zero is a spike: the setting describes noise in that history. A lookback of 20 whose neighbours from 16 to 26 all behave alike sits on a plateau, and a plateau is what you keep. Doubling the slippage and shifting the start date by a few months belong to the same check.
Check 7: how many trades does a backtest need, and which thresholds do you set beforehand?
The uncertainty of the average trade shrinks with the square root of the trade count, so a run with a few dozen trades is a hint rather than a result, and every pass/fail number (trades, efficiency, drawdown, profit factor after costs) must be written before the search so the result cannot move the goalposts. An average built from a dozen trades moves a long way when three trades are swapped; one built from several hundred barely moves.
uncertainty of the mean trade = spread of the single trades ÷ √N
Here the spread of single trades is how far a typical trade sits from the average (the standard deviation), N is the number of trades and √N is its square root. Four times the trades give half the uncertainty.
Hypothetical example
The average trade is 1,000 units and single trades typically sit 4,000 units from that average. With 41 trades the uncertainty is about 4,000 ÷ 6.4 ≈ 625 units, so the true average could be well below 1,000. With 160 trades it is about 316 units; with 640 trades about 158 units. Only the two larger samples put the average clear of zero by a comfortable margin.
Write the pass lines before the search. Example values many systematic traders use:
| Threshold | Example pass line | Why |
|---|---|---|
| Completed trades | A few hundred in total; fewer than 30 out-of-sample counts as a hint only | The average trade is not stable below that |
| Walk-forward efficiency, pooled | 50 percent or better | Half the in-sample edge survives re-chosen settings |
| Profit factor after costs | 1.3 or better | Profit factor is gross profit divided by gross loss; a value near 1 is cost and noise |
| Monte Carlo drawdown | 95th percentile fits the account | The historical drawdown is one deal of the cards |
| Parameter neighbours | Same sign two steps either side | Plateau, not spike |
The run below shows why the trade count decides so much. AlphaPrime ran a search on ES daily bars from 2019-01-02 to 2025-06-30, long and short, one contract, with $20 commission and $25 slippage charged on every trade. The rule is a stand-in, not the subject of this article: an Efficiency Ratio rule as the only entry and exit, no stop, no target, no filter. The candidate shown is the one the search kept out of 1,024 it tried, chosen for its return against its drawdown; picking the kept one after the fact flatters it, and the numbers describe one past path, not what the rule earns or what a reader can expect.
Source: AlphaPrime product engine (CPU), run 2026-10-31. ES daily bars, 2019-01-02 to 2025-06-30, long and short, 1 contract, $20 commission and $25 slippage per trade.
| Number | Result |
|---|---|
| Net profit (closed trades) | $50,967.5 |
| Max drawdown | $19,100 |
| Trades | 41 |
| Win rate | 56.1% |
| Profit factor | 2.107 |
| Average trade | $1,243.11 |
| Time in market | 15.84% |
Read the trade count first: 41 trades over about six and a half years. By the formula above, 41 trades leave the $1,243.11 average trade with a wide margin of error, and the $50,967.5 net profit and $19,100 maximum drawdown are one sequence of those 41 trades. Against a pass line of a few hundred trades written beforehand, this candidate is a hint to carry into a walk-forward run with more windows or a shorter bar size, not a result to trade.
Why does the best backtest in a search still need all seven checks?
The kept candidate is the best of many tries and carries a selection bonus that only unseen data and pre-written thresholds can discount, and a strategy that passes alone still has to be tested together with the others you already trade. A selection bonus is the part of the best backtest in a search that comes from having searched, not from the rule.
Give 1,024 people a coin each and ask them to flip it ten times. About one of them flips ten heads, and that person has no skill. The ES run above kept one candidate out of 1,024 tries, so part of its result is this bonus, and the in-sample numbers cannot say how much. Bailey, Borwein, López de Prado and Zhu (2014) show that the expected best backtest in a search rises with the number of tries even when every rule is noise, and Harvey and Liu (2015) propose discounting a result for the number of tests behind it. The size of the effect and the research behind it are in backtest overfitting: why the best backtest is usually inflated.
This is why checks 4 to 7 exist and why their lines are written first: unseen data does not carry the bonus, and a threshold written before the run cannot be moved to fit the result. One more gate follows the seven. A strategy that passes alone changes your account only in combination with what you already trade, and two strategies that lose in the same weeks add their drawdowns; how to measure that is in portfolio backtest: the combined drawdown is the one you live through.
Next step: take one strategy you are considering, write its seven pass lines on a single page with today's date, and run check 1 on its data before you look at its equity curve again.
Frequently asked questions
What should you validate before trading a strategy live?
Check the inputs first: adjusted rolls and clean bars, commission and slippage charged on every trade, and next-bar execution. Then test on data the rules never saw: a time-ordered out-of-sample block, a walk-forward run with pooled efficiency around half or better, a Monte Carlo drawdown range and a parameter plateau. Finally confirm the trade count is in the hundreds and that every pass line was written before the run.
How many trades does a backtest need to be trustworthy?
Enough that the average trade cannot move much when a few trades change. The uncertainty of the average shrinks with the square root of the trade count, so four times as many trades halve it. Many systematic traders want a few hundred trades in total and treat fewer than about 30 out-of-sample trades as a hint only. A shorter bar size or a walk-forward design produces more trades from the same history.
What walk-forward efficiency counts as a pass?
Walk-forward efficiency divides out-of-sample profit per unit of time by in-sample profit per unit of time. A pooled figure across all windows of about 50 percent or better is the common pass line. One weak window is normal because markets change; a pooled figure well below the line, or out-of-sample profit that lives in a single window, is a fail. Write the line down before the run.
How do you avoid overfitting a backtest?
Keep the number of tuned settings small relative to the trade count, include costs from the first run, and judge only on data the search never touched. Prefer settings whose neighbours behave alike over a single value that stands out. Remember that the best candidate out of many tries carries a selection bonus, so set every pass threshold before the search and apply it to the out-of-sample and walk-forward results, not the in-sample curve.
References
- Pardo, R. (2008). The Evaluation and Optimization of Trading Strategies (2nd ed.). John Wiley & Sons.
- Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society, 61(5), 458–471.
- Harvey, C. R., & Liu, Y. (2015). Backtesting. The Journal of Portfolio Management, 42(1), 13–28.