VALYNOXSTRATEGY RESEARCH TECHNOLOGY
AlphaPrimeAlphaPrime
Sign inFree 7-day trial
ACADEMY / Backtesting and validation

Can you trust a backtest? Six checks before you go live

A backtest is a record of what rules did on old prices under stated costs and execution settings, not a forecast. Trust it only after six ordered checks: no look-ahead or data errors, realistic costs, enough out-of-sample trades, a walk-forward run, no sign of over-optimization, and a small forward test.

By · Published · updated · Editorial policy

Contents
  1. Can you trust a backtest? Only after six checks
  2. Check 1: does the backtest peek ahead or run on bad data?
  3. Check 2: do the results survive commission and slippage?
  4. Checks 3 and 4: how many out-of-sample trades are enough, and why does walk-forward give you more?
  5. Check 5: how do you spot over-optimization?
  6. Check 6: why forward test small before trading full size?
  7. Frequently asked questions
  8. References

Can you trust a backtest? Only after six checks

Can you trust a backtest? Only after it passes six checks in a fixed order: no look-ahead or data errors, realistic costs, enough out-of-sample trades, a walk-forward run, no sign of over-optimization, and a small forward test. A backtest is a record of what rules did on old prices under stated costs and execution settings, not a forecast. A failure at any check sends the strategy back to research.

The order matters. A leak in check 1 makes every later number meaningless, and a cost error in check 2 changes which strategies even reach check 3. What backtesting is in futures explains the record itself; how backtesting software compares lists what a tool must let you set before the six checks can be run at all.

Six validation gates in a fixed order: leaks and data, costs and slippage, out-of-sample trades, walk-forward, overfit tests and a forward test; failing any gate sends the strategy back to research, and only a strategy that passes all six is traded at full size Six gates, in a fixed order Pass one to reach the next 1 2 3 4 5 6 Leaks and data Costs, slippage OOS trades Walk-forward Overfit tests Forward test Fail any gate Back to research Pass all six Trade full size Any failure stops the strategy. So: run the gates in this order.
Six gates in a fixed order. A failure at any gate stops the strategy there and sends it back to research.

Check 1: does the backtest peek ahead or run on bad data?

Most backtests that collapse live used information the trader could not have had at the time. Look-ahead bias is any case where a rule reads a price before that price existed: a fill on the same bar whose close gave the signal, an indicator that repaints past values, a roll gap left in a continuous contract, a bad tick that triggers a stop. Run the one-bar shift test first: fill every signal on the next bar's open instead of the signal bar's close, then compare the two reports. A result that falls apart was leaking. The common leaks are listed in look-ahead bias in backtests.

Two rows of five daily bars with a signal at the close of bar 3: the leaking backtest fills on that same bar, the honest backtest fills at the open of bar 4; the one-bar shift test compares the two results Fill on the next bar The signal comes at bar 3's close Signal Leaking Honest Same bar Next bar If results collapse, it leaked. So: run the one-bar shift test.
A signal computed on a bar's close can only be filled on the next bar. The one-bar shift test compares these two results.

Check 2: do the results survive commission and slippage?

Compare the gross average trade with the full round-trip cost, because an edge close to that cost disappears with one extra tick of slippage. Put commission, exchange fees and slippage into the search from the first run; adding them afterwards removes most finalists and wastes the search time. The free expectancy calculator shows how much of the average trade survives.

Hypothetical example

Gross average trade $90. Round-trip commission and fees $30. One tick of slippage each side on a $10 tick: $20. Net average trade $40. If slippage doubles to $40, the net falls to $20, less than a quarter of the gross figure.

Hypothetical average trade: 90 dollars gross, minus 30 dollars commission and fees and 20 dollars slippage, leaves 40 dollars net; if slippage doubles, the net falls to 20 dollars Costs take over half Average trade, hypothetical Gross Fees Slippage Net If 2× slip +$90 −$30 −$20 +$40 +$20 Double slippage halves the net. So: price costs into the first run.
Costs are small per trade and large in total. They decide whether an edge exists at all.

Checks 3 and 4: how many out-of-sample trades are enough, and why does walk-forward give you more?

Judge the untouched part of the history in completed trades, not years, because the uncertainty of the average trade shrinks only with the square root of the trade count. Out-of-sample data is the part of the history that never helped choose a rule or a setting. Two years of it hold two dozen trades for a monthly swing rule and a thousand for an intraday rule; same dates, very different evidence. The one formula:

average-trade uncertainty = spread of single trades ÷ √N

The spread of single trades is how far single results typically sit from the average, N is the number of out-of-sample trades, and √N is its square root. A few hundred out-of-sample trades is a workable aim; under about 30 is a hint, not a result.

A falling curve of the uncertainty of the average trade against out-of-sample trade count: relative to 100 trades, 30 trades give 1.8 and are only a hint, 400 trades give 0.5, so four times the trades halve the uncertainty 4× the trades, half the doubt Relative, 100 trades = 1.0 Uncertainty Trades 30 → 1.8 100 → 1.0 400 → 0.5 Under 30 trades is only a hint. So: count trades, not months.
The uncertainty of the average trade falls with the square root of the trade count, so going from 100 to 400 trades only halves it.

Walk-forward analysis is a test that optimises settings on one window, runs them unchanged on the window that follows, steps forward and repeats, then stitches the test parts into one curve. That stitched curve is the only one you could actually have traded, it holds more out-of-sample trades than a single split of the same history, and it shows whether re-chosen settings keep working as the windows move forward (Pardo, 2008). In AlphaPrime (Valynox), walk-forward is one step of strategy validation, where the pass thresholds are written before the search starts and the later checks stop when a strategy fails an earlier one.

Walk-forward analysis: four windows step forward in time, each optimised on an in-sample part and tested on the out-of-sample part after it; the test parts are stitched into one equity line, the only curve you judge, and its length is read in trades Judge only the test parts Four windows step forward in time Optimise Test Judge only this Stitched tests hold more trades. So: read its length in trades.
Walk-forward repeats the optimise-then-test split along the history. Only the stitched test parts are judged, and their length is read in trades.

Check 5: how do you spot over-optimization?

The best result from a large search is flattered by the search itself, so read a headline number only after a parameter plateau test, a Monte Carlo reshuffle of the trade order and a check that profit is spread across time. The run below tested NQ daily bars from 2019-01-02 to 2025-06-30, long and short, one contract, with a Bears Power rule as a stand-in instead of the subject itself; it is the one candidate kept out of 1,024 tried, for its return against its drawdown, picking the kept one after the fact flatters it, and no reader can expect those numbers.

Source: AlphaPrime product engine (CPU), run 2026-10-31. NQ daily bars, 2019-01-02 to 2025-06-30, long and short, 1 contract, $20 commission and $10 slippage per trade.

NumberResult
Net profit (closed trades)$130,510
Max drawdown$86,105
Trades227
Win rate52.86%
Profit factor1.229
Average trade$574.93
Time in market55.42%
Look at the marked out-of-sample stretch, not the whole curve: that stretch is where the strategy is judged, and the drawdown panel under it is the one you would have lived through.
Look at the marked out-of-sample stretch, not the whole curve: that stretch is where the strategy is judged, and the drawdown panel under it is the one you would have lived through. NQ · daily · 2019-01-02 to 2025-06-30 · CPU engine backtest
A scale of the forward test's average trade after costs: below zero is red and means stop, from 50 percent of the backtest value upward is a green band that means continue, with the backtest value marked near the centre Judge it by a band set first Forward average trade, after costs Stop Continue Backtest 0 50% Weaker but in band is normal. So: write the band before you start.
Reshuffling the order of the same 227 trades gives a range of possible drawdowns. Compare the width of that range with the single reported drawdown, which is only one draw from it. NQ · daily · 2019-01-02 to 2025-06-30 · CPU engine backtest

Over-optimization is tuning a rule so closely to one stretch of history that it describes the noise in that stretch. Three tests expose it. First, run the neighbouring settings: a lookback that works at 20 and fails at 18 and 22 is a spike, not a plateau. Second, reshuffle the trade order with the free Monte Carlo simulator and read the drawdown range, not the single value; if you will run several strategies together, the portfolio backtest with its combined drawdown is the number to plan around. Third, count how many quarters contributed. Bailey et al. (2014) show that the more variants a search tries, the more the best one is inflated, and Bailey and López de Prado (2014) give a way to discount a Sharpe ratio for the number of trials. More ways to break a strategy before trusting it are in robustness testing for trading strategies.

Two profit-versus-lookback curves from 10 to 30: a spike that works only at 20 and fails at 18 and 22 is rejected because it describes noise; a plateau where neighbouring settings agree is carried forward Spike or plateau? Profit by lookback, 10 to 30 Spike: reject Plateau: keep 10 20 30 10 20 30 Neighbours must agree. So: test 18 and 22, not just 20.
A setting that works only at one exact value is a lucky spike. A setting whose neighbours agree is a plateau.

Check 6: why forward test small before trading full size?

A short live or simulated run at the smallest contract size turns the backtest assumptions about fills, slippage and session timing into measured numbers. A forward test is a run of the frozen rules on prices that did not exist when the rules were written. Before it starts, write the tolerance: for example, live average trade after costs must stay within a set share of the backtest figure, and measured slippage must not exceed the value used in check 2. After enough trades to read, compare backtest and live results against that tolerance. Inside it, move to the planned size; outside it, go back to check 2 with the measured costs.

Next step: take the strategy you trust most, move every fill from the signal bar's close to the next bar's open, and compare the two reports. That one-bar shift is check 1, and it costs nothing but a few minutes.

Frequently asked questions

If a backtest makes money, will it make money live?

No. A backtest is a record of what rules did on old prices under stated costs and execution settings; it is not a forecast. The six checks narrow the gap between backtest and live results by removing leaks, pricing costs correctly, grading on untouched data and measuring real fills at small size.

How many trades does a backtest need to be reliable?

Count out-of-sample trades, not years. The uncertainty of the average trade shrinks with the square root of the trade count, so four times as many trades halve it. A few hundred out-of-sample trades is a workable aim, and walk-forward analysis produces more of them from the same history.

How do you tell over-optimization from a real edge?

Run the neighbouring settings: a plateau where neighbours agree is a better sign than a single spike. Check that profit is spread across quarters, read the Monte Carlo range of drawdowns rather than one value, and discount the headline result for the number of variants the search tried.

Are backtests reliable at all?

They are reliable as a record of what a set of rules did on past prices under the costs, sessions and fill rules you set. They are not reliable as a prediction of future results. Treat the report as evidence to be checked, and let the six checks decide how much weight it carries.

References

  • Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society, 61(5), 458–471.
  • Bailey, D. H., & López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. The Journal of Portfolio Management, 40(5), 94–107.
  • Pardo, R. (2008). The Evaluation and Optimization of Trading Strategies (2nd ed.). John Wiley & Sons.