Backtest overfitting: why the best backtest is usually inflated

Why does the best of many backtests look good by luck?

The best of many backtests is usually inflated because selection itself adds luck. When you try many parameter sets and keep the top one, part of its score is the maximum of random noise, and that share grows with every variation you try. Research by Bailey, López de Prado and colleagues shows how to estimate and discount it.

Ask 100 people to flip a fair coin 10 times each. Nobody has any skill at flipping coins. Still, the chance that at least one person gets 8 or more heads is above 99%. The chance that someone gets 9 or 10 heads is about two in three.

Now ask the "best flipper" to do it again. Their next score will most likely be close to 5 heads, like everyone else's. Their first result was real. It just was not evidence of skill. It was the best of 100 random outcomes.

A parameter sweep is the same experiment. Each parameter set is one person flipping coins. The best backtest is the best flipper. Part of its result can be an edge in the market, and part of it is the luck of being the maximum of many tries. Backtest overfitting is choosing a strategy because it scored best among many tries, so that part of its backtest result comes from luck rather than from a real edge.

What did the research on backtest overfitting find?

The research shows that a strong-looking backtest is easy to produce by trying variations, and the more you try, the more likely the winner is overfit. David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu showed this in three papers (listed in the references below). Their main findings, in plain words:

• A good-looking backtest is easy to produce. You do not need millions of tries. A fairly small number of variations on a few years of data is enough to find one that looks strong, even when none of them has an edge.
• The more variations you try, the better the best one looks by luck alone. The probability that your winner is overfit rises with the number of tries.
• The number of tries is usually hidden. Most backtests are reported without saying how many variations were tested. Without that number, nobody can judge how much of the result is selection. That includes you, if you do not keep count.
• Overfit strategies can do worse than zero. If market behaviour tends to reverse after a run (for example, a period of strong trends followed by choppy prices), parameters fitted tightly to one period can lose money in the next. The expected live result is then negative, not just flat.

The 2014 Notices of the AMS paper gives an example: with five years of daily data, after about 45 independent variations the best one is expected to show an annualized Sharpe ratio of about 1 in the backtest, even when its expected Sharpe ratio on new data is zero.

The Sharpe ratio is the average return divided by how much the returns vary (their standard deviation), usually scaled to one year. It measures return per unit of bumpiness. A value of 1 is often treated as a good strategy, which is why this example matters.

How much Sharpe ratio can luck alone produce?

Luck alone can push the best Sharpe ratio to about 1 after roughly 50 variations on five years of data, and above 1.1 by 100. You can estimate this "luck bar" for your own search. The luck bar is the Sharpe ratio the best of your variations would reach even if none of them had any edge. Estimate it with this formula:

Luck bar ≈ M ÷ √Y

• M is the typical best score among N random draws. It comes from the table below and grows slowly as N grows.
• Y is the number of years in your backtest. √Y is its square root. Longer histories shrink the luck bar, because random Sharpe ratios settle closer to zero over more data.
• The result is in annualized Sharpe ratio.

The formula assumes three things: the variations are independent of each other, returns are roughly bell-shaped, and no variation has a real edge. Treat it as a rough yardstick, not an exact answer.

Real variations are not independent. A 50-bar and a 52-bar moving average give almost the same trades. The effective number of independent tries is therefore smaller than the raw count, but it is hard to know how much smaller. Use the raw count as the strict end of the range and a lower estimate as the lenient end. If the result only passes with the lenient number, it is weak.

What does the Deflated Sharpe Ratio measure?

The Deflated Sharpe Ratio (DSR) is a single number, from the 2014 paper by Bailey and López de Prado, that turns the idea above into a probability. It asks: given how many variations were tried, how confident can we be that the winner's true Sharpe ratio is above what luck would produce?

It takes into account:

• The number of tries. More tries raise the bar the winner must clear.
• How spread out the Sharpe ratios of all tries were. A wide spread means luck can push the best one higher.
• The length of the track record. Short records are noisier.
• The shape of the returns. Skewness measures lopsidedness. Kurtosis measures how often extreme outcomes occur, often called "fat tails". A strategy with many small wins and rare large losses has a Sharpe ratio that looks better than its risk, and the DSR lowers it.

The output is a probability between 0 and 1. A value near 1, such as 0.95, means the result is unlikely to be explained by selection and noise. A value near 0.5 or lower means the winner is about what you would expect from the number of tries alone.

What does the Probability of Backtest Overfitting measure?

The Probability of Backtest Overfitting (PBO) is a measure, from the 2017 paper, of how often your way of picking a winner fails on unseen data; it looks at the selection process rather than at one winner. The idea, simplified:

1. Cut the history into equal blocks, for example 10 blocks.
2. Pick half of the blocks as a "choosing" set and keep the other half as a "checking" set. Do this for every possible way of splitting the blocks in half. With 10 blocks there are 252 ways.
3. For each split, find the best variation on the choosing set. Then see where that variation ranks among all variations on the checking set.
4. Count how often the chosen winner ranks below the median on the checking set.

That share is the PBO. If your process has no real skill at picking winners, the chosen one lands above or below the middle about equally often, and the PBO is near 0.5. A PBO well below 0.5 means the in-sample winners tend to stay in the better half on unseen data. A PBO above 0.5 means picking the backtest leader is worse than picking at random.

The PBO needs the results of all the variations you tried, not just the winner. This is one more reason to keep a record of every run.

How many tries does a futures trader really make?

Variations add up faster than most traders notice.

Everything you looked at before choosing counts as a try:

• every parameter combination in a sweep,
• every indicator or rule you swapped in and out,
• every market, bar size or session you tested,
• every earlier round of the same research, even if you threw the results away.

The same problem appears when many different trading rules are tested on the same data, and the bar for the winner rises in the same way.

Automated strategy generation makes this effect larger. A search over millions of candidates will surface winners with very high backtest scores, because the luck bar rises with every candidate. That does not make large searches useless. It means the winner must clear a bar set by the size of the search, and it must be checked on data the search never touched. For a longer discussion, see overfitting in strategy generation.

How do you keep the discount honest?

You keep the discount honest by counting every try, holding back untouched data and judging the winner against the luck bar for its batch.

• Count your tries. Keep a log of every batch: how many variations, which markets, which date range. You need this number to set a luck bar, to compute a DSR or to run a PBO check.
• Keep untouched data. Set aside the most recent part of your history, for example the last two years, before you start. Test only your final candidates on it, once. If you look at it, change a rule and look again, it is no longer untouched and becomes part of the search.
• Prefer stable parameter regions. Look at the results for neighbouring parameter values. A broad plateau, where nearby settings also do well, is more likely to reflect a real pattern. A single sharp peak surrounded by poor results is the signature of a lucky draw.
• Use more history when you can. The luck bar falls as the test gets longer, so the same Sharpe ratio is more convincing over ten years than over three.
• Test the process, not just the winner. Walk-forward analysis repeats the choose-then-check cycle across the history and shows whether your way of picking parameters holds up.
• Include costs from the first run. Commission and slippage remove many small, noise-driven edges before they can win the selection; see trading costs in backtests.
• Judge the winner against its batch. Compare the best result with the luck bar for the number of tries. Discount it before you compare it with anything else.

What this means if you use AlphaPrime (Valynox): keep each search batch in the strategy databank so the number of tries is recorded, and check the winners with walk-forward analysis in strategy validation.