Deflated Sharpe ratio: correcting for how many variants you tested
When you do not know how many variants you tested, deflate the Sharpe ratio with a generous upper bound on trials and compare it with an effective-N estimate. A fresh holdout that you look at only once is the final check.
By Soren Huang · Published · updated · Editorial policy
Contents
- What does the deflated Sharpe ratio correct for?
- How fast does the luck hurdle grow with the number of trials?
- How do you count trials honestly?
- Should you use an upper bound or an effective N?
- How do you calculate the deflated Sharpe ratio step by step?
- Why is a fresh holdout looked at once the cleanest check?
- What can no correction method fix?
- Frequently asked questions
- References
What does the deflated Sharpe ratio correct for?
The deflated Sharpe ratio asks whether your best backtest beats the best result that luck alone would give after N trials. If you do not know N, use a generous upper bound, because the luck hurdle rises slowly with N. Estimate an effective N for correlated variants, then confirm the result on a fresh holdout you look at once.
Imagine that a thousand people each flip a coin ten times. Nobody is skilled at flipping coins, yet at least one of them will almost certainly get nine or ten heads. If you only look at the winner, you see a talent that does not exist. A strategy search works the same way. Every variant you test is one more person flipping coins, and your best backtest is the winner of that contest.
The Sharpe ratio is the average return of a strategy divided by the standard deviation of its returns. It measures return per unit of fluctuation. Multiple testing is the problem that appears when you test many variants and keep the best one: the best score is high partly because you searched. The deflated Sharpe ratio is the probability that a strategy's true Sharpe ratio is above the best Sharpe ratio you would expect from luck alone, given the trials you ran (Bailey & López de Prado, 2014). If you need the raw Sharpe ratio first, the free Sharpe ratio calculator computes it from a return series.
| Input | What it is | Effect on the deflated Sharpe ratio |
|---|---|---|
| Number of trials (N) | How many independent variants you tested before you chose one | More trials raise the luck hurdle, so the result goes down |
| Spread of Sharpe ratios across trials | How much the Sharpe ratios of your trials differ from each other | A wider spread raises the luck hurdle |
| Sample length (T) | The number of return observations, for example daily returns | A longer sample makes the estimate more precise, so the result moves further from 0.5: up if your Sharpe ratio is above the hurdle, down if it is below |
| Skew | Whether returns are lopsided | Negative skew lowers the result |
| Kurtosis | How often extreme returns occur | Fat tails lower the result |
Skew is a measure of whether returns are lopsided. Negative skew means many small gains and occasional large losses. Kurtosis is a measure of how often extreme returns occur compared with a normal bell curve. A normal distribution has a kurtosis of 3, and higher values mean fatter tails. For the same Sharpe ratio and the same number of returns, negative skew and fat tails make the measured Sharpe ratio less reliable. The deflated Sharpe ratio therefore comes out lower.
How fast does the luck hurdle grow with the number of trials?
The luck hurdle grows with the number of trials, but slowly: above about ten trials, each tenfold increase adds about one unit of spread or less. This is the most useful fact when you do not know your trial count.
The hurdle is the Sharpe ratio that the best of N luck-only strategies would reach on average:
Luck hurdle ≈ spread × luck multiple
Here, spread is the standard deviation of the Sharpe ratios across your trials. It shows how far apart your variants' results lie. The luck multiple is the expected best score among N independent draws from a normal bell curve. The table gives its value for typical trial counts. It uses the approximation from Bailey and López de Prado (2014).
| Independent trials (N) | Luck multiple (spreads above zero) |
|---|---|
| 1 | 0 |
| 10 | about 1.5 |
| 100 | about 2.5 |
| a thousand | about 3.2 |
| 10,000 | about 3.9 |
| 100,000 | about 4.4 |
Suppose your count is too low by a factor of two. Near 100 trials, that moves the hurdle by only about 0.2 spreads. If your count is too low by a factor of ten, the hurdle moves by about 0.7 to 1 spread in the range most traders work in.
How do you count trials honestly?
A trial is every version of a strategy whose result could have influenced which strategy you kept, including the versions you threw away. Count these:
- Every parameter combination in a sweep. Suppose you test a moving-average length from 10 to 50 in steps of 5. That is nine trials, not one. Combine it with three stop distances and you have 27.
- Every rule variant. This includes each entry type, each exit and each filter you added or removed.
- Every market, bar size and session that you tried the idea on.
- Rejected ideas tested on the same data. This includes ideas you dropped after one look at an equity curve.
- Earlier research on the same history. This covers your own work from past years. It also covers ideas from books or forums that became popular because they looked good on this market's past.
- Reruns after you changed costs, dates or settings, when you made the change because of the result.
An identical rerun that reproduces the same numbers is not a new trial. Items 4 and 5 are the parts you can never count fully. That is why a generous upper bound is often more honest than an exact tally. For the reason why the top result of a large search is inflated, see backtest overfitting: why the best backtest is usually inflated.
In automated search, the count can be very large. The published AlphaPrime (Valynox) benchmark recorded 1,268,200,000 strategy evaluations in 18 h 23 min on one RTX 3090. It was measured on 2026-10-05 with TY 37-minute bars from 2015-01-01 to 2026-09-25, 114,371 bars per candidate, and commission and slippage included in every evaluation. Real speed depends on the graphics card, the data length and the research settings. If you treated every one of those evaluations as an independent trial, the luck multiple would be about 6. Most candidates in a search like that are near-copies of each other, however, which is why effective N matters. If you keep each search run as a batch in AlphaPrime's strategy databank, you have a written record to count from instead of memory.
Should you use an upper bound or an effective N?
Use both. A conservative upper bound tells you whether the strategy survives your worst-case count. An effective-N estimate tells you how much of that count was real variety. Effective N is the number of truly independent trials that your correlated variants are worth. If you ask 400 people for directions and 360 of them read from the same map, you got about 40 opinions, not 400.
| Conservative upper bound | Effective N | |
|---|---|---|
| How to get it | Add up everything in the list above, then multiply generously for ideas you did not log | Group variants whose returns move together and count the groups |
| Strength | Hard to fool yourself | Reflects how many different ideas you really tested |
| Weakness | Treats near-copies as independent, so it can be too strict | Needs the return series of the variants, and you can steer it by choosing how to group |
| Best use | No log of what you tested | A log of returns from a systematic search |
To estimate effective N:
- Save the daily return series of every variant, or of a large random sample of them.
- Measure the correlation between each pair, that is, how closely their returns move together.
- Group variants that move closely together. A common way to do this is hierarchical clustering, which repeatedly joins the two most similar variants or groups until the remaining groups are clearly different.
- Count the groups. That count is your effective N.
- Measure the spread of Sharpe ratios across one representative per group, not across all variants.
Grouping works better than a single average correlation. Suppose 300 near-copies of one idea sit next to 100 unrelated ideas. The average correlation looks high, yet you tested more than 100 different ideas.
Hypothetical example
You logged 400 variants of one breakout idea. The best variant has an annualized Sharpe ratio of 1.6. The spread across trials is 0.5, and this example assumes the same spread in every row. You also tried ideas that you did not log, so you set an upper bound of 4,000 trials. Grouping the logged variants by correlation gives about 40 groups. Applying the same ratio to the bound gives about 400.
| Count used | N | Luck multiple | Hurdle | Best variant (1.6) |
|---|---|---|---|---|
| Upper bound, as independent trials | 4,000 | about 3.6 | about 1.8 | below |
| Logged variants, as independent trials | 400 | about 3.0 | about 1.5 | just above |
| Effective N from the bound | about 400 | about 3.0 | about 1.5 | just above |
| Effective N from the logged variants | about 40 | about 2.2 | about 1.1 | clearly above |
The strategy clears the hurdle only when you leave out unlogged ideas or group the near-copies. That is a borderline result. A different correction will not settle it, but new data can.
The two choices that move the answer most are whether you include unlogged ideas and whether you group near-copies. Fine details of the method matter much less. When the verdict changes between your bound and your effective N, treat the strategy as unproven and go to the holdout.
How do you calculate the deflated Sharpe ratio step by step?
You calculate it by measuring the luck hurdle and then asking how likely it is that the true Sharpe ratio lies above that hurdle. The probabilistic Sharpe ratio is the probability that the true Sharpe ratio is above a chosen benchmark, given the sample length, skew and kurtosis. The deflated version sets that benchmark to the luck hurdle.
- Take the chosen strategy's net returns after costs at one fixed period, for example daily.
- Compute the Sharpe ratio in that same period without annualizing it. Also compute the skew, the kurtosis and the number of returns T. Check whether your software reports kurtosis or excess kurtosis (kurtosis minus 3). The formula uses plain kurtosis.
- Choose N. Run the calculation once with the upper bound and once with effective N.
- Measure the spread of the per-period Sharpe ratios across the trials.
- Multiply that spread by the luck multiple to get the hurdle.
- Enter the Sharpe ratio, the hurdle, T, the skew and the kurtosis into the probabilistic Sharpe ratio formula in Bailey and López de Prado (2014). The result is the deflated Sharpe ratio.
- Read the result as a probability. A value near 1 clears the hurdle with room to spare. A value near 0.5 means the strategy is about as good as the best luck-only result. Many researchers use 0.95 as their line.
Why is a fresh holdout looked at once the cleanest check?
A fresh holdout that you look at once is the cleanest check because it is a single trial. It needs no trial count, no correlation estimate and no deflation. A holdout is a stretch of data that you set aside before research starts and do not use for any decision until the final test. It is the exam that the student has never seen, not the practice questions that the student memorized.
- Set the holdout aside before any research. The most recent stretch of history is a common choice.
- Write down the pass rule before you look, for example: net profit after costs above zero, and a drawdown no worse than the one in the research period.
- Freeze the strategy, including its costs and session settings.
- Test one final candidate. If you test 50 candidates on the holdout, you are back to 50 trials.
- Run the test once. If the strategy fails, do not retune it and test again on the same data. The holdout is now part of your research data.
A holdout has one weakness: it is short, so its result is noisy. Assume returns are independent and normally distributed. With one year of data, the measured annualized Sharpe ratio typically misses the true value by about 1 or a little more; statisticians call this typical miss the standard error (Lo, 2002). One year therefore cannot reliably tell a Sharpe ratio of 0.5 from one of 1.5. The two checks complement each other. The deflated Sharpe ratio uses the long history, and the holdout removes the counting problem.
Walk-forward windows are out-of-sample within one run. If you rerun the research many times and keep the best walk-forward result, those windows are no longer fresh. The guide to walk-forward analysis explains how to set up the windows. AlphaPrime's strategy validation covers walk-forward analysis and lets you set your own thresholds, so you can write down the pass rule before the run.
What can no correction method fix?
Corrections adjust for how hard you searched. They cannot repair a backtest that was wrong, trials that you never counted, or a market that changed.
- A biased backtest. Missing commission and slippage, fills at the same bar close that triggered the signal, or bad data all inflate the Sharpe ratio before any deflation happens.
- Trials beyond your bound. If the real count was far above the bound you used, the hurdle is too low.
- Adjusting the correction until the strategy passes. Trying several values of N or several methods and keeping the friendliest one is another trial.
- A change in the market. Every method assumes that the future behaves like the sample.
- A short sample. The correction can be right and the answer can still be uncertain.
Other methods deal with the same problem. White's Reality Check (White, 2000) and Hansen's test for superior predictive ability (Hansen, 2005) use resampling across all the rules you tested. Sullivan, Timmermann and White (1999) applied this approach to technical trading rules. The probability of backtest overfitting (Bailey et al., 2017) compares the in-sample and out-of-sample rankings of many trials. Harvey and Liu (2015) give haircuts for Sharpe ratios based on the number of tests. All of these methods need the full set of trials you ran, so none of them solves the problem of an unknown N either. Only a holdout avoids that problem.
What this means if you use AlphaPrime: keep your searches as batches so that you can count trials. Judge the best candidate against both your upper bound and your grouped count, and keep a final stretch of data closed until one last run.
Frequently asked questions
What is a good deflated Sharpe ratio value?
The deflated Sharpe ratio is a probability between 0 and 1. Many researchers treat a value above 0.95 as strong evidence that the true Sharpe ratio is above the luck hurdle. A value near 0.5 means the strategy is about as good as the best result you would expect from luck after the same number of trials.
What number of trials should I use in the deflated Sharpe ratio?
When you do not know the number of independent trials, run the calculation once with a generous upper bound that includes unlogged ideas and once with an effective N from grouping correlated variants. If both pass, the result is robust; if only the smaller count passes, test the strategy on a fresh holdout.
Does walk-forward analysis remove the need for multiple testing correction?
No. Walk-forward analysis tests each run on data that the run did not tune on. If you repeat the research many times and keep the best walk-forward result, you have made many trials again. Count those runs, or confirm the result on a holdout that you look at once.
Can I put an annualized Sharpe ratio into the deflated Sharpe ratio formula?
The Sharpe ratio, the spread across trials and the sample length must all use the same period. If your returns are daily, use the daily Sharpe ratio and the number of days. Mixing an annualized Sharpe ratio with a daily count gives a misleading result.
References
- Bailey, D. H., & López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. The Journal of Portfolio Management, 40(5), 94–107.
- Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance, 20(4), 39–69.
- White, H. (2000). A Reality Check for Data Snooping. Econometrica, 68(5), 1097–1126.
- Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics, 23(4), 365–380.
- Sullivan, R., Timmermann, A., & White, H. (1999). Data-Snooping, Technical Trading Rule Performance, and the Bootstrap. The Journal of Finance, 54(5), 1647–1691.
- Harvey, C. R., & Liu, Y. (2015). Backtesting. The Journal of Portfolio Management, 42(1), 13–28.
- Lo, A. W. (2002). The Statistics of Sharpe Ratios. Financial Analysts Journal, 58(4), 36–52.
What AlphaPrime does here
AlphaPrime's strategy validation offers Basic, Standard and Extensive validation, walk-forward analysis and your own thresholds. The Performance Report shows equity, drawdown, trades and periodic performance, which you can use to judge a candidate's returns before the final holdout run.