ACADEMY / Backtesting and validation

Kolmogorov–Smirnov test: does out-of-sample look like in-sample?

The Kolmogorov–Smirnov test compares the whole distribution of out-of-sample trade results with the in-sample distribution and reports how likely a gap that large would be if both came from the same strategy behaviour. A low p-value means the out-of-sample trades behave differently from the trades the strategy was built on. It judges similarity, not profit.

By Soren Huang · Published 2026-10-30 · updated 2026-10-31 · Editorial policy

Contents

What does a Kolmogorov–Smirnov test tell you about a strategy?

The Kolmogorov–Smirnov test (KS test) is a statistical test that measures how far apart two distributions are and how surprising that distance would be if both samples came from the same source. In-sample data is the part of the history used to choose the rules and parameters. Out-of-sample data is the part held back and only used to test the finished rules. Each part produces a list of trade results, and the KS test compares those two lists as whole shapes.

Think of two school classes that both scored an average of 70 on the same exam. One class has every student between 65 and 75. The other has half the students near 40 and half near 100. The averages match, but the classes are not alike. A strategy can behave the same way: a positive out-of-sample total can come from one large trade sitting on top of many small losses, while the in-sample period produced steady medium-sized wins. The sign of the total says the two periods agree. The distributions say they do not. The article on whether a positive out-of-sample P&L sign means skill explains why the sign alone is weak evidence; the KS test is one tool that asks the stricter question.

Two groups with the same average of 70: one tight between 65 and 75, one split near 40 and near 100. The mean matches, the shape does not.

What does the KS test actually compare?

The KS test compares two cumulative distribution curves, and its statistic D is simply the largest vertical gap between them. A cumulative distribution is a curve that shows, for every possible trade result, the share of trades that finished at or below that result. To build it, line up all trades from the worst loss on the left to the best gain on the right, then at each result level count what fraction of the trades sit at or to the left of that level. The curve starts at 0 (no trades are below the worst trade) and ends at 1 (all trades are at or below the best trade).

Draw the in-sample curve and the out-of-sample curve on the same chart. If the two periods behave alike, the curves run close together. If the out-of-sample period has more losers, its curve climbs earlier and sits above the in-sample curve on the left side. If it has fewer large winners, it reaches 1 sooner on the right. The test finds the single result level where the two curves are farthest apart vertically and reports that distance as D.

The KS statistic D is the largest vertical gap between the in-sample and out-of-sample cumulative curves of trade results.

Comparing cumulative curves has two practical advantages for traders. First, it uses every trade, not only the average or the total, so a shift in the size of losers or the frequency of winners shows up even when the mean stays similar. Second, it makes no assumption about the shape of the distribution. Trade results are rarely bell-shaped: a strategy with a tight stop and a wide target has many small losses and a few large gains, and the KS test handles that shape without adjustment.

What do the statistic and the p-value mean in plain words?

D says how far apart the two trade distributions are, and the p-value says how often a gap at least that large would appear by chance if both samples really came from the same distribution. The one formula in this article is the definition of D:

D = max | F_is(x) − F_oos(x) |

Here x is any trade result level, for example a net result of −3 points or +12 points. F_is(x) is the share of in-sample trades that finished at or below x. F_oos(x) is the same share for the out-of-sample trades. The vertical bars mean "ignore the sign, take the size of the difference". max means "look at every level x and keep the largest difference you find". D is always between 0 and 1. A D of 0.10 means that at the worst point the two curves disagree by ten percentage points of trades. A D of 0.40 means they disagree by forty percentage points, which is a large difference in behaviour.

The p-value is the probability of seeing a D at least this large when both samples truly come from the same distribution. If you flip a fair coin ten times and get seven heads, that is not surprising; it happens often by chance. If you flip it two hundred times and get 140 heads, that is very surprising, and you would doubt the coin is fair. The p-value plays the role of "how surprising": a high p-value (say 0.40) means a gap this size is ordinary noise between two samples of the same thing; a low p-value (say 0.01) means a gap this size would be rare if nothing had changed, so the more likely explanation is that the out-of-sample trades really do behave differently.

A small gap between the curves gives a high p-value; a large gap gives a low p-value and points to a change in how the strategy trades.

Most people use a threshold of 0.05 (5%) for the p-value, meaning they treat a gap that would appear by chance less than one time in twenty as a real difference. The number itself matters less than the order of events: choose the threshold before you look at the result, and keep it the same for every strategy you test.

How do you run the KS test on in-sample vs out-of-sample trade returns?

The procedure is six steps, and the split between the two periods must be fixed before any numbers are read. Most statistics libraries and spreadsheet add-ins include a two-sample KS function that performs steps 3 to 5 for you; the work that matters is in steps 1, 2 and 6. The closed-trade list in the performance report of AlphaPrime (Valynox) supplies the per-trade results for step 1, which you then split at the boundary date in step 2.

Collect per-trade net results after costs. Use the result of each closed trade with commission, fees and slippage already deducted, in one unit (points, ticks or currency). Costs change the shape of the distribution, so a test on gross results answers a different question. The article on trading costs in backtests explains what to deduct.
Split the list exactly at the in-sample/out-of-sample boundary. Every trade opened before the boundary date belongs to the in-sample group; every trade opened after it belongs to the out-of-sample group. Do not move the boundary after seeing the result.
Sort each group and build its cumulative curve. Order each group from the worst loss to the best gain and compute, at each result level, the share of that group's trades at or below the level.
Measure D. Find the result level where the two curves differ the most and record that vertical difference.
Read the p-value. The software reports it from D and the two sample sizes.
Compare with the threshold chosen beforehand. If the p-value is below the threshold, record a fail. If it is at or above the threshold, record a pass and move on to the next check.

From a trade list to a KS result: the split and the threshold are fixed before any number is read.

Hypothetical example

A breakout strategy on an index future produces 400 in-sample trades and 100 out-of-sample trades, all recorded as net points after costs. The in-sample curve and the out-of-sample curve are farthest apart at a result level of −4 points: 38% of in-sample trades finished at or below −4 points, while 50% of out-of-sample trades did. D is therefore 0.12. With these sample sizes the p-value is about 0.20. The threshold chosen before the test was 0.05, so the strategy passes: a gap of 0.12 between a sample of 400 and a sample of 100 is ordinary. Had D been 0.20 instead (for example 58% of out-of-sample trades at or below −4 points), the p-value would have been about 0.003 and the strategy would fail, meaning the out-of-sample period produced losers far more often than the period it was built on.

What does a KS fail mean, and what does it not mean?

A KS fail means the out-of-sample trades behave differently from the trades the strategy was built on; a pass means no such difference was detected, and neither result says anything about profit. The most common reasons for a fail are overfitting and a changed market. Overfitting is the case where the rules were tuned to noise in the in-sample data, so the in-sample trade results describe luck that the out-of-sample period could not repeat. Bailey, Borwein, López de Prado and Zhu (2014) show how selecting the best of many backtests inflates in-sample results and leads to out-of-sample behaviour that does not resemble them; the article on why the best backtest is usually inflated covers this in detail. A changed market is the case where the rules were sound for the in-sample years but the instrument now trades with different volatility, different session liquidity or different costs.

	KS pass	KS fail
What it says	Out-of-sample trade results look like in-sample results	Out-of-sample trades behave differently
Next step	Continue with the other validation checks	Look at the two curves, then check for overfitting or a changed market
Does it say the strategy is profitable?	No	No
Does it predict next year?	No	No
Does it say which way the shift went?	Not needed	No; read the curves

A pass and a fail each answer one question; profit, the future and the direction of the shift are outside the test.

The test is symmetric: a strategy whose out-of-sample trades are much better than in-sample fails just as hard as one whose trades are much worse, because D only measures distance. To learn the direction, look at the chart: if the out-of-sample curve sits above the in-sample curve on the loss side, the out-of-sample period had more losers. A pass with a losing strategy is also common. If a strategy lost money steadily in-sample and lost money the same way out-of-sample, the KS test passes, because the behaviour is consistent. Consistency is the question; quality is a separate check.

How many trades do you need before the KS test means anything?

With few trades the smallest gap the test can detect is large, so a pass on a small out-of-sample sample says little. The reason is the same as with the coin: seven heads in ten flips is not evidence of anything, and a 40-percentage-point gap between two groups of 20 trades is not either. The test only calls a difference real when the gap is bigger than what random sampling alone would produce, and random sampling produces big gaps when samples are small.

The table below shows the approximate smallest D that counts as a fail at the 5% level when both samples have the same number of trades, using the standard large-sample approximation of the KS test. The detectable gap shrinks roughly with the square root of the trade count: four times as many trades halves the gap the test can see.

Trades in each sample	Smallest detectable D (5% level, approx.)	What that gap means
20	about 0.43	Curves can differ by 43 percentage points of trades and still pass
50	about 0.27	Only a very large shift in behaviour is detected
100	about 0.19	A moderate shift is detected
500	about 0.09	A small shift is detected

The smallest gap the KS test can detect at the 5% level shrinks as the number of trades in each sample grows; small samples pass easily.

Three practical consequences follow. First, the smaller of the two samples sets the limit. An out-of-sample period with 30 trades cannot be rescued by an in-sample period with 600; the detectable gap stays large. Second, a pass on a small sample is not evidence of consistency, only an absence of evidence against it, so treat the KS test as a filter that removes clear failures rather than a verdict that confirms a strategy. Third, if a strategy trades too rarely to collect enough out-of-sample trades, the better fix is a longer out-of-sample period or a walk-forward design with several out-of-sample windows joined together, not a looser threshold.

Which other checks belong next to the KS test?

The KS test answers one question about distribution shape and works best as one step inside a wider validation sequence. Three checks sit naturally beside it, in this order:

Walk-forward analysis. A single in-sample/out-of-sample split gives one KS result from one boundary date. Walk-forward analysis re-tunes on each in-sample window and tests on the window that follows, which produces many out-of-sample segments. Joining those segments gives a larger out-of-sample sample for the KS comparison and shows whether the behaviour held across different market conditions rather than one lucky period.
Cost stress. Rerun the out-of-sample period with slippage doubled and repeat the comparison. A strategy whose distribution shifts sharply under higher costs depends on cheap execution, and that dependence does not show in a single KS result computed at one cost level.
A correction for the number of variants tested. If you generated many strategy variants and kept only the ones that passed, the survivors include weak passes that a larger out-of-sample sample would not have supported. Bailey and López de Prado (2014) propose the deflated Sharpe ratio to adjust for the number of trials; the article on the deflated Sharpe ratio explains how to apply it. The same logic applies to the KS test: with a 5% threshold, about one in twenty strategies that really are consistent will fail by chance, and an inconsistent strategy passes whenever the out-of-sample sample is too small to show the shift, so the number of strategies you tested and the size of each sample belong in the record.

AlphaPrime handles the walk-forward part of this sequence in its strategy validation. Its strategy validation applies consistent research standards with custom thresholds, includes walk-forward analysis, offers Basic, Standard and Extensive validation levels, and stops the later checks when a strategy fails an earlier one, so time goes to the candidates that survive the first filters.

What this means if you use AlphaPrime: the per-trade results AlphaPrime produces, with fees, slippage and next-bar execution included in every evaluation, are the right input for an in-sample versus out-of-sample comparison, and the walk-forward windows in strategy validation give you the out-of-sample trades that comparison needs.

Frequently asked questions

What is a good p-value for a KS test on a trading strategy?

A p-value at or above the threshold you chose before testing, most often 0.05, means no difference was detected between the in-sample and out-of-sample trade distributions. Fix the threshold once and apply it to every strategy; a higher p-value only means the observed gap is more ordinary, not that the strategy is better. A pass on a small out-of-sample sample is weak evidence, so read the p-value together with the trade count.

Can I run the KS test on daily returns instead of individual trades?

Yes. The two-sample KS test only needs two lists of numbers, so daily or weekly net returns of the strategy work as inputs. Daily returns give more data points for a strategy that trades rarely, but many days will be zero when no position is open, which flattens both curves. Per-trade results describe the strategy's behaviour more directly when the trade count is large enough.

How is the KS test different from a t-test?

A t-test compares the averages of two groups and assumes the data are roughly bell-shaped. The KS test compares the whole cumulative distributions and makes no shape assumption. Two periods can have the same average trade and still fail a KS test if the spread or the frequency of large losers changed, which is exactly the kind of change a trader wants to see.

Does a KS pass mean my strategy is not overfitted?

No. A pass means the out-of-sample trades were not detectably different from the in-sample trades, given the sample sizes. With a small out-of-sample sample the test can only detect very large shifts, and the more variants you screen, the more such weak passes you collect. Use the pass as one filter next to walk-forward analysis, cost stress and a correction for the number of variants tested.

What is the difference between a one-sample and a two-sample KS test?

A one-sample KS test compares one list of numbers with a theoretical distribution, such as a normal curve. A two-sample KS test compares two lists of numbers with each other. Comparing in-sample and out-of-sample trade results is a two-sample test: neither list is a theory, both are observed trades.

References

Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society, 61(5), 458–471.
Bailey, D. H., & López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. The Journal of Portfolio Management, 40(5), 94–107.