In-sample vs out-of-sample: how to split and how much is enough

Contents

1. What are in-sample and out-of-sample data, and why split at all?
2. How should you split the data?
3. How much out-of-sample is enough: trades or years?
4. What are the signs of curve fitting?
5. What should you do when out-of-sample is weaker but not broken?
6. Why keep a final untouched period?
7. How does this connect to walk-forward analysis?
8. Frequently asked questions
9. References

What are in-sample and out-of-sample data, and why split at all?

Split your history by date: tune rules and settings on the earlier part (in-sample), then run them unchanged on the later part (out-of-sample). A 70:30 to 80:20 split is common. Judge enough out-of-sample by completed trades, not years, and keep a last block of history untouched for one final test.

In-sample data is the part of the price history you use to choose a strategy's rules and settings. Out-of-sample data (樣本外) is the part you set aside and use only to run those rules unchanged and read the result. Think of a student: the in-sample period is the study material, the out-of-sample period is the exam. A student who has seen the exam questions in advance will score well, and the score tells you nothing about what they learned. To compare the two parts on one scale, run each through the free Sharpe ratio calculator.

The split exists because data that helped pick a strategy cannot fairly grade it. When you test many rule combinations on one data set and keep the best, part of what you keep is a real pattern and part is noise that happened to line up with that history. The in-sample result cannot separate the two. Only data the search never touched can, which is why overfitting in strategy generation is checked out-of-sample and nowhere else.

How should you split the data?

Split by time, with the in-sample period first and the out-of-sample period after it, in a proportion of roughly 70:30 to 80:20. A time split copies how you will actually trade: build on the past, trade into a future you have not seen. A random shuffle of bars, where every other week lands in the test set, leaks information across the split, because market conditions on neighbouring days are closely related.

1. Pick the split date before you run anything. Write it down. Moving the date after a bad result is a form of tuning.
2. Give the in-sample part about 70 to 80 percent of the history. It needs enough trades for the optimizer to work with. The out-of-sample part needs enough trades to grade, which the next section covers.
3. Check that both parts contain more than one kind of market. Look for a trending stretch, a sideways stretch, a volatile stretch and a quiet stretch on each side of the split. If the out-of-sample years are one long bull run, a strategy that is simply long will pass for the wrong reason.
4. Use the same costs, sessions and execution settings on both sides. A strategy that was tuned with next-bar execution and then tested with same-bar fills is not being tested at all.

Choice	Time split (recommended)	Random shuffle of bars
Order of data	In-sample first, out-of-sample after	Mixed across the whole history
Information leakage	None by construction	Neighbouring bars share conditions
Matches live trading	Yes: build on the past, trade forward	No: you cannot trade a shuffled future
Market regimes	Must be checked by hand on both sides	Spread automatically, but at the price of leakage

How much out-of-sample is enough: trades or years?

Enough out-of-sample is measured in completed trades, not calendar years. Two years of data hold about 24 trades for a swing strategy that trades once a month and about a thousand trades for an intraday strategy that trades twice a day. The first sample tells you very little; the second tells you a lot. Same dates, very different evidence.

The reason is how an average behaves. An average built from a dozen trades can move a long way if two or three trades are replaced; an average built from a few hundred trades barely moves. The one formula worth knowing:

uncertainty of the average trade = spread of single trades ÷ √N

Here the spread of single trades is how far individual trade results typically sit from the average (the standard deviation), N is the number of out-of-sample trades, and √N is its square root. Because the square root grows slowly, you need four times as many trades to halve the uncertainty.

A practical rule of thumb many systematic traders use: aim for at least a few hundred out-of-sample trades before trusting a per-trade figure, and treat anything under about 30 trades as a hint rather than a result. Bailey et al. (2014) show the mirror image of this: the more strategy variants you try, the longer the history you need before the best of them can be separated from luck. Before you trust a per-trade figure, the trade list in the performance report of AlphaPrime (Valynox) lets you count how many completed trades the out-of-sample period holds.

What are the signs of curve fitting?

Curve fitting shows up as an in-sample equity curve that is steep and smooth and an out-of-sample curve that goes flat or down at the split date. Curve fitting is tuning a strategy so closely to one stretch of history that it describes the noise in that stretch rather than a repeatable pattern. The shape change at the split line is the clearest sign, but there are others you can check before the out-of-sample run.

Sign	What you see	What a plausible edge looks like
Shape change at the split	Steep, smooth climb in-sample; flat or falling out-of-sample	Rougher climb in-sample; similar but somewhat shallower slope out-of-sample
Settings that only work at one value	A 20-bar lookback is good, 18 and 22 are poor	A broad region of neighbouring settings all behave alike
Many parameters for few trades	Eight tuned numbers and 60 in-sample trades	Few parameters, hundreds of trades per parameter
Profit concentrated in a few trades	Remove the five biggest winners and the result is flat	Result shrinks but stays positive after removing the top trades
Profit concentrated in time	One quarter made the year; other quarters are flat or negative	Most quarters contribute; no single period dominates

The second and third rows can be tested before you spend the out-of-sample data. Run the neighbouring settings and look at the parameter surface; the methods in walk-forward analysis cover this in detail. The research behind these signs, and why the best backtest in a large search is almost always inflated, is summarized in backtest overfitting: why the best backtest is usually inflated.

What should you do when out-of-sample is weaker but not broken?

Some drop from in-sample to out-of-sample is normal, so judge the drop with a rule you wrote before looking, using per-trade figures after costs. The in-sample result was the best of many tries and carries a selection bonus that the out-of-sample run does not. A shallower out-of-sample slope alone is not a failure. A reversal of sign, or a result that lives in one lucky month, is.

A workable approach is to measure how much of the in-sample average trade survives out-of-sample and to set zones in advance:

1. Compute the average trade after commission and slippage on both sides. Use the same cost settings; trading costs in backtests explains how to set them. Divide the out-of-sample figure by the in-sample figure.
2. Negative or near zero: reject. The strategy described noise.
3. Roughly 30 to 70 percent: weaker but not broken. Run the extra checks below and decide from them.
4. Above about 70 percent: pass to the next check. Little of the in-sample result was selection bonus.

The extra checks for the middle zone are the same ones that expose curve fitting: count how many out-of-sample months or quarters were positive, remove the five largest out-of-sample winners and see whether the result stays above zero, rerun with double the slippage, and confirm that neighbouring settings behave alike. A strategy that passes three of four is usually worth carrying forward; one that fails three is usually not. Write the pass count you require before you run the checks.

What you must not do is re-tune. Changing a stop, a lookback or a filter because the out-of-sample curve looked soft turns the out-of-sample period into in-sample data. The next section explains why that happens even without obvious re-tuning.

Why keep a final untouched period?

Every look at an out-of-sample result slowly turns that data into in-sample data, so reserve a last block of history that nothing touches until the strategy has passed every other check, and test on it exactly once. The leak is gradual. You test ten candidates out-of-sample, keep the three that did best, and the out-of-sample period has just helped choose a strategy. Bailey, Borwein, López de Prado and Zhu (2017) formalize this: the probability that the best-looking strategy is overfit rises with the number of candidates that were compared on the same data. Harvey and Liu (2015) reach the same conclusion from a different angle and propose discounting results for the number of tests behind them.

The fix is a third block with a different allowance of looks:

In practice this means setting aside the most recent six to twelve months of your history, or whatever length holds enough trades for the strategy's frequency, and not loading it into any research run. When one strategy has passed the in-sample search, the out-of-sample grading and the robustness checks, run it on the reserved block once, read the result against the same pre-written rule, and accept or reject. If you reject, the block is spent for that strategy family; the next idea gets graded on it only if the ideas are genuinely unrelated.

How does this connect to walk-forward analysis?

Walk-forward analysis repeats the in-sample and out-of-sample split many times along the history, which produces more out-of-sample trades from the same data. Walk-forward analysis is a test that tunes settings on one window, runs them unchanged on the window that follows, moves forward, and repeats until the joined out-of-sample segments form one result. For the swing strategy with 24 trades in a single two-year out-of-sample block, a rolling design over ten years can yield several times as many out-of-sample trades, each one taken with settings chosen before its data existed. The full method, window sizes and how to read the joined curve are covered in walk-forward analysis: why one backtest is not enough.

Two things do not change. Every window still obeys the one-direction rule from the first section: settings flow from in-sample to out-of-sample and never back. And the final untouched period still sits outside all windows. The walk-forward run is itself a check that you may run several times with different window lengths, so its joined out-of-sample curve is "looked at a few times" in the language of the previous section, not "looked at once". AlphaPrime runs walk-forward analysis as one of the steps in strategy validation and stops the later checks when a strategy fails an earlier one, which keeps the number of candidates that reach the later data small.

What this means if you use AlphaPrime: set the validation thresholds in strategy validation before the search starts, so the in-sample, out-of-sample and walk-forward decisions are made by rules you wrote in advance, and keep a recent block of your own bar data out of every research run for the one final test.

Frequently asked questions

What does out-of-sample mean in trading?

Out-of-sample data is the part of the price history you set aside and did not use when choosing a strategy's rules or settings. You run the finished strategy on it unchanged and read the result. Because the strategy never saw this data during tuning, the out-of-sample result is a fairer estimate of how the rules behave on new prices than the in-sample result is.

Can I use a random split instead of a time split?

For trading strategies, split by date, not at random. Neighbouring days share volatility, trend and news, so a random shuffle lets information from the test period leak into the tuning period. A time split, with the out-of-sample period after the in-sample period, also matches how you will trade: build on the past, trade into an unseen future.

How many out-of-sample trades do I need?

Enough that the average trade cannot move much if a few trades change. Many systematic traders aim for at least a few hundred out-of-sample trades and treat fewer than about 30 as a hint rather than a result. The uncertainty of the average shrinks with the square root of the trade count, so four times as many trades halve the uncertainty. If a single out-of-sample block holds too few trades, walk-forward analysis produces more from the same history.

Is it acceptable to re-optimize after a weak out-of-sample result?

Not if you want the out-of-sample result to keep its meaning. Changing a setting because the out-of-sample curve looked soft makes that data part of the tuning. Judge a weak result with a rule written before you looked, run the extra checks (profit spread over time, top trades removed, doubled slippage, neighbouring settings), and accept or reject. If you decide to change the strategy, treat it as a new candidate and grade it on data it has not seen.

References

• Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society, 61(5), 458–471.
• Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance, 20(4), 39–69.
• Harvey, C. R., & Liu, Y. (2015). Backtesting. The Journal of Portfolio Management, 42(1), 13–28.