Robustness testing: try to break a strategy before you trust it

Contents

1. What is robustness testing in trading?
2. How do you test whether a trading strategy is robust?
3. What is a parameter plateau?
4. How much extra slippage should a strategy survive?
5. Should a strategy work on other markets?
6. Which sessions, periods and data changes should a strategy survive?
7. Which order should robustness tests run in?
8. What does a failed robustness test tell you?
9. Frequently asked questions
10. References

What is robustness testing in trading?

Robustness testing is deliberately changing things a real strategy should not care about (nearby parameters, costs, the market, the session, the exact prices) and keeping only what survives. Passing one test proves little: the same NQ breakout rule that was profitable in all nine parameter neighbours lost money on 12 of 17 markets.

A bridge inspector does not drive one car across a bridge and call it safe. He loads it with more weight than normal traffic, from more than one direction, and watches what bends. A single backtest is the one car. Robustness testing is the extra load: you move the parameters, raise the costs, change the market, cut out sessions, swap the data, and you keep the strategy only if it still stands. The goal is not to find the strategy with the best backtest. The goal is to find the strategy you could not break.

This matters more today than it did twenty years ago because of the search. When software tries thousands of rule combinations for you, some combination will fit the quirks of one stretch of history by chance, and that combination will have the best backtest. Quirks are the default result of a large search, not the exception. The backtest overfitting research explains why the best of many backtests is usually inflated. Robustness testing is the practical response: it does not ask how good the best backtest was, it asks what happens when the conditions that produced it are disturbed.

One measured example runs through this whole article. The rule is a long-only breakout on NQ 60-minute bars: enter when the close crosses above the highest high of the previous N bars, exit when the close crosses below a simple moving average of M bars, with N = 20 and M = 50. The data runs from 2018-09 to 2026-10, one contract, with $14.75 commission per side including 2.5 extra ticks, plus 1 tick slippage. Both results come from the hosted backtest of AlphaPrime (Valynox), run 2026-10-31, and the full tables are in the sections below.

• Parameter neighbours: all nine cells of a 3×3 grid around N = 20 / M = 50 finished with a positive net profit.
• Other markets: the same rule, same period, finished above zero on 5 of 17 futures markets and below zero on the other 12.

Same rule, same dates, same cost model. One test says "solid"; the next says "fitted to one chart". A trader who stopped after the parameter grid would have trusted a rule that the next test took apart. That is the whole case for running a set of tests, writing the pass line for each one down beforehand, and reading them together.

How do you test whether a trading strategy is robust?

Each robustness test is aimed at one kind of fake signal, so you run a set of them, and no single test replaces another. A fake signal is a backtest result that comes from something other than a repeatable pattern in prices: a curve-fit parameter, an edge thinner than the cost of trading, one lucky chart, a calendar accident, five good days, one vendor's exact prices, a selection process that only worked backwards, or a lucky order of trades. The table lists the tests, what each one changes, and what a pass looks like.

Test	What you change	Which fake it catches	What a pass looks like
Nudge the parameters	Every numeric input moved up and down, all combinations rerun	A curve-fit peak: one sweet spot with losing neighbours	Most neighbours keep the sign; the chosen value sits inside a broad region
Raise the costs	Extra commission or slippage per round trip, in steps	An edge thinner than the spread	Net profit stays positive at a multiple of the realistic round-trip cost
Switch the market	Same rule on related instruments, each with its own costs	A rule fitted to one chart	The sign holds on most related markets; the worst related market is not a disaster
Cut sessions and periods	Trades split by weekday, hour, year, day or night session	A calendar accident	No single slice carries the whole profit
Remove the biggest winners	The five largest winning trades deleted	Five good days	The equity curve still rises
Add data noise	Same rule on a second vendor's bars	Dependence on exact prices	The two trade lists mostly agree
Walk-forward analysis	Tune on one window, test unchanged on the next, repeat	A selection process that does not hold forward	The joined out-of-sample curve keeps a useful share of the in-sample result
Monte Carlo on trades	The trade order reshuffled or resampled many times	A lucky trade order	The drawdown at a chosen confidence level fits the account

Two of the tests work on the process rather than on the rule. Walk-forward analysis is a test that tunes the parameters on one window of data, applies them unchanged to the window that follows, then moves forward and repeats, so it catches a way of choosing parameters that only worked when you could see the answer. A Monte Carlo backtest is a test that reshuffles or resamples the strategy's own trades many times to show the range of equity curves and drawdowns the same trades could have produced, so it catches a smooth curve that came from a lucky order. Neither of them changes the market, the costs or the data, which is why they come last in the list and not first.

Every test needs enough trades to mean something. There is no exact number, but below about 100 trades the tests start to measure noise instead of the strategy: with 40 trades, removing the five best removes an eighth of the sample, and a weekday cut leaves eight trades per weekday. A rule that produces 60 trades in ten years is not ready for robustness testing; it is ready for a longer history or a shorter bar size.

What is a parameter plateau?

A parameter plateau is a region of parameter values where the neighbours of the chosen value all keep most of the result, and a spike is one best cell surrounded by losers. The plateau is what a real pattern looks like, because a pattern in prices does not switch on at a 20-bar lookback and off at 18. The spike is what a curve fit looks like, because the search found the one value that lined up with the noise of one history.

Think of a hill and a flagpole. If you stand on a hill, a step in any direction changes your height a little. If you stand on top of a flagpole, any step is a fall. A parameter grid tells you which one you are standing on. The practical rule that follows is to pick the parameter from the middle of the hill, not from its highest point: the highest point of a hill is still partly luck, and the middle is where a small drift in the market costs the least.

The measured grid below is what a plateau looks like in a real run. The rule is the NQ 60-minute breakout from the first section. N, the number of bars in the breakout lookback, takes the values 10, 20 and 40; M, the length of the exit moving average, takes 20, 50 and 100. Every combination was run with the same commission plus 1 tick slippage, on the same bars, one contract, long only.

Three things to read in this grid, in the order you should read them:

1. The share of cells with the right sign. All nine are positive. That is the pass criterion for this test, and it is the only number the test is really about. The chosen cell, 20 / 50, is not the best cell; it sits below the middle of the nine by net profit, and that is fine. The test asks whether the neighbourhood is sound, not whether you picked the peak.
2. The shape. Within every row, a longer exit average earned more and traded less: 20 / 20 made $44,300 on 990 trades, 20 / 100 made $376,032 on 513 trades. That is a slope, not a spike. A slope tells you which direction the pattern lives in, and it also tempts you to chase it. Resist the temptation to move to 100 just because it is the top of this grid; a grid that stopped at 100 cannot tell you what happens at 150, and the right response to a slope is to widen the grid, not to jump to its edge.
3. The weakest cells. 10 / 20 and 20 / 20 have profit factors of 1.032 and 1.042. Profit factor is gross profit divided by gross loss, so these cells kept barely more than they gave back. They pass the sign test, but they tell you the short exit is where the rule is thin.

One condition makes this plateau worth something: it already includes commission plus 1 tick slippage. A grid run at zero cost shows a wide, comfortable plateau for almost any rule that trades often, because every one of those small, frequent trades looks free, and the plateau shrinks or vanishes once real fills are charged. AlphaPrime's backtesting charges fees and slippage inside every evaluation, with next-bar execution, so a grid it produces is already net of costs; a grid from any tool that was run at zero cost needs the cost stress of the next section before you trust its plateau.

How much extra slippage should a strategy survive?

The extra cost a strategy can take before its profit is gone equals its average trade, so the cost stress test is a way of asking how many ticks of safety margin the rule has. Slippage is the difference between the price your backtest assumed and the price a real order would fill at, and it is the cost most backtests underestimate, because it depends on the order type, the time of day and the mood of the market on the day you trade. The question is not whether your slippage estimate is exactly right. It is how wrong it can be before the strategy stops working.

The arithmetic needs one formula. Break-even extra cost per round trip = net profit ÷ number of trades. The left side is the extra charge per trade that would bring the net profit to zero. Net profit is the backtest result after the costs already charged, and the number of trades is how many round trips paid for it. The result is simply the average trade: every trade pays the extra cost, so the extra cost that erases the profit is the profit per trade.

The table shows the straight line you should expect. Each $50 step removes the same amount of profit, because each of the 709 trades pays the extra $50, and the line crosses zero between $150 and $200, where the average trade of $193 says it should. The lesson is that you do not need to run the stress table at all to know where the break-even is; you need the average trade, and the trading costs in backtests article shows how to compare that average trade with the round-trip cost you really pay.

What makes this rule pass the cost stress is the size of the average trade in ticks. One NQ tick is $5.00 per CME's contract specifications, so $200 of extra cost per round trip is forty ticks, or twenty extra ticks of slippage on each side, which is far more than a liquid NQ market order normally sees. Compare that with an intraday rule on ES whose average trade is three or four ticks: going from one tick of slippage per side to two adds two ticks per round trip, and that step alone takes half or more of the average trade. Rules like that die here, and it is better that they die in the test than in the account. Exchanges can change tick sizes and contract terms, so check the exchange's contract page for the market you test.

Set the pass line before you look at the result, as a multiple of the realistic round-trip cost rather than as a fixed dollar figure, so that it scales with the market. A common form is "the rule must stay profitable at double the realistic round-trip cost" or "at one extra tick per side on top of a realistic estimate". For the NQ rule, the realistic cost in the run is $14.75 commission per side including 2.5 extra ticks, plus 1 tick slippage, and the rule survives far more than double that, so it passes. The test says nothing about whether the realistic estimate itself is right; it says how much room there is for it to be wrong.

Should a strategy work on other markets?

A rule should keep its sign on related markets, not on random ones, and the measured run shows why this is the hardest test: the same rule that passed every parameter neighbour was above zero on only 5 of 17 markets. The logic of the test is that a rule which captures something general about how prices move, such as the tendency of a breakout to follow through, should show the same sign on markets that move in a similar way, even if the profit is different because the tick values and the costs are different. A rule that only works on the chart it was found on was fitted to that chart.

Related means markets that share the behaviour the rule is built on. For an NQ breakout rule, ES, YM and RTY are the obvious relatives: all four are US stock index futures that trade the same hours and react to the same news. Gold and silver are a reasonable second circle for a trend rule. Grains, bonds and currencies are a fair test of whether the rule is a general trend rule, but failing there is a weaker signal than failing on ES. The run below applied the N = 20 / M = 50 rule, unchanged, to every one of the 17 markets of the hosted backtest, each with its own commission and slippage.

Read the result in circles, from the nearest relatives outwards. Among the stock index futures, ES and YM kept the sign and RTY did not, and the ES result of $17,401 with a profit factor of 1.030 is close enough to zero that a slightly different cost estimate could flip it. Gold was the strongest market in the run, well above NQ itself, and silver also finished above zero. Bonds, grains, currencies, energy and copper all lost, and the bond markets had the lowest profit factors: ZB at −$156,490 (0.508) and ZN at −$80,205 (0.464). The rule is long only, so it can only earn in a market that spent part of the period rising, and that is one reason a long-only breakout looks so different across this list.

What this run shows is that the rule is not a general trend rule and that even inside its own family it is not reliable: a rule you would trade on NQ, YM and ES is a different thing from a rule you would trade on any market. What it does not show is whether a short-side version, a different bar size or a different exit would change the picture, because none of those were run. It also says nothing about why GC and SI worked; a result on two metals markets is a hint for further research, not a finding.

The pass line for this test has two parts, and you write both down first: the share of related markets that must keep the sign, and the worst loss on a related market that you accept. A rule that loses a little on its weakest relative can still pass; a rule that loses on most of its family fails, however good the origin market looks. Failing here is the most common result for a rule that came from a large search, and that is the reason to run this test before the expensive ones.

Which sessions, periods and data changes should a strategy survive?

Cut the trades by weekday, hour, year and session, remove the five biggest winners, and rerun the rule on a second vendor's bars, because a result that depends on one weekday, five good days or one vendor's exact prices was never an edge. These tests change nothing about the rule. They change which trades you count, or which prices the rule sees, and a real pattern should not care about either.

The slicing tests work on the trade list you already have, so they need no new backtest. Run them in this order:

1. By year. Split the net profit by calendar year. One or two losing years are normal for a trend rule; one winning year that carries the whole result is a period accident, usually a single strong trend.
2. By weekday. Group the trades by the weekday of entry. If one weekday holds most of the profit and the others are flat or negative, the rule found a calendar quirk, and a quirk of that kind rarely repeats.
3. By hour. Group intraday trades by the hour of entry. A rule that only earns in the first hour after the cash open is not wrong, but it is a different rule from the one you tested, and it should be written as an opening-hour rule and retested as one.
4. By session. Separate trades taken in the day session from trades taken in the overnight session, where the market has one. A rule found on full-session bars that only earns overnight is depending on thin liquidity, where slippage is worst; the night sessions in futures backtests article covers what changes when the overnight bars are removed.
5. Remove the five biggest winners. Delete the five largest winning trades and redraw the equity curve. This is the one check to keep if you could keep only one. When the curve goes flat, the strategy never had an edge; it had five good days.

The data test is different in kind, because it changes the inputs instead of the outputs. Two vendors' bars for the same contract are rarely identical: one builds the bar from a different feed, one stamps the bar at a different time boundary, one handles the roll between contract months a day apart. The differences are usually one tick here and there, and a rule written on exact prices turns those ticks into different trades. "Close crosses above the 20-bar high" fires when the close is one tick above the line and stays silent when it is one tick below. With vendor A the trade exists; with vendor B it does not; from that bar on, the two equity curves are no longer the same strategy.

The way to run this test is simple and manual. Rerun the same rule, unchanged, on the second vendor's bars for the same period. Put the two trade lists side by side, sorted by entry time, and count the trades that appear in one list and not the other. A robust rule produces mostly the same trades with a handful of differences at the edges, and the net profit of the two runs lands in the same region. A fragile rule produces two lists that diverge after the first disagreement and two net profits that have little to do with each other. Apply the same pass line you set for the parameter neighbours: if you required most neighbours to keep the sign, require the second data source to keep the sign and to agree on most trades. If you have only one data source, the nearest substitute is the backtest data quality checks on the one you have, because bad ticks and shifted time stamps produce the same kind of fragile signal as a vendor difference.

Which order should robustness tests run in?

Run the parameter grid and the cost stress first, because they are cheap and remove the most candidates, then the multi-market run and the session cuts, then walk-forward analysis and Monte Carlo, and stop a candidate at the first pass line it misses. The order matters because the tests have very different prices. A parameter grid is a handful of extra backtests; a multi-market run is one backtest per market; the slicing tests need no new backtest at all; walk-forward analysis is a full optimization for every window, which can be hundreds of runs; a Monte Carlo on trades is cheap again, but it only means something for a strategy that has survived everything before it. Spending walk-forward time on a candidate that the parameter grid would have removed in seconds is the most common waste in strategy research.

1. Parameter grid. Pass line: the share of neighbours that must stay profitable. Removes the curve-fit peaks, which are the largest group of fakes a search produces.
2. Cost stress. Pass line: the extra cost per round trip the rule must survive, written as a multiple of the realistic round trip. Removes the thin edges, which are the second largest group.
3. Multi-market. Pass line: the share of related markets that keep the sign and the worst loss on a related market. Removes the rules fitted to one chart.
4. Session and period cuts, biggest-winner removal. Pass line: no single slice carries the result, and the curve still rises without the five best trades. Removes calendar accidents and five-good-days strategies.
5. Walk-forward analysis. Pass line: the joined out-of-sample curve keeps a set share of the in-sample result, and most windows are positive. Removes selection processes that only worked backwards; the walk forward optimization explained article covers the window choices, and the in-sample vs out-of-sample article covers how much data to hold back.
6. Monte Carlo on trades. Pass line: the drawdown at a chosen confidence level fits the account you will trade. Removes smooth curves that came from a lucky order; the free Monte Carlo simulator produces the drawdown range from a pasted list of trade results.

The stop rule is what makes the order pay. A candidate that misses the parameter pass line is out, and nothing further is spent on it; a candidate that reaches walk-forward has already shown a plateau, a cost margin, a family of markets and a trade list that does not depend on five days. In AlphaPrime's strategy validation, the checks run in sequence against thresholds you set and a candidate that misses one threshold stops there, so walk-forward and the later checks only run on the survivors of the cheaper tests, which is the same funnel drawn above.

Two habits keep the funnel honest. First, the pass lines are written before the first candidate is run, and they do not move afterwards; a pass line that is adjusted after seeing the results is a result, not a test. Second, the count of candidates that entered and the count that left each stage is recorded, because that count is itself information: if nine out of ten candidates fall at the parameter grid, the generation settings are producing curve fits and the fix is upstream, in the search, not in the tests. The overfitting in strategy generation article covers what to change in the search when that happens.

What does a failed robustness test tell you?

A failure tells you which kind of fake you found and that the search should continue, while a pass only narrows the odds. A failed parameter grid says the search found a peak; a failed cost stress says the pattern is real but too small to pay for; a failed multi-market run says the rule learned one chart; a failed biggest-winner test says the history had five good days. Each failure points at a specific weakness, which is more useful than a vague "it did not work", and each one tells you what to change before the next search. A pass tells you less. It says that one kind of fake was not present, nothing more, and a candidate that passes every test can still be a fake that happened to survive.

The reason a pass proves little is the search behind it. Run enough strategies through enough tests and some junk passes all of them by chance, because ten checks on a thousand candidates is still a search, now a search for the candidate that passes checks. Sullivan, Timmermann and White reran thousands of technical trading rules on a century of Dow data and showed that the best rule has to be judged against the size of the search it came from, and that the best rule's edge did not hold in the ten years after their sample. White's Reality Check is the statistical form of that idea: it tests whether the best of many models beats a benchmark once the whole set of models that were tried is taken into account. Bailey and his co-authors go one step further and estimate the probability that the configuration which looked best in-sample does worse than the median configuration out-of-sample, and with enough trials that probability is high unless something limits the search. Robustness tests do not remove this problem; they reduce the number of candidates, so the search among survivors is smaller than the search among raw backtests.

The one measured example of what survivors look like afterwards is a run on ES and TAIEX futures. Held-out data is a stretch of bars at the end of the history that is kept outside the research entirely, so that nothing, neither the generation nor the walk-forward, can see it.

Run on 2026-10-15 with an RTX 3060 Ti, factory settings, conservative costs and 1 contract. ES 2014-01 to 2026-09 (held out from 2022-10); TAIEX futures 2015-10 to 2025-03 (held out from 2022-04). The held-out bars were not in the database during generation or walk-forward.

• ES 60-min: about a thousand evaluated, 250 passed basic filters, 126 passed walk-forward. On held-out data 91% of those that passed walk-forward were profitable, versus 53% of those that failed.
• TAIEX futures 60-min: about a thousand evaluated, 306 passed basic filters, 168 passed walk-forward. On held-out data 85% of those that passed walk-forward were profitable, versus 66% of those that failed.

Read this run for the gap, not for the pass share. Both held-out periods were rising markets, ES about +116% and TAIEX futures about +29%, and in a market that more than doubled, junk with a long bias makes money too. That is why over half of the ES strategies that failed walk-forward still finished profitable on held-out data: the market carried them. The 91% on its own is not evidence that walk-forward found real strategies; the difference between 91% and 53% is, because both groups were lifted by the same market and only the passed group was lifted that much further. The same reading applies to TAIEX futures: 85% against 66%, a smaller gap in a smaller rise.

This run shows that, on these two markets and these two held-out periods, strategies that passed one robustness test were profitable on unseen data more often than strategies that failed it. It does not show what any one strategy will do next, it does not show how large the profits were, and it used walk-forward as the only filter: whether the parameter grid and the multi-market check would widen the gap has not been measured. A share of profitable strategies on a past held-out period is a statement about a filter, not a forecast about an account.

The final decision, then, is never one test and never one number. It is the set of pass lines you wrote down before the first run, applied in order, with the candidate stopped at the first one it misses, and the survivors read against the size of the search they came from. A strategy that reaches the end of that funnel has earned a smaller claim than "it works": it has earned "nothing I tried could break it". That is the most a backtest can give you, and it is a great deal more than the best backtest in the grid.

Frequently asked questions

What is robustness testing of a trading strategy?

Robustness testing is rerunning a strategy after changing things that should not matter much: nearby parameter values, higher costs, a related market, a different session or period, and slightly different price data. Each change catches a different kind of fake signal. A strategy that keeps most of its edge through all of them is more likely to rest on a real pattern than on a quirk of one history.

Is walk-forward analysis a robustness test?

Yes. It checks whether the way you choose parameters still works on data after the optimization window, so it tests the selection process rather than the rule. It does not test other markets, higher costs or a second data source, so it belongs in the set next to those checks, and because it is the most expensive test it should run after the cheaper checks.

How many trades do you need before robustness testing makes sense?

There is no exact number, but below roughly 100 trades most results are noise. With 40 trades, removing the five best removes an eighth of the sample, and a weekday cut leaves eight trades per weekday. A rule with too few trades needs a longer history or a shorter bar size before the tests can say anything.

What is the difference between a parameter plateau and a spike?

A plateau is a region where the neighbouring parameter values all keep most of the result, which is what a real pattern looks like because a pattern in prices does not switch off when a lookback moves from 20 to 18. A spike is one best cell surrounded by losing cells, which is what a curve fit looks like. Pick parameters from the middle of a plateau, not from its highest point.

Does passing every robustness test mean the strategy is real?

No. Passing all tests means none of the specific fakes they look for was found, and when many candidates are tested some junk passes every test by chance. Read the survivors against the size of the search they came from, and treat a pass as a reason to trade small and keep measuring rather than as proof.

References

• Sullivan, R., Timmermann, A., & White, H. (1999). Data-Snooping, Technical Trading Rule Performance, and the Bootstrap. The Journal of Finance, 54(5), 1647–1691.
• White, H. (2000). A Reality Check for Data Snooping. Econometrica, 68(5), 1097–1126.
• Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance, 20(4), 39–69.
• CME Group. E-mini Nasdaq-100 Futures contract specifications (contract size $20 × index, minimum tick 0.25 index points = $5.00). cmegroup.com/markets/equities/nasdaq/e-mini-nasdaq-100.contractSpecs.html