Walk-forward out-of-sample P&L: does a positive sign mean skill?

Does a positive out-of-sample P&L mean the strategy was chosen well?

No. A positive walk-forward out-of-sample P&L only shows that the strategy made money in those test windows. If the market rose and the strategy was mostly long, it could have earned that without any skill. To judge selection quality, compare each candidate with a baseline over the same windows, across many windows and trades.

Out-of-sample P&L is the profit or loss a strategy makes on data that played no part in choosing it. In walk-forward analysis, you optimize on one window of history (in-sample), trade the chosen settings unchanged on the next window (out-of-sample), and repeat. One fold is one such pair of an in-sample window and the out-of-sample window that follows it.

The sign of the out-of-sample P&L answers one question: did the strategy make money here? Selection quality asks a different question: did the selection process pick something better than an unskilled choice would have earned in the same period? Think of a sailing race. A boat that arrives early with the wind behind it may have a mediocre captain. You only learn about the captain by comparing boats that sailed the same course on the same day.

Why can a rising market make almost any long strategy look good?

A long position earns the market's move whether its entries were smart or random, so a rising test window lifts every long strategy. Buy-and-hold is the result of holding one contract long for the whole window and rolling into the next contract at each expiry. On a back-adjusted continuous series, it is roughly the price change from the first bar of the window to the last, before roll costs.

A long-only strategy that is in the market most of the time inherits most of that move. If the out-of-sample windows happen to fall in a strong uptrend, it will show a positive sign even if its entry and exit rules add nothing. The mirror case works the same way. A short strategy looks clever in a falling market. A long strategy that loses less than a falling market may have been chosen well, even though its sign is negative.

How do you separate the strategy's contribution from the market's move?

Subtract what a no-skill position with the same direction and the same time in the market would have earned in the same window. The simplest version is one line:

Excess = S − e × M

• S is the strategy's out-of-sample P&L in the window, after commission and slippage.
• M is the buy-and-hold P&L of the same number of contracts over the same window.
• e is the strategy's net exposure: the share of bars it was long minus the share of bars it was short. A strategy long 60% of the time and short 10% of the time has e = 0.5.

The excess result is the part of the out-of-sample P&L that the market's move does not explain. A baseline is the result a no-skill rule would have produced under the same conditions. Different baselines remove different amounts of luck:

A random-entry baseline keeps the strategy's number of trades, holding times and direction, places the entries at random bars, and charges the same costs. Repeat it many times and you get a range of no-skill results for that window. A strategy whose result sits near the middle of that range shows no evidence of skill there. One that sits near the top of the range shows some, in that window.

Charge the baseline the same costs as the strategy when the baseline trades. Otherwise a strategy can look worse only because it pays for its trades and the baseline does not; see trading costs in backtests. For strategies that trade both directions, compare the long side and the short side separately. The long and short analysis in the Performance Report gives you that split.

Why is one out-of-sample sign weak evidence?

A sign taken from a few trades, in one fold and one market regime, is close to a coin flip, and choosing among many candidates makes a lucky positive almost certain. Four separate problems stack up:

1. Few trades. With a handful of trades in a window, one large winner or loser decides the sign. The sign then describes one trade, not a rule.
2. One fold. A single out-of-sample window is one draw. A strategy with no edge at all ends a window positive close to half the time, because its results are scattered around zero.
3. Regime dependence. A regime is a stretch of market with a typical behaviour, such as a steady trend, a choppy range or a high-volatility selloff. If every out-of-sample window falls in one regime, all of them share the same tailwind or headwind.
4. Many candidates. If you look at ten unrelated candidates with no edge, the chance that at least one ends positive in a window is above 99.9%. The best-looking survivor is then the expected result of searching, not a discovery.

Research on data snooping makes the same point formally. White (2000) proposed the Reality Check, which tests whether the best of many models beats a benchmark after allowing for the fact that many models were searched. Sullivan, Timmermann and White (1999) applied this idea to a large set of technical trading rules. Bailey et al. (2017) measure the probability of backtest overfitting: how often the configuration that ranks best in-sample falls below the median out-of-sample. Harvey and Liu (2015) argue that backtested results should be discounted for the number of tests behind them. More on this in backtest overfitting research.

How do you rank candidates on the same out-of-sample windows?

Rank candidates on their excess over a baseline, computed on identical windows with identical costs, and prefer consistency across windows over a large total. Comparing raw P&L from different windows compares market levels, not candidates.

1. Fix one set of out-of-sample windows and use it for every candidate.
2. Use the same costs, contract size and execution assumptions for every candidate.
3. In each window, compute the excess for each candidate against its exposure-matched baseline.
4. Record four numbers per candidate: total excess, the number of windows with positive excess, the worst window's excess, and the trades per window.
5. Rank first on the count of windows that beat the baseline and on the worst window. Use total excess as a tiebreaker.
6. Repeat with a second window layout, for example different in-sample and out-of-sample lengths. A ranking that only holds for one layout is fragile.

In AlphaPrime (Valynox), walk-forward analysis is part of strategy validation, with custom thresholds you set. The strategy databank lets you search and sort candidates and revalidate them, so every finalist can be checked on the same windows and settings before you compare it with the market's move.

What can walk-forward analysis tell you, and what can't it?

Walk-forward analysis tells you whether your way of choosing and re-tuning a strategy held up on unseen data in the regimes your history contains. It does not tell you how that process will behave in a regime the history never showed.

Regime changes are where the method is weakest. Each in-sample window looks backward. After a break in market behaviour, the first out-of-sample windows test settings fitted to a market that has just ended. Those windows show you how the process behaves at a turning point, so read them separately rather than letting the good windows on either side average them away. Make sure your history contains several different regimes, and look at excess per regime as well as in total.

What this means if you use AlphaPrime: run walk-forward on every finalist with the same windows and costs, then read each window's result next to the market's move over that window before you rank anything. A candidate that beats its baseline in most windows and has a tolerable worst window tells you more than the one with the largest positive total.