Machine learning in algo trading: uses, overfitting, validation
Machine learning in trading predicts moves, classifies conditions or discovers rules, but noisy market data makes it overfit easily. Remove leakage, test forward in time on data the model never saw, count every model you tried and judge results after costs.
By Soren Huang · Published · updated · Editorial policy
Contents
- How is machine learning used in algorithmic trading?
- Why does machine learning overfit market data so easily?
- How do leakage and look-ahead get into features and labels?
- How do you validate a machine learning trading model?
- Is machine learning better than rule-based strategy search?
- Frequently asked questions
- References
Machine learning in algorithmic trading is used to predict returns from features, to classify market conditions or trade setups, and to discover trading rules. Market data carries little signal and much noise, so models overfit easily. Traders keep them honest by removing leakage, testing on later data with walk-forward splits, counting every model tried and judging results after costs.
How is machine learning used in algorithmic trading?
Most machine learning for algo trading does one of three jobs: it predicts a number, it sorts situations into classes, or it searches for readable rules. Machine learning is a set of methods that learn a rule from examples instead of having a person write the rule by hand.
Two more terms come up in every project. A feature is an input value computed from market data, such as the 14-bar RSI or the distance of price from a moving average. A label is the answer the model is trained to predict, such as "did the next 10 bars close higher?"
| Use | What the model learns | Typical label | Typical output |
|---|---|---|---|
| Feature-based prediction (regression) | How features relate to a future value | Return over the next N bars | An expected move, for example +0.1% over 5 bars |
| Classification | Which group a situation belongs to | Up or down, trade or skip, trending or ranging | A probability, for example 0.56 that the next 10 bars close higher |
| Rule discovery | If-then conditions that separate good and bad trades | The trading result of each rule | A rule a person can read, for example "buy when RSI is below 30 and price is above its 200-bar average" |
A popular practical use is the filter. An existing rule-based strategy produces the signals. A classifier then decides which of those signals to take, or how large the position should be. The model only has to answer a narrower question, which gives noise less room.
Why does machine learning overfit market data so easily?
Machine learning overfits market data because the repeatable part of price moves is tiny next to the random part, and a flexible model learns the random part too. Think of trying to record one whisper in a busy train station. A sensitive microphone records the whisper, but also every announcement and footstep. A model trained on that recording learns the station, not the whisper.
The signal-to-noise ratio is the size of the repeatable pattern compared with the size of the random movement around it. In photo recognition, a cat is a cat every time. In markets, the same feature pattern can be followed by an up move or a down move almost equally often. Overfitting is what happens when a model fits the random details of its training data so closely that it fails on new data.
Three things make it worse in trading:
- Markets change. A market is non-stationary, which means its behaviour drifts over time. A pattern learned from one volatility regime may not exist in the next one.
- There are fewer independent examples than bars. If each label covers the next 10 bars, two neighbouring rows share 9 of those bars. Ten thousand rows can hold far fewer independent answers.
- Many trials. Every model type, feature set and setting you try is a trial. Pick the best of many and some of its quality is luck. White (2000) built a test for this data-snooping problem. Bailey et al. (2014) showed how easily a strong-looking backtest appears from pure noise when enough variants are tried. Our article on why the best backtest is usually inflated covers this research in more detail.
Hypothetical example
A trader trains a tree-based classifier on 5,000 daily bars with 200 features. In-sample accuracy is 71%. On the next two years, which the model never saw, accuracy is 50.4%. The trader then tries 300 different settings and keeps the one with 54% on those two years. That 54% is now partly a result of the search itself, because the test period was used to choose the winner.
How do leakage and look-ahead get into features and labels?
Leakage is the most common reason a machine learning backtest looks too good: the model is reading information from the future. Leakage is any information in the training or test data that would not have been known at the moment the trade decision was made. Look-ahead bias is the backtest result of leakage: trades that use prices or values from after the decision.
| Source | How it leaks | Fix |
|---|---|---|
| Feature uses the bar's close, and the trade fills at that same close | The decision uses a price you could not trade at | Compute features at the close of bar t, fill at the open of bar t+1 |
| Scaling features with the mean and spread of the full history | Test-period statistics enter the training data | Fit any scaling on the training window only |
| Labels that cross the split | A training label's future bars lie inside the test period | Remove those rows (purging) and leave a gap |
| Random shuffling into folds | Nearly identical neighbouring bars land in both training and test sets | Split by time, never randomly |
| Choosing features by looking at all the data | The test period helped pick the inputs | Select features inside each training window |
| Price levels from a back-adjusted continuous contract | Past prices are shifted using roll gaps that happened later | Use returns or ratios, or handle rolls on unadjusted data |
Purging is the removal of training rows whose label period overlaps the test period. An embargo is a gap of bars between the training and test periods that neither side uses, so the end of one period cannot inform the start of the other.
A quick leak test helps: delay every feature by one extra bar and rerun the test. A real edge usually shrinks a little. A leak often collapses completely.
How do you validate a machine learning trading model?
Validate the model the way you would trade it: train on the past, test on the period that follows, move forward, and judge the joined results after costs. Walk-forward analysis is a test that repeats this train-then-test cycle across the history and joins only the test periods into one result. Our guide to walk-forward analysis and window lengths explains how to set the windows.
- Lock away a final holdout. Keep the most recent part of the history untouched. Use it once, at the very end.
- Split the rest by time. Train on one window, test on the next, then move forward. Retrain at the same rhythm you would use live.
- Purge and embargo at every split. Apply the leakage fixes from the table above in each window, not just once.
- Turn predictions into trades. Accuracy is not profit. Simulate the trades with commission, slippage and next-bar fills. Our article on trading costs in backtests shows how to set them.
- Count your trials. Write down every model, feature set and setting you tried. Harvey and Liu (2015) argue that results should be discounted for the number of tests run. The deflated Sharpe ratio is one way to apply such a correction.
- Vary the setup. Rerun with different window lengths and retraining rhythms. A model that only works with one combination is fragile. Bailey et al. (2017) describe a way to estimate how likely it is that the chosen variant is overfit.
Hypothetical example
A classifier is right 55% of the time in walk-forward tests. Its winners average 8 ticks and its losers average 10 ticks. Before costs, the average trade is 0.55 × 8 − 0.45 × 10 = −0.1 ticks. With 2 ticks of round-trip costs, it is −2.1 ticks. A model that is "right" more often than wrong still loses money.
Is machine learning better than rule-based strategy search?
Neither approach is protected from overfitting; they differ mainly in what they search and how easily you can inspect the result. Rule-based strategy search is a method that generates and backtests many combinations of readable entry and exit rules within set parameter ranges. Both methods fit candidates to the same history, so both need the same validation discipline.
| Machine learning model | Rule-based strategy search | |
|---|---|---|
| What is searched | Model weights or tree splits learned from features | Combinations of rules, indicators and parameter values |
| Form of the result | A trained model, often with many internal values | Entry and exit rules a person can read |
| How you inspect it | Indirectly, through feature importance or test results | Directly, by reading the rules and their trades |
| Where overfitting hides | Model flexibility, leaky features, many settings tried | Many rule combinations tried, narrow parameter choices |
| Running it live | Usually needs a separate program to compute predictions | Can be written in a trading platform's scripting language |
| Main strength | Can find interactions between many inputs | Simple to check, explain and monitor |
The limits are shared. Neither method turns a backtest into a forecast. Both get worse as the number of trials grows without correction. Both fail when costs or execution are modelled loosely. ML adds the leakage risks above. Rule search adds the risk of a long list of conditions that each removed a few bad trades from the past.
AlphaPrime (Valynox) takes the rule-based route. Its strategy generation turns research ideas into testable trading rules, with the indicators, parameter ranges, entry and exit conditions and search constraints you choose. Its strategy validation includes walk-forward analysis, so each candidate is tested on periods it was not fitted to.
What this means if you use AlphaPrime: treat each generated rule set like an ML candidate, which means costs in the search, walk-forward checks on unseen periods and an honest count of how many variants you looked at before choosing one.
Frequently asked questions
Can machine learning predict futures prices?
Machine learning can find weak patterns in past data, but market data is noisy and markets change, so any edge is small and can fade. A model's backtest describes the past, not the future. The useful question is whether the edge survives walk-forward tests after costs.
Which machine learning algorithm is best for trading?
No single algorithm is best. Simpler models such as logistic regression or shallow decision trees have fewer ways to memorize noise, which matters with low signal-to-noise data. Choose the model after the validation setup is fixed, and count each model you try as a trial.
What is data leakage in trading machine learning?
Data leakage is information in the training or test data that was not available when the trade decision was made. Common causes are features computed with the same bar's close used for the fill, scaling with full-history statistics, labels that cross the train-test split, and random shuffling of time series. Leakage makes backtests look far better than live trading.
Why is cross-validation with random folds wrong for market data?
Neighbouring bars are almost identical, so random folds put near-copies of the same situation in both the training and test sets. The model then looks skilled because it has effectively seen the test data. Split by time instead, and leave a gap between training and test periods.
References
- White, H. (2000). A Reality Check for Data Snooping. Econometrica, 68(5), 1097–1126.
- Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society, 61(5), 458–471.
- Harvey, C. R., & Liu, Y. (2015). Backtesting. The Journal of Portfolio Management, 42(1), 13–28.
- Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2017). The Probability of Backtest Overfitting. Journal of Computational Finance, 20(4), 39–69.
What AlphaPrime does here
AlphaPrime's strategy generation searches readable rule-based strategies within the constraints you set, and its strategy validation adds walk-forward analysis and Basic, Standard or Extensive validation before you keep a candidate.