Backtesting software compared: 4 tool types, 6 checks before you pay

Contents

1. What should you look for in backtesting software?
2. What are the four types of backtesting tools?
3. How do you compare them on data, fills and costs, validation, speed, export and price?
4. What do free and built-in testers leave out?
5. When does a strategy builder make sense, and when is it overkill?
6. Which type fits which trader?
7. How do you run a trial so the comparison is fair?
8. Frequently asked questions
9. References

What should you look for in backtesting software?

Compare backtesting software on six points, in this order: whether it accepts your own data, how it fills orders and whether costs are on by default, what validation it offers, how many candidates it tests in the time you have, how a strategy leaves the tool, and price last. The four tool types differ most on the fourth point.

A backtest is a run of fixed trading rules over historical bars that reports the trades those rules would have taken. The software only decides how honest that run is and how many runs you can afford. Choosing it is like buying a used car: check the engine, the brakes and whether it fits your garage before you look at the price sticker. Whichever tool you pick, check the numbers it reports against the free drawdown calculator.

1. Own data in. You must be able to load the bars you actually trade, including night sessions and your own contract rolls. A tool that only tests on data it supplies locks your research to that data.
2. Fills and costs. Commission and slippage should be part of every run, and an order decided on a bar's close should fill on the next bar, not on that same close.
3. Validation built in. Walk-forward analysis is a test that tunes a strategy on one window of data and then runs the chosen settings unchanged on the window that follows. If the tool cannot do this, you will do it by hand or not at all.
4. Candidates per run. Count how many rule or parameter variants the tool can evaluate under identical settings in one session. This decides whether you refine one idea or search for new ones.
5. Export out. A strategy that cannot leave the tool as source code has to be rewritten by hand, and every rewrite is a chance to change the rules.
6. Price last. A free tool that defaults to zero costs can cost more than any subscription, because it approves strategies that lose after fees.

What are the four types of backtesting tools?

Platform testers, Python libraries, cloud platforms and strategy builders solve different jobs: the first three test one strategy you already wrote, and the fourth searches many candidate rules for you from constraints you set. Knowing which job you have narrows the choice before any feature list.

• A platform tester is the backtest function built into a charting or trading platform. You write the rules in the platform's own language, run them on the chart's data and read a report inside the same program. It is the natural first stop for a trader who already has a chart open.
• A Python library is a set of code functions that run your rules over a data file you load yourself. You write the strategy, the data loader, the cost model and often the report. Everything is under your control, and everything is your job.
• A cloud platform is a backtesting service that runs in a browser on the vendor's servers. You write or click together one strategy and the service supplies compute and often data. Nothing is installed, and your strategy and results live on the vendor's machines.
• A strategy builder is software that generates many candidate rule sets from constraints you set, such as allowed indicators, parameter ranges and exit types, and evaluates all of them under the same data, costs and execution settings. AlphaPrime (Valynox) is a strategy builder of this kind: it generates, backtests and validates candidates, keeps them in a databank and exports the survivors as source code.

The difference is who writes the rules. With the first three types, every strategy that gets tested is one you thought of. With a builder, the software proposes rules within your limits, and your work shifts from writing rules to setting constraints and judging results.

How do you compare them on data, fills and costs, validation, speed, export and price?

One checklist with the same six questions for each type shows that no type wins every row, so the right choice depends on which rows matter for your workflow. The table below gives the usual answer for each type; a specific product can differ, so confirm each cell during a trial.

Check	Platform tester	Python library	Cloud platform	Strategy builder
Own data accepted	Usually yes, through the platform's data feed or import	Yes, any file you can read into code	Depends; some accept uploads, some only supply their own	Usually yes; AlphaPrime runs on the customer's own historical bar data
Next-bar fills and costs on by default	Usually no; costs are often zero until you set them	Depends on setup; you write the fill logic	Depends on setup	Usually yes; AlphaPrime evaluates each candidate with fees, slippage and next-bar execution
Walk-forward or out-of-sample built in	Depends; some platforms offer it as an add-on	Depends on setup; you write the loop	Depends on setup	Usually yes; AlphaPrime's validation includes walk-forward analysis
Candidates per run	One, or a parameter grid in an optimizer	One, or a loop you write	One	Many; speed depends on the processor or graphics card
Export to other platforms	Usually no; the strategy stays in the platform's language	Usually no; you translate by hand	Depends; often tied to the vendor's own execution	Usually yes; AlphaPrime exports source code to MultiCharts, TradingView, TradeStation and NinjaTrader
Price	Free or included with the platform	Free library plus your time	Subscription	Paid, one-time or subscription

Read the table by row, not by column. If you only ever refine one strategy you wrote yourself, the fourth row does not matter and a platform tester scores well. If you search for new rules across several markets, the fourth and fifth rows decide, and the price row moves to the end. Export in particular is easy to overlook: the supported platforms for AlphaPrime are listed on the platform export page, and a trader who already runs a platform should check that row first.

What do free and built-in testers leave out?

Free and built-in testers usually default to zero costs and a single in-sample run, and some let an order fill on the bar that produced the signal, so a strategy that passes there still needs costs, next-bar execution and an out-of-sample check before the result means anything. None of these is hard to add. The problem is that the default report looks finished when it is not.

Slippage is the difference between the price the backtest assumed and the price a real order would have received. Next-bar execution is the rule that an order decided on the close of a bar fills at the earliest on the next bar, because the close is only known once the bar has ended. A tester that fills at the same close uses a price the trader never saw in time; this is the most common form of look-ahead bias in backtests. Fills on the next bar, with fees and slippage set for each run, are how AlphaPrime's backtesting handles the two checks this section describes.

The third missing piece is validation. A single run over the whole history is an in-sample test: the rules were chosen while looking at that same history. Walk-forward analysis holds out the later data from each tuning step so the result comes from bars the tuning did not see. Many testers can do this only through manual date changes, which is workable for one strategy and impractical for fifty. For the cost side of the problem, the guide to trading costs in backtests shows how to compare the average trade with the round-trip cost.

When does a strategy builder make sense, and when is it overkill?

A strategy builder earns its place when you want to test thousands or millions of rule variants under the same settings, with speed depending on the processor or graphics card; for refining one hand-written strategy, a platform tester or a Python library is enough. The deciding number is how many candidates you want to evaluate, not what hardware you own.

Think of it as searching for a flat. Checking three flats you already found is a walk. Checking every flat in a city against your budget, size and location is a search engine job. Each candidate strategy is one flat, and a builder's job is to visit all of them with the same measuring tape.

1. One hand-written strategy. You know the rules and want to see the trades. A platform tester or a Python library does this well, and a builder adds nothing you need.
2. A few hundred parameter variants. The same rules with different lookbacks and stops. A platform optimizer or a Python loop handles this, and costs and next-bar fills can be set once for the whole grid.
3. Millions of rule combinations. Different indicators, entry conditions, exit types and parameter ranges, each evaluated with the same costs and execution rule. This is the job a builder exists for. AlphaPrime's strategy generation turns a set of allowed rules, parameter ranges and search constraints into testable candidates and evaluates each one with fees, slippage and next-bar execution.

Scale changes one more thing. Testing many candidates means many chances to find a lucky one, so a builder without built-in validation is worse than a tester with none, because it finds inflated results faster. Bailey and colleagues showed how the best backtest out of many trials tends to perform worse out of sample than its in-sample result suggests (Bailey, Borwein, López de Prado and Zhu, 2014). A builder that runs walk-forward checks and thresholds on every candidate addresses this directly; a builder that only ranks by net profit does not.

For the scale of the third rung, the published figure is the only product number worth quoting: AlphaPrime completed 1,268,200,000 strategy evaluations in 18 h 23 min on one RTX 3090, measured 2026-10-05 on a real research setup (TY 37-minute bars from 2015-01-01 to 2026-09-25, 114,371 bars per candidate, commission and slippage included in every evaluation). Real speed depends on the graphics card, the data length and the research settings, and the conditions are on the benchmark page. The same features run on a CPU; the graphics card only changes how long the run takes.

Which type fits which trader?

Match the tool to how you work: discretionary traders checking one idea, programmers who want full control, traders who want browser access and no install, and systematic traders searching for new rules each have a natural fit. The table names the fit and the one thing to check hardest in it.

Trader profile	Natural tool type	Main thing to check
Checks one chart idea before trading it	Platform tester	Costs and the fill setting; both often default to the optimistic value
Programmer who wants full control	Python library	Data cleaning and execution code; the library will not catch a same-bar fill you wrote
Wants browser access and no install	Cloud platform	Data ownership and export; can your strategy and results leave the service as files
Searches for new rules across markets	Strategy builder	Validation settings and export; a search without out-of-sample checks produces inflated winners

Many traders end up with two tools: a builder or library to find and validate rules, and a platform tester to confirm the exported code on the chart they trade from. That pairing works only when the first tool exports readable source code, which is why the export row sits before price in the checklist.

How do you run a trial so the comparison is fair?

Compare tools on the same market, the same bars, the same costs, the same execution rule and the same date range, run one known strategy in each, then compare the trade lists trade by trade rather than the headline profit. Any difference you find then belongs to the tool, not to the setup.

1. Same market and contract. Pick one futures contract and one roll method, and use it everywhere. A continuous series built with a different roll rule in each tool produces different bars before any strategy runs.
2. Same bars and date range. Export the bars from one source and load the same file into every tool that accepts uploads. Fix the start and end dates, and include the same sessions, day only or full session.
3. Same commission and slippage. Set the same round-trip commission and the same slippage per side in every tool. If a tool cannot set one of them, note it; that is a finding about the tool.
4. Same next-bar execution rule. Decide on the close, fill on the next bar's open, in every tool. If a tool fills on the signal bar by default, change it before the run.
5. Run one known strategy in each tool. Use a simple rule set you understand completely, for example a moving average crossover with a fixed stop, so you can check each trade by hand.
6. Compare trade lists trade by trade. Line up entry time, entry price, exit time and exit price. Headline profit can agree by accident when two mistakes cancel; trade lists cannot.

A good result from this trial is two trade lists that agree on nearly every trade, with the few differences explained by a known cause such as a gap at the session open. A bad result is lists that differ in the number of trades, because that means the tools disagree on when a signal fires, and no amount of profit comparison will tell you which one is right.

What this means if you use AlphaPrime: run the six checks on it like on any other tool, load your own bars, confirm that fees, slippage and next-bar execution are set the way you trade, and compare an exported strategy's trades with your platform tester before you rely on it.

Frequently asked questions

What is the best backtesting software for futures traders?

There is no single best backtesting software, because the four tool types do different jobs. A platform tester suits one chart idea, a Python library suits programmers who want control, a cloud platform suits traders who want nothing installed, and a strategy builder suits a search across many rule variants. Apply the same six checks, data, fills and costs, validation, candidates per run, export and price, in that order.

Is free backtesting software good enough?

For testing one strategy you wrote, often yes, provided you change three defaults: set commission and slippage above zero, fill orders on the next bar, and keep an out-of-sample period the rules never touched. Free testers rarely block these settings; they just do not start with them. The cost of a free tool is the time you spend adding what it leaves out.

Do I need a GPU to backtest trading strategies?

No. A single strategy or a few hundred parameter variants run on an ordinary processor. A graphics card matters when you evaluate millions of candidate rule sets under the same settings, where the run time depends on the card, the data length and the settings. Every AlphaPrime feature works the same in Classic and Reserve; only the time per run differs.

What is the difference between a backtesting tool and a strategy builder?

A backtesting tool evaluates rules you wrote and reports the trades. A strategy builder generates candidate rules within constraints you set, evaluates all of them under the same data, costs and execution rule, and validates the results. The builder shifts your work from writing rules to setting constraints and judging survivors.

Why should I compare trade lists instead of total profit when trialling software?

Total profit can match between two tools while the trades differ, because two errors can cancel out. A trade-by-trade comparison of entry time, entry price, exit time and exit price shows exactly where the tools disagree, usually on fill timing or session handling. That is the information you need to decide which tool to trust.

References

• Bailey, D. H., Borwein, J., López de Prado, M., & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society, 61(5), 458–471.