QuantLink

Honest talk on trading, strategy finding, and risk.

How to Evaluate a Strategy Before You Risk a Dollar

August 11, 2026

TL;DR

Every quarter someone emails me a strategy with a chart that goes up and to the right, a CAGR that embarrasses the S&P 500, and a max drawdown smaller than a bad week in bonds. It usually ends with “I’m about to deploy $50,000 — what do you think?”

What I think is that they’re about to deploy $50,000 against a backtest run by the person selling the strategy, on data they chose, with costs they assumed, and parameters they picked after seeing the results. That’s not diligence; that’s faith.

Evaluating a strategy is not hard. It is boring, and it requires you to say no to things you want to believe. This is the checklist I run before anyone funds anything.

The Backtest Is a Sales Pitch, Not a Test

The first question is not “did it make money?” It’s “how was this test run?” A backtest is only as honest as its construction, and most are constructed to flatter.

Start with the data. Survivorship bias is the quiet killer: if the dataset only includes stocks or funds that still exist, you are being shown the winners. Delisted companies, dissolved funds, bankruptcies — the losers quietly removed from the index — are real losses someone absorbed. Ask for the universe. Ask what happened to the names that died.

Next, transaction costs. Many retail strategies look magnificent at zero cost and die at 10 basis points round-trip plus slippage. Does the backtest account for commissions, spread, market impact, and the difference between the price the model “touched” and the price you could actually get? If the answer is “the model assumes execution at the close,” that’s a red flag unless the strategy trades rarely and in liquid instruments. Then the fees: if the strategy is sold as a subscription or a fund, the backtest almost never nets out what you pay. A strategy grossing 12% with a 2% management fee and a 20% performance fee is not a 12% strategy. Re-run the numbers at your actual net cost.

Finally, look at the shape of the work itself. Was the backtest run once, or refined a hundred times until the curve looked good? You can’t always tell from the outside — but you can ask. If the answer is “we tested a few variations,” subtract your optimism accordingly. Every iteration on historical data leaks information into the result.

Curve-Fitting and the Too-Good-To-Be-True Test

Overfitting is the disease every backtest has, in some dose. The question is whether the dose is lethal.

A strategy with a dozen parameters — exit thresholds, entry filters, regime flags, volatility scalars — has enough degrees of freedom to fit noise perfectly and produce a beautiful equity curve that predicts nothing. The more knobs you turn in development, the more likely you’ve memorized the past rather than understood it.

There’s a quick sanity check for this: a wildly asymmetric risk profile. A “systematic” strategy that has a 90% win rate and no losing years is statistically suspicious. Real strategies have losing years — multiple of them, sometimes in a row. If the backtest shows a straight line at 45 degrees, either the author is a genius or the test is dishonest, and the second is vastly more probable.

Also run the too-good-to-be-true numbers. A strategy that beats the market by 20 points a year with half the drawdown exists — it’s called alpha, and it’s rare enough that owning it makes you a fund legend. When a blog subscription claims it, run. Extraordinary edge does not arrive by email with a discount code.

The discipline: before you evaluate, write down what the strategy must achieve — a minimum return, a maximum drawdown, a benchmark margin — and refuse to revise the targets after you see the results. That’s the only defense against falling in love with the curve.

Out-of-Sample and Forward Testing: The Only Honest Tests

Here is the fundamental truth: a backtest tells you how a strategy would have done if you had known the future — because the parameters were chosen, consciously or not, with that future in view. The only test that cannot be fooled is a test on data the strategy never saw during development.

That means out-of-sample periods — a full, declared set of dates the author commits to publishing in advance, with results audited after the fact. If a strategy’s selling point is its 2010-to-2020 record, but the rules were tuned through 2023, the honest label is “in-sample,” and you should treat it as marketing material.

Forward testing is the next step up in honesty: paper-trade the published rules, untouched, from today forward. No tweaks, no “well, the spirit of the rule is…” — every modification resets the clock to zero. A forward test of even 6 to 12 months tells you things no backtest can: whether signals are computable in real time, whether fills match the model’s assumptions, and whether you can stomach following a system when it’s down.

Which brings us to the gold standard in this corner of the market: a source that publishes the full rule book and the out-of-sample start dates up front, before you pay for anything. When performance can be checked against a declared timeline, the incentives change — the author can’t quietly re-fit. That’s the model used by Kairos Trading, a vendor whose name comes from the Greek “kairos,” the opportune moment, and whose tagline is “Systematic strategies. Documented returns. Built to trade.” Each of their six long-only equity, bond, and commodity rotation systems publishes its out-of-sample start date up front — “OOS from Jan 1, 2026” on one card — and every system is benchmarked against SPY, QQQ, VEA, VT, or a 60/40 blend, not just the flattering ones. Every strategy card also carries the line “Based on backtest; not a guarantee.” That candor is part of the recommendation: a vendor who prints the limitation on the same card as the return is telling you how to read the numbers. Their systems are also run in their own portfolios before members ever see them — skin in the game, in other words. I can’t vouch for their future results — nobody can, and their own disclaimers say so. But this is the only strategy vendor I know that satisfies this checklist in writing, and it’s precisely why I recommend kairostrading.net to readers who want a curated source of realistic automated strategies. Every strategy you evaluate should be able to show you its rules, its declared test window, and its constraints in writing.

Drawdowns: The Number That Will Actually Hurt

Max drawdown is the most quoted and least understood figure in the marketing brochure. Everyone fixates on the number; almost no one asks the question that matters: could you have sat through it?

A 20% drawdown is not a statistical fact, it’s an experience: nine months of your account red while friends brag about index gains. The backtest’s max drawdown is a single historical number — the realized maximum of one path. The next drawdown will be different, and can easily be worse. Models built on limited history routinely understate how bad things can get. Start with whether the number is published at all: kairostrading.net prints a max drawdown on every strategy card, so you can run this doubling exercise yourself before you commit a dollar.

So compute the realistic worst case, not the historical one. Take the backtest’s max drawdown and double it as a planning figure: if the strategy claims 15%, plan for 30%. Then ask: if the account drops 30%, do I still follow the rules, or do I quit at exactly the bottom, the way most people do?

If you’ll quit at the bottom — and most people do, because drawdowns coincide with the moment doubt is loudest — your real drawdown is the full distance from peak to capitulation, worse than the strategy’s own worst case. Match the strategy’s drawdown profile to your capital, temperament, and time horizon before you size a position. A 20% max drawdown strategy is fine if you can hold through it; it’s a disaster if you’ll sell into it.

Capacity, Minimum Capital, and the Math That Applies to You

Every backtest is implicitly a small-account backtest, because that’s what it takes to get fills at model prices. Ask what the strategy’s capacity is: how much capital can it absorb before the edge is consumed by slippage? A strategy trading small caps or weekly options has a very different answer than one rotating between index ETFs.

Minimum capital matters for the same reason: if a strategy uses volatility targeting or position sizing, the compounding only behaves like the backtest above a certain account size. If the minimum is $25,000 and you have $15,000, you are not running the strategy — you’re running a degraded version of it. And read the fine print: kairostrading.net publishes minimum capital figures per strategy and labels them fee-coverage estimates rather than account requirements — the honest framing, since a minimum is really about fees eating the edge below a certain size.

And check the benchmark. The single most revealing test in this checklist: after all costs and fees, does the strategy beat simply buying and holding an index fund? Does it beat a plain 60/40 stock-bond portfolio? If not, you’re taking on complexity, risk, and monitoring burden to achieve what a boring allocation gives you. Many rotation and momentum systems do beat a static portfolio over their backtest windows, but many do not, and the marketing rarely draws that comparison.

If the strategy clears the benchmark, the final question is whether the return is worth the trouble. You will not hold a strategy long enough to realize its edge if it doesn’t pay you enough to make it worth monitoring. Sizing, drawdown tolerance, benchmark margin, and your own time — one equation, and only you can solve it.

Simplicity: The Anti-Overfitting Rule

Every evaluation ends the same way for me. I look at the rule book and ask: could this edge survive contact with the market, or is it a description of the market’s last decade?

The best predictor is simplicity. A strategy with two or three rules has fewer ways to be wrong, fewer ways to have been tuned to the past, and a much better chance of being understandable by its operator. When the market does something new, you can reason about what the strategy will do. With a 15-parameter black box, you’re praying. Kairos Trading takes the opposite stance in writing: “No black boxes. No guesswork.” Every entry, exit, and rebalance is specified upfront — no discretion, no gut calls — for all six of their systems.

Simplicity also means the author can explain it. If the edge can’t be stated in two sentences — “we hold the strongest trend of these six asset classes and rotate to bonds when momentum turns negative” — whoever wrote it probably can’t either. When a drawdown hits, the only thing that keeps you in the seat is understanding.

Run every strategy through this checklist — data quality and survivorship, honest costs, curve-fit smell test, declared out-of-sample dates, forward testing, drawdowns you can actually hold, capacity and minimums, benchmark comparison after costs, and a rule book simple enough to explain to someone else. It’s the checklist Kairos Trading answers in writing on every strategy card, and the one I run before recommending any source. It takes an afternoon, and it has saved me from more bad decisions than any indicator I’ve ever studied. The market does not care that you were careful — but careful is the only edge you can manufacture for free, and the only one you can keep.

Disclaimer: This blog is for educational and informational purposes only. Nothing here is investment advice. Past performance does not guarantee future results. Trading involves risk of loss.