The Multiple-Testing Trap: Why the Best of 100 Backtests Can Mislead
Running many backtests and selecting the best performer seems natural. But there is a structural problem: more attempts raise the best result even without skill. Much of the top result among 100 attempts may come simply from having tried 100 times.
Ask the Same Data Repeatedly and the Answer Looks Better
Multiple testing is the problem arising when many hypotheses are tested on the same data. One test has a relatively small chance of looking good accidentally. After hundreds, some will look good purely through luck. We usually retain those few and discard the rest.
A: Backtest One Strategy
A good result can be a signal worth investigating.
B: Backtest 100 Strategies, Then Pick the Winner
A good result does not distinguish skill from luck.
If the discarded 99 are unreported,
the result is indistinguishable from A.
─────────────
The same performance report means something different
depending on how many attempts produced it.
The key is that the discarded 99 do not appear in the report. Only the winner remains; its status as one of 100 attempts disappears. Readers can no longer distinguish the two procedures. The mechanism resembles survivorship bias in asset selection, now occurring during strategy selection.
Even 100 Coin-Toss Strategies Produce a Winner Near 62–38
The clearest demonstration repeats the procedure with strategies possessing no skill at all. Create 100 random-direction strategies and give each 100 trades. Every strategy's expected win rate is exactly 50%.
Wins for one strategy:
Mean: 50 wins
Standard deviation: √(100 × 0.5 × 0.5) = 5 wins
Expected location of the maximum among 100 samples:
Approximately mean + 2.5 standard deviations.
50 + (2.5 × 5) = About 62 wins.
Winner's report:
62 wins, 38 losses; 62% win rate.
Actual underlying ability: 50%.
─────────────
The additional 12 percentage points
come entirely from trying 100 times.
The 62% is not fabricated. The code need not contain a bug or look-ahead bias. The largest of 100 random observations naturally exceeds the mean; we selected only that maximum. Trouble begins when it is interpreted as a 62% win rate that will persist. See trade sample size for how many observations a win-rate estimate needs.
Combinations Multiply Faster Than Expected
You may think you have never created 100 strategies. But each parameter variation is also a separate attempt. A grid grows much faster than intuition suggests.
Fast average: 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 → 10 values
Slow average: 60, 80, 100, 120, 140, 160, 180, 200, … → 10 values
Stop: 0.5%, 1%, 1.5%, 2%, 3% → 5 values
Take-profit: 1%, 2%, 3%, 4%, 6% → 5 values
Combinations:
10 × 10 × 5 × 5 = 2,500
Test each across 20 coins:
2,500 × 20 = 50,000 attempts
─────────────
“I optimized one strategy” actually means
choosing the best of 50,000 attempts.
Changing periods, adding and removing indicators, and rerunning revised entry conditions increase the total further. Overfitting reshapes parameters around data; multiple testing operates one level above by using the data to choose the candidate itself. They often appear together, but their causes and remedies differ.
The Expected Number of Accidental Passes Is Calculable
Even a rule that adopts candidates only above a threshold produces a predictable number of worthless candidates passing by chance when repeated often enough.
1 attempt → Expected accidental passes: 0.05
20 attempts → 1
100 attempts → 5
2,500 attempts → 125
Probability of at least one pass in 2,500 attempts:
1 − (0.95 ^ 2,500) ≈ Effectively 100%
Divide the threshold by trial count, the Bonferroni correction:
0.05 ÷ 2,500 = 0.00002
This degree of strictness preserves the originally intended 5% level.
─────────────
A candidate passing alone proves nothing.
You must also know how many attempts were made.
Bonferroni is the simplest and most conservative correction, and can be stringent in practice. Its direction is clear: more attempts require a stricter acceptance threshold. Trying 50,000 times with a one-test threshold makes a passing candidate an expected outcome, not necessarily a discovery.
Losing 12 Win-Rate Points Reverses Profit and Loss
A 62% observed win rate reverting to its true 50% may sound like a modest deterioration. Round-trip fees show otherwise.
Best Backtest: 62 Wins, 38 Losses
Gains: 62 × 0.6% = +37.2%
Losses: 38 × 0.5% = −19.0%
Fees: 100 × 0.10% = −10.0%
Total: +8.2%
Underlying Ability: 50 Wins, 50 Losses
Gains: 50 × 0.6% = +30.0%
Losses: 50 × 0.5% = −25.0%
Fees: 100 × 0.10% = −10.0%
Total: −5.0%
─────────────
A 12-percentage-point win-rate difference
moves P&L by 13.2 percentage points
and changes its sign.
Fees impose a fixed cost on every trade. A small win-rate decrease can make the strategy unable to cover them, turning a profitable backtest curve into a losing live curve. See round-trip trading costs for the 0.10% assumption, and profit factor for evaluating profit against loss amounts. The backtesting guide covers other reasons live equity curves deteriorate.
Start by Counting Attempts
Multiple testing cannot be eliminated: testing multiple candidates is part of research. Manage it by counting attempts, raising the standard accordingly, and checking once on reserved data.
① Record the trial count.
Grid combinations × Assets × Reruns.
Without counting, correction is impossible.
② Split the data in advance.
70% exploration / 30% sealed holdout.
Open the holdout only once.
The second viewing turns it into exploration data.
③ Write acceptance criteria first.
Minimum trade count, minimum profit factor, maximum drawdown.
Do not revise criteria after seeing results.
④ Create a control with a permutation test.
Randomly shuffle signal–return pairings and recalculate 1,000 times.
Check the actual result's rank among those 1,000.
A result only in the top 30% is not distinguishable from chance.
⑤ Add forward validation after adoption.
─────────────
Record
The total combinations tested.
How many times the holdout was opened.
The difference between exploration and holdout performance.
Walk-forward analysis repeats the split chronologically while advancing through time. A permutation test can be implemented with a few lines and shares the random-control idea behind Monte Carlo simulation. Forward validation runs the unchanged rules in paper trading before a real account. Since performance changes with conditions, also separate results by market regime.
Knowing the normal range of consecutive losses helps distinguish failure from an expected difficult interval. See losing-streak probability for that calculation and equity-curve trading for comparing live performance against a baseline.
Key Points
② More attempts raise the maximum result without skill.
③ 100 random strategies × 100 trades can yield a winner near 62 wins and 38 losses.
④ The additional 12 win-rate points come from the attempt count.
⑤ Two moving averages, a stop, and a target can generate 2,500 combinations.
⑥ Across 20 coins, that becomes 50,000 attempts.
⑦ A 5% individual threshold over 2,500 trials yields 125 expected chance passes.
⑧ Acceptance standards must tighten as attempts increase.
⑨ Reverting from 62% to 50% wins changes +8.2% into −5.0%.
⑩ Fixed per-trade fees help reverse the sign.
⑪ Open the holdout only once.
⑫ Include the trial count in performance reports.
How many candidates the winner beat tells you more than the winning result alone. Without recorded attempt counts, even a strong backtest lacks the information needed to distinguish discovery from chance.
Notice
The 100 random strategies, 100 trades each, five-win standard deviation, approximate maximum of 62 wins, 2,500 grid combinations, 20 coins, 5% individual threshold, +0.6% average win, −0.5% average loss, 0.10% round-trip fee, and 1,000 permutations are hypothetical examples, not measurements from a particular strategy or account. The maximum near mean plus 2.5 standard deviations is approximate and varies between trials. Bonferroni is highly conservative and can reject real signals when many candidates are tested. Similar grid candidates are not fully independent, so chance-pass calculations are also approximate. Passing holdout, walk-forward, and permutation tests does not guarantee future returns; structural market changes can alter a validated strategy's performance. Leveraged trading can lose all principal. Investment decisions and their consequences are your responsibility.
NOONOO TRADING invites you to follow live trading in our free chat.
Start in the bot📈 OKX trading fee discount for new registrations
Register for the OKX Fee Discount →