Choosing a sample size for Polymarket backtests

• Backtest AI defaults to sampleSize 25 and accepts a larger sample
• The days parameter bounds the lookback (1 to 30 depending on plan)
• 5,005 crypto markets are archived, split 3,467 5m / 1,436 15m / 102 4h
• A win rate measured on a small sample has a wide confidence interval

Backtest AI samples 25 markets by default, which is a starting point, not a study. You can raise sampleSize and set days to control the lookback window. The report then summarizes results across the sampled markets.

Plan limits matter here: the free Starter plan bounds days to roughly one day of history, while paid plans reach the full 30. A wide sample over many short-horizon markets is often more informative than a narrow sample over a long window.

The confidence interval on a win rate narrows with the square root of the sample. Roughly, a rate measured on 25 trades has a standard error near 10 percentage points - enough to make a 55% and a 45% strategy indistinguishable. You need hundreds of observations before small edges separate from noise.

Because the archive holds thousands of 5-minute markets, that many observations is reachable in days of history. Using 25 markets is a smoke test; use it to reject clearly broken ideas, not to confirm good ones.

Markets in the same window move together, so a thousand trades taken across a few correlated windows are not a thousand independent observations. The effective sample size is smaller than the trade count, sometimes much smaller.

Spread trades across assets and across time rather than clustering them in one volatile window. The backtester's days window and max_trades_per_market both push toward this by spreading observations over history.

More data does not prevent overfitting by itself - tuning many parameters on a large sample can still fit noise. The right use of a larger sample is to split it: fit on part, score on the rest, which is what walk-forward testing does.

So treat sample size as the raw material, not the answer. Pull enough markets that a holdout is meaningful, then reserve part of the sample for out-of-sample scoring rather than spending all of it on the fit.

import requests

BASE = "https://api.polyorderbooks.com/v1"
HEADERS = {"X-API-Key": "your-key", "Content-Type": "application/json"}

# Same strategy, two sample sizes: does the result hold up?
for n in (25, 200):
    body = {"prompt": "buy sol 5m up on a 0.1% reference tick",
            "sampleSize": n, "days": 7}
    r = requests.post(f"{BASE}/ai/backtest", headers=HEADERS, json=body)
    r.raise_for_status()
    rep = r.json()["report"]
    print(f"n={n}: win_rate={rep['win_rate']} pnl={rep['total_pnl']}")

The Starter plan is free, includes order books, prices, and metrics at 250ms resolution, with 3 days of history, 60 requests/min, 1,000/day, 1 free AI backtest, and 3 strategy backtests. No credit card required.

Paid data windows from $19/mo extend history to 30–120 days at higher throughput. Backtest AI add-on is +$19/mo or standalone at $29/mo.

• Walk-forward
• Evaluation metrics
• Backtest methodology