Polymarket Data in DuckDB

• CREATE TABLE books AS SELECT * FROM read_parquet("polymarket_books_*.parquet"); loads a directory of Parquet in one line.
• CSV path: read_csv_auto("slice.csv") type-infers timestamps, prices, and depths.
• REST-to-file: a tiny worker script fetches a window and writes Parquet; DuckDB then owns analysis.
• For enterprise, S3 delivery drops Parquet/CSV/JSON that COPY FROM s3:// reads directly.

• Per-contract summaries: SELECT market, count(*) FILTER (WHERE ask_depth = 0) / count(*) AS pct_onesided FROM books GROUP BY market.
• Spread stats: quantile_cont(sell_price - buy_price, 0.5) over windows by market type.
• Settlement cross-check: join final-minute flags to resolution outcomes (the archive carries both).
• Window aggregates: date_bin(1m, ts) chunks 250ms rows into minute bars for coarser studies.

Columnar scans make multi-hundred-million-row aggregates immediate on a laptop; Books, prices, and metrics come back over the same endpoints at the same 250ms resolution, so a dashboard needs one schema, not three.

There is no ETL server to babysit — point DuckDB at files and query. When the dataset outgrows the laptop, the same SQL runs on ClickHouse or BigQuery.

Point DuckDB at the books directory and compute per-contract one-sidedness over the final hour: SELECT market, count(*) FILTER (WHERE ask_depth = 0)/count(*) AS p FROM books GROUP BY 1.

Step back to 1m bars with date_bin('1 minute', ts) when rendering dashboards; 250ms stays in the file, the bars stay in RAM.

The full 8,349-slice September BTC run at 250ms is tens of millions of rows; a settlement group-by over the final-hour window should answer in a few seconds, not minutes.

DuckDB is also the engine for the guide's published studies — the same SQL reproduces the archive flags from raw 250ms files.

DuckDB is the SQL engine that matches 250ms granularity without ceremony: columnar scans on Parquet mean the September BTC settlement run, tens of millions of rows, answers in seconds on a laptop.

The honest boundary is sharing, not compute — a single analyst gets laptop scale for free; a team that wants governed access upgrades the store, not the queries.

The 8,349-contract September BTC window is the standard stress dataset in this guide — it stays queryable at 250ms under DuckDB without special tricks.

First query is the settlement scan: point read_parquet at a month of BTC frames and run the final-hour one-sided count per contract; on a laptop, DuckDB returns it in seconds, which is the threshold that tells you this stack works.

Complement it with a window sanity query: select min(ts), max(ts), count(*) from the same files; the range should equal the nominal window exactly, and count should equal seconds x 4 for 250ms.

DuckDB is the proof that hundreds of millions of 250ms frames do not need an infrastructure project to query. The settlement scans, one-sided rates, and point-in-time lookups that this guide uses as examples all run in seconds from Parquet files on a laptop.

The tuple that keeps DuckDB honest is three checks: min/max ts equals the nominal window, row count equals seconds times four at 250ms, and a cross-check against the CSV export for one day. Pass those three and the SQL you write today is the SQL you trust next year.

When the team grows, the files move to a shared bucket and the same queries run under Athena or BigQuery with minimal edit — the schema, not the engine, is the contract you maintain.

• API data guide
• Data download
• Order book data
• MCP server