Polymarket Data in Parquet

• Compression: price/depth columns are low-cardinality and highly duplicate; Parquet typically 10–20x smaller than CSV.
• Columnar projection: a spread query reads two columns, not whole rows.
• Direct reads: DuckDB, Polars, arrow, ClickHouse, and BigQuery all read Parquet natively — no ETL.
• Type safety: timestamps and numerics typed at write time instead of re-inferred at read time.

• One file (or daily partition) per market/series: books.parquet containing ts, seq, bid, ask, mid, bidDepth, askDepth, vol.
• Add a _metadata footer for row-group stats (min/max per file) so readers prune whole files.
• Snowflake-like partitioning: /market_type=updown/date=2026-09-01/books.parquet
• Keep resolution in the filename (books_250ms.parquet) — re-deriving it later is guessing.

• Self-serve downloads expose CSV and JSON per market; convert via the SDK's pandas/to_parquet step for scripts.
• Bulk Parquet/CSV/JSON is available on enterprise S3 delivery for full-archive work.
• Parquet then feeds DuckDB/ClickHouse for the heavy SQL — the lifecycle most published studies use.

Slice a market into a single books.parquet with ts+seq+quotes+depths+vol; row-group stats make DuckDB skip whole groups for a windowed query.

Name files books_250ms_PART.parquet and keep a COLUMNS.md; six months later the file is self-describing, which is the point of the format.

A 250ms month for one market is ~10.4M rows; Parquet rarely needs more than a few hundred MB for that, versus GBs of CSV — a size sanity check on your files is a proxy for schema health.

The archive ships bulk Parquet on enterprise delivery, so lake teams get typed files without writing a transform.

Parquet is the material the rest of the pipeline assumes: typed timestamps, column pruning, and 10x-plus compression over CSV turn an unwieldy 250ms record into a structure any SQL engine reads without ETL.

The honest discipline is naming and row-group hygiene: resolution and partitioning encoded in the path, stats in the footer — because a Parquet file that lies about what it holds is worse than a CSV you can eyeball.

A full market-month at 250ms (~10.4M rows) as Parquet is typically a few hundred MB versus GBs of CSV — the size delta is itself a schema sanity check.

First file: one market, one day at 250ms, written as books_250ms.parquet, and immediately queried by DuckDB with a window filter to demonstrate column pruning reduces the scan.

Check size and stats next: PARQUET file_size and the min/max per row group should match the nominal ts range; that double check is the receipt every later pipeline trusts.

Parquet is the layer that makes every other choice work: typed timestamps, columnar scans, and compression that keeps a full 250ms market-month at a few hundred megabytes on a laptop.

The habits that protect the lake are cheap: store resolution in the name, sort by (market, ts), write row groups with healthy min/max, and keep a checksummed manifest. Each is one line of code and each prevents a silent-corruption debugging session.

From DuckDB on a laptop to Athena across a bucket, every reader speaks the same format — so the file layout you choose today is the one every future tool inherits.

• API data guide
• Data download
• Order book data
• MCP server