Exporting Polymarket data to Parquet

• Parquet is columnar and compressed, so it is far smaller than the equivalent JSON
• Types are preserved, so timestamps and floats survive a round trip
• Partitioning by date or asset lets scanners skip irrelevant files
• pandas, Polars and DuckDB all read Parquet natively

Parquet rewards tidy tables. Flatten snapshots to one row per timestamp and outcome before writing, with best bid, best ask and depth columns. Nested ladders do not map cleanly to columns and defeat columnar reads.

Store the timestamp as a real timestamp type and prices as floats. The point of Parquet over JSON is that the reader does not have to re-derive types on every load.

If you usually query one asset or one day at a time, partition the dataset on that field. A partitioned dataset lets the query engine skip whole files, which is where the speed comes from.

Do not over-partition. Thousands of tiny files carry overhead; a partition per asset per day is usually plenty for this workload.

Once data is Parquet, the Python ecosystem reads it natively: pandas for small frames, Polars for fast columnar work, DuckDB for SQL without a server. The export is the point at which fetching stops being the bottleneck.

Combine this with local caching and you have a clean split: fetch once under quota, cache the raw JSON, and export a tidy Parquet that every analysis reads.

For large exports, write a partitioned dataset rather than one file. PyArrow write_to_dataset takes a partition column and lays the files out in directories, so a query for one coin and one day reads only those files.

Partition on the columns you filter most, commonly coin and date. The directory layout is the index; getting it wrong means every query scans everything, which defeats the format.

Keep the schema stable across writes so a dataset assembled from many exports reads as one table. Cast columns to consistent types at write time, or a later read will fail on a mismatch between partitions.

import os
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

from polyorderbooks import PolyOrderbooksClient

rows = []
with PolyOrderbooksClient(api_key=os.environ["POLYORDERBOOKS_API_KEY"]) as c:
    books = c.get_market_books(
        "xrp-updown-5m-1787486400",
        start_ts="2026-10-17T07:00:00Z",
        end_ts="2026-10-17T08:00:00Z", resolution="1s",
    )
for outcome, snaps in books["data"].items():
    for s in snaps:
        bids, asks = s.get("bids") or [], s.get("asks") or []
        rows.append({
            "t": pd.to_datetime(s["t"]),
            "outcome": outcome,
            "best_bid": bids[0][0] if bids else None,
            "best_ask": asks[0][0] if asks else None,
        })

table = pa.Table.from_pandas(pd.DataFrame(rows))
pq.write_table(table, "xrp_books.parquet", compression="zstd")
print("rows written:", table.num_rows)

The Starter plan is free, includes order books, prices, and metrics at 250ms resolution, with 3 days of history, 60 requests/min, 1,000/day, 1 free AI backtest, and 3 strategy backtests. No credit card required.

Paid data windows from $19/mo extend history to 30–120 days at higher throughput. Backtest AI add-on is +$19/mo or standalone at $29/mo.

• DuckDB
• Local caching
• Data formats