Workflow
Polymarket data formats: CSV, Parquet, and JSON
Polymarket order book data comes from the API as JSON, but storing it locally requires choosing a format. This guide compares CSV, Parquet, and JSON for file size, query speed, and tool compatibility.
What this guide covers
- JSON: universal, human-readable, but verbose and slow to parse at scale
- CSV: simple, universal, but no type safety or compression
- Parquet: columnar, compressed, fast queries, but requires specialized tools
- PolyOrderbooks datasets ship in Parquet for size and query speed
JSON — the API default
The PolyOrderbooks API returns JSON. Every endpoint — markets, books, prices, metrics — returns a JSON array. This is the natural format for ingestion.
JSON is human-readable, universally supported, and easy to debug. You can open a JSON file in any text editor and understand the data.
The downside is verbosity. A single order book snapshot with 20 price levels on each side can be 500+ bytes in JSON. At 1-second resolution, a 5-minute market generates ~150KB of JSON for the books endpoint alone.
JSON parsing is also slower than columnar formats at scale. Reading 10,000 snapshots from JSON takes noticeably longer than reading the same data from Parquet.
CSV — simple and universal
CSV is the simplest tabular format. Every spreadsheet application, database, and programming language can read CSV files. It is a safe default for sharing data with others.
For order book data, CSV works well for flat structures like price series or metrics. The challenge comes with nested data: bids and asks arrays do not flatten cleanly into CSV columns.
CSV files are larger than Parquet because there is no compression and no type encoding. A numeric column in CSV stores each value as a text string.
For datasets under 100MB, the size difference is negligible. For datasets in the gigabyte range, Parquet makes a significant difference in both storage and query speed.
Parquet: the analyst choice
Parquet is a columnar storage format designed for analytical queries. It compresses data aggressively, typically 3 to 10x smaller than JSON or CSV.
Columnar layout means queries that touch only a few columns read only those columns from disk. A query for spread values does not load the bids and asks arrays.
Parquet preserves types natively. Numbers are stored as numbers, not text. This eliminates type confusion and speeds up computations.
The downside is tooling. Parquet requires libraries like pandas, DuckDB, or Polars to read. You cannot open it in a text editor.
| Feature | JSON | CSV | Parquet |
|---|---|---|---|
| File size (relative) | 1x | 0.7x | 0.1-0.3x |
| Parse speed | Slow | Medium | Fast |
| Type safety | Partial | None | Full |
| Human-readable | Yes | Yes | No |
| Columnar queries | No | No | Yes |
| Tool support | Universal | Universal | pandas, DuckDB, Polars |
| Best for | API ingestion, debugging | Sharing, small datasets | Analysis, backtesting, storage |
Converting between formats
Converting JSON to Parquet is a one-liner in pandas: pd.read_json("data.json").to_parquet("data.parquet").
Converting JSON to CSV is similarly simple: pd.read_json("data.json").to_csv("data.csv", index=False). Flatten nested columns first.
Parquet to CSV: pd.read_parquet("data.parquet").to_csv("data.csv", index=False). Useful for sharing with stakeholders who do not have Parquet-compatible tools.
For order book data with nested bids/asks, flatten before converting to CSV. Extract best_bid, best_ask, spread, and total depth as separate columns.
Code examples
import pandas as pd, os
for fmt in ["json", "csv", "parquet"]:
size = os.path.getsize(f"books.{fmt}")
print(f"{fmt}: {size / 1024:.1f} KB")Free tier
The Starter plan is free, includes order books, prices, and metrics at 1-second resolution, with 3 days of history, 60 requests/min, 1,000/day, 1 free AI backtest, and 3 strategy backtests. No credit card required.
Paid data windows from $19/mo extend history to 30–120 days at higher throughput. Backtest AI add-on is +$19/mo or standalone at $29/mo.
FAQ
Which format should I use?
Parquet for analysis and backtesting (fastest queries, smallest files). CSV for sharing with non-technical stakeholders. JSON for API ingestion and debugging.
How much smaller is Parquet than JSON?
Typically 3 to 5x smaller for order book data. The exact ratio depends on the number of price levels and the compression ratio.
Can I query Parquet files without loading them into memory?
Yes. DuckDB and Polars can query Parquet files directly without loading them into memory. This is useful for large datasets.
Do PolyOrderbooks datasets come in Parquet?
Yes. Sample datasets are provided in Parquet format for optimal file size and query performance. You can convert to CSV or JSON as needed.