Workflow
Building a Polymarket data pipeline in Python
A production data pipeline for Polymarket order book data handles extraction with rate limiting, transformation into analysis-ready formats, and loading to durable storage.
What this guide covers
- Extract: paginated API pulls with rate limit handling
- Transform: flatten snapshots, compute spread and depth metrics
- Load: Parquet files or database tables with partitioning
- Schedule: cron, Airflow, or simple timer-based scripts
ETL architecture overview
The pipeline has three stages: Extract pulls data from the PolyOrderbooks API, Transform converts raw JSON into analysis-ready data, and Load writes it to durable storage.
The Extract stage must handle rate limits gracefully. On a paid plan (300 rpm), you can pull about 5 markets per second.
The Transform stage flattens nested JSON into tabular data. Raw book snapshots have nested bids and asks arrays; transform extracts best bid, best ask, spread, total bid depth, and total ask depth.
The Load stage writes transformed data to Parquet files or database tables. Partition by date and coin for efficient querying.
Extract: pulling data with rate limits
Start with a market index. Pull the full /markets list once and cache it. Refresh the index daily or weekly.
For each market slug, pull books, prices, and metrics. Use the SDK built-in retry logic, or implement your own exponential backoff.
Batch your requests. Rather than pulling one market at a time, queue all your slugs and process them with a fixed-rate limiter.
Save raw JSON responses to a staging area before transforming. This gives you a fallback if the transform step fails.
Transform: flattening and computing metrics
The most important transform is flattening book snapshots. Extract the best bid price, best ask price, spread, total bid depth, and total ask depth from each snapshot.
Compute derived metrics: spread as a percentage of the midpoint, depth imbalance as (bid_depth - ask_depth) / (bid_depth + ask_depth), and VWAP as the size-weighted average price.
Handle edge cases: empty books (no bids or asks), crossed markets (best bid > best ask), and missing timestamps. Log these anomalies rather than filtering them out.
Output the transformed data as a DataFrame ready for loading. If using Parquet, ensure correct dtypes: timestamps as int64, prices as float64, sizes as float64.
Load and schedule
For Parquet storage, partition files by date and coin: data/2026-09-10/btc/books.parquet. This makes date-range queries fast.
For database storage, PostgreSQL with a JSONB column works well for raw data, or a columnar store like ClickHouse for analytical queries.
Schedule the pipeline with cron for simplicity: 0 */4 * * * runs every 4 hours. For more complex workflows, use Apache Airflow or Prefect.
Monitor pipeline health by logging the number of markets processed, the number of API requests made, and the total data volume written.
Code examples
import requests, time
class RateLimitedExtractor:
def __init__(self, api_key, rpm=300):
self.base = "https://api.polyorderbooks.com/v1"
self.headers = {"X-API-Key": api_key}
self.interval = 60.0 / rpm
self.last_call = 0
def _throttle(self):
elapsed = time.time() - self.last_call
if elapsed < self.interval:
time.sleep(self.interval - elapsed)
self.last_call = time.time()
def get(self, path, params=None):
self._throttle()
r = requests.get(f"{self.base}{path}", headers=self.headers, params=params)
if r.status_code == 429:
wait = int(r.headers.get("Retry-After", 5))
time.sleep(wait)
return self.get(path, params)
r.raise_for_status()
return r.json()Free tier
The Starter plan is free, includes order books, prices, and metrics at 1-second resolution, with 3 days of history, 60 requests/min, 1,000/day, 1 free AI backtest, and 3 strategy backtests. No credit card required.
Paid data windows from $19/mo extend history to 30–120 days at higher throughput. Backtest AI add-on is +$19/mo or standalone at $29/mo.
FAQ
How many API requests does a daily pipeline need?
It depends on how many markets you track. Each market requires one request per endpoint (books, prices, metrics). Tracking 500 active markets means ~1,500 requests per run.
Should I use the SDK or raw requests?
The SDK handles retries and pagination automatically, which saves code. Raw requests give you more control over error handling and logging. For production pipelines, raw requests with explicit retry logic are often preferred.
How do I handle new markets appearing?
Refresh your market index at the start of each pipeline run. New markets appear in the /markets list before they start trading.
What if the API is down during a scheduled run?
Log the failure and retry on the next scheduled run. Do not retry immediately because if the API is down for maintenance, immediate retries waste rate limit budget.