Analytics2024
HORIZONDASHBOARD
PnL and Horizon Analysis at sub-100ms on multi-TB trading datasets.
<100ms
Continuous aggregate reads
3600s
Widest horizon window
6+
Rollup refresh policies
SCROLL
PnL and Horizon Analysis at sub-100ms on multi-TB trading datasets.
A stealth trading desk needed GMX perpetuals mark-out analysis: horizon curves from 1s to 3600s, trader leaderboards, and per-position lifecycles on data landing from a Substreams SQL sink into Timescale.
Timescale continuous aggregates and tuned chunk intervals feed Polars/asyncpg batch jobs; Streamlit with Plotly/Altair renders trader-facing views; Docker Compose wires Postgres, roles, and the app for reproducible analyst environments.
Hypertables in the terabyte class — raw scans per chart click were unacceptable; compression policies and continuous aggregates had to precompute the heavy lifting while keeping refresh lag bounded.
Reconcile Substreams SQL sink batches against chain head and Substreams cursors.
Chunk intervals and partitioning keys tuned to query patterns (trader, market, time).
Rollups for horizons, funding, and leaderboard metrics with refresh policies.
Offline recomputes and backfills without blocking Streamlit threads.
Plotly/Altair charts with cached dataframes for snappy interactions.
Typical dashboard reads hit pre-aggregated continuous aggregate tables instead of scanning raw swap rows — the difference between an interactive desk review and a coffee-break query.
Typical dashboard reads hit pre-aggregated continuous aggregate tables instead of scanning raw swap rows — the difference between an interactive desk review and a coffee-break query.
PnL windows from 1s through 3600s plus trader rankings aligned to the same Substreams-derived facts powering risk — so “who moved the book?” questions didn’t require a separate Jupyter notebook.
Moved CPU from request time to refresh time — mandatory at TB scale.
Docker Compose gave analysts the same Postgres extensions locally as prod, reducing “works on my laptop” chart bugs.
Vectorized transforms beat row-wise Python when recomputing horizon curves after code fixes.
Streamlit stayed thin — all heavy lifting in SQL or Polars kept the UI responsive.
zkEVM Deployment
We take on a small number of projects at a time. If the problem is hard, we're interested.