add columns to polymarket models

[hildobby]

added columns to existing tables (needed for much more efficient queries) and created 3 tables:

• one for protocol tvl
• one for user balance changes
• one for currently open positions (view)

[cursor]

PR Summary

Introduces new analytics surfaces and enriches existing Polymarket models for better querying and pricing.

• Adds new incremental models: polymarket_polygon_protocol_tvl (hourly TVL/net flow from USDC.e transfers) and polymarket_polygon_users_balance_changes (inflow/outflow/internal per wallet); adds polymarket_polygon_open_positions view (latest positions with latest_price/open_interest)
• Enhances polymarket_polygon_market_details with polymarket_link_slug, parsed start/end timestamps, and computed final_price
• Extends trades: polymarket_polygon_market_trades_raw and ..._market_trades now include maker_asset_id/taker_asset_id and expose polymarket_link_slug
• Upgrades polymarket_polygon_positions to incremental, adds daily pricing join and price/usd_value, and filters late/partial days
• Refactors polymarket_polygon_market_prices_daily forward-fill to use utils_days and date bounds
• Updates _schema.yml to document new columns/models and add tests

^{Written by Cursor Bugbot for commit d775b77. Configure here.}

[cursor]

Comment @cursor review or bugbot run to trigger another review on this PR

[hildobby]

PR is ready btw, it has a a few new spells and some new columns in a few tables

I also optimised the spells a bunch since runtime was crazy on some of those. It's far from perfect but should require a lot less compute than they used to. I made sure to keep all old columns for backward compatibility.

The only missing thing is for the market details spell which needs to have uploaded base table switched from dune's (deprecated afaik) to polymarket's uploaded one but it's missing a needed column so I pinged Primo to ask if he could add it. We can merge as is and tackle that in another follow up PR

@0xBoxer

[hildobby]

@jeff-dude could you review this PR please?

[jeff-dude]

@jeff-dude could you review this PR please?

before diving deep, want to give you heads up that we have three open PRs to materialize polymarket views into incremental tables. once those are deployed and rebuilt, you can merge changes into your branch here. then we can look into the additional columns and tables. our changes will not change the data output or schema at all, simply materialize into tables for performance downstream.

[hildobby]

@jeff-dude could you review this PR please?

before diving deep, want to give you heads up that we have three open PRs to materialize polymarket views into incremental tables. once those are deployed and rebuilt, you can merge changes into your branch here. then we can look into the additional columns and tables. our changes will not change the data output or schema at all, simply materialize into tables for performance downstream.

do as you need, I switched out of dune to using another more reliable source for polymarket data since most things blow past dune's query runtime anyways because the spells are in a rough state and merging this took way too long

[jeff-dude]

do as you need, I switched out of dune to using another more reliable source for polymarket data since most things blow past dune's query runtime anyways because the spells are in a rough state and merging this took way too long

understood. this is why we materialized, to help with query performance. we are also spending time Q2 on enhancing prediction market datasets.

[hildobby]

@jeff-dude yes but materialising them won't fix this fyi. you can't even get volume from them right now and need to defer to decoded even tables, same for tvl (costs insane compute because dates are string for example) and many other super basic metrics. the queries are very suboptimal in their current state, hence my PR here. In this PR I tried to keep all currently active columns while adding new ones that I think are more useful (so it will remain with a ton of technical debt of unneeded stuff either way)

[jeff-dude]

@jeff-dude yes but materialising them won't fix this fyi. you can't even get volume from them right now and need to defer to decoded even tables, same for tvl (costs insane compute because dates are string for example) and many other super basic metrics. the queries are very suboptimal in their current state, hence my PR here. In this PR I tried to keep all currently active columns while adding new ones that I think are more useful (so it will remain with a ton of technical debt of unneeded stuff either way)

okay thanks for clarifying. i have shared internally with the team for reference.

[tomfutago]

bugbot run

[cursor]

Bugbot couldn't run

Bugbot is not enabled for your user on this team.

Ask your team administrator to increase your team's hard limit for Bugbot seats or add you to the allowlist in the Cursor dashboard.