> For the complete documentation index, see [llms.txt](https://docs.trover.tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.trover.tech/engineering/verify-2026-09-02-nft-chain-head.md).

# Verify — NFT terminal: blank tiles, chain lag, on-chain cosmetics, Advanced Charts (2026-09-02)

## No migration in THIS round — but one is outstanding and must run first

`packages/db/migrations/20260831120000_nft_token_metadata` shipped in `8bd8b4f` and was **never applied**. The api image carrying it is deployed and answering, so `/collections/:slug/sales` reaches a `nft_token_metadata` table that does not exist and returns `500 P2021`. That single fact is why **avg mint, sales 10m/1h/1d, 1h cancels, 1h sweep, metadata fetched and rank variety** all read `—` while floor, volume, supply, owners, listings and top-10 render normally: those eleven tiles come from that one endpoint.

Confirmed on 2026-09-02T06:47Z:

```
$ curl -s '.../collections/proofmachines/sales?chain=robinhood&limit=5'
{"statusCode":500,"code":"P2021", … "The table `public.nft_token_metadata` does not exist …"}
```

## Deploy

Services changed: **api + worker**. Web ships via Vercel on push.

```bash
cd ~/trover
git pull --ff-only origin main

# 1. Build FIRST. The migrate service runs from its image, not from the working
#    tree, so `run --rm migrate` on a stale image reports "No pending
#    migrations to apply" while the newest migration is simply absent from it.
docker compose -f docker-compose.yml -f docker-compose.production.yml build migrate api worker

# 2. Now the migration really applies. This alone fills the eleven blank tiles.
docker compose -f docker-compose.yml -f docker-compose.production.yml run --rm migrate

# 3. Abandon the scan backlog and resume at the chain tip. Both scanners already
#    treat a missing cursor as "start at latest", so deleting the rows uses the
#    head-only path rather than a new reset branch.
#    Note the table is `stabledex_indexer_cursors` (no underscore after "stable").
docker compose -f docker-compose.yml -f docker-compose.production.yml exec -T postgres \
  sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM stabledex_indexer_cursors WHERE source LIKE '"'"'nft-onchain-%'"'"';"'

# 4. Roll the services.
docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --force-recreate api worker
```

Order matters twice. Build before migrate, or the migration silently does not apply. Migrate before the new worker starts, or its first metadata writes fail the same way the API did.

Pushes from the laptop still need `git push origin HEAD:main` until the stale local `main` branch is retired.

## What changed

### The scanners can now hold the chain head

Measured before: \~1,000 blocks per \~82s against 1,000 blocks produced per 100s — a **1.2× catch-up rate**, i.e. \~20% headroom over the chain itself. That is why 30h of backlog barely moved in a day, and why any burst put the feed permanently behind.

The cause was that `batch: { wait: 16 }` was configured on the transport and never engaged: every lookup was awaited one at a time, so nothing was ever concurrent for it to batch. Per window, the scanners now resolve each *kind* of lookup once instead of once per log —

* block timestamps for the window's blocks, in one concurrent burst (`prefetchBlockTimes`);
* mint unit prices for the window's transactions, likewise;
* one market row per distinct contract, with an in-flight promise map in `registerOnchainCollection` so concurrent callers for the same unseen contract share one `name()` read and one upsert;
* writes grouped by collection, four collections at a time, **events within a collection still sequential** — `recordVerifiedOnchainEvent` upserts that collection's candle rows and parallel writers would race them.

`maxMintsPerTick` / `maxSalesPerTick` are replaced by a 10s wall-clock budget: the count stopped a tick that still had thousands of blocks of budget left.

**New env `NFT_ONCHAIN_MAX_LAG_BLOCKS`** (default 216,000 ≈ 6h at \~10 blocks/s). A cursor further behind than this is logged at **error** with the exact block span and reset to the tip. This encodes the choice made this round — a current feed with an admitted hole beats an hours-stale one — and makes it impossible for a backlog to build silently again.

### Lag is now visible

Both scanners write `trover:nft-onchain-lag:{mints,sales}:<chain>` to Redis each tick (head block, cursor block, lag in blocks, lag in **seconds** read from the two blocks' actual timestamps), TTL 5 minutes. No migration.

`realtimeHealth()` reads it and reports `onchainLagBlocks` / `onchainLagSeconds` on `/collections/:slug/health`. It also now counts toward the `realtime`/`degraded`/`delayed` verdict: a scanner at the tip is on its own enough to call the feed live, and one that has fallen behind demotes it however healthy the sockets look. Previously `chainHeadLagBlocks` was derived only from `robinhood_wss:` source rows — of which there are none — which is exactly how a 30-hour backlog reported as `null`.

A lane absent from Redis (not running) reads as `null`, not as zero lag.

### Cosmetics now come off the chain

Nothing in the codebase ever called `tokenURI()`. Images, traits and therefore rarity ranks came only from OpenSea, whose stream reports `connecting` and has never connected — so chain-discovered collections showed blank `/activity` media, empty scatter hover cards, and would have read \~0% metadata coverage even after the migration landed.

New `apps/worker/src/nft-token-metadata.ts`:

* `enqueueTokenMetadata` — both scanners queue the tokens a window touched, batched per collection (not per token), at a priority below every user-facing job.
* `resolveNftTokenMetadata` — reads `tokenURI` for the batch, resolves the document (`data:application/json` decoded inline with no network at all; `ipfs://` through **new env `NFT_IPFS_GATEWAYS`**, default `ipfs.io` then `cloudflare-ipfs.com`; `ar://` through arweave.net; `http(s)` direct), and merges image/animation/traits onto the same `NftTokenMetadata` row OpenSea writes to, so neither source blanks the other. Tokens that already carry traits are skipped. A dead gateway gets a 60s cooloff rather than one timeout per token, and a job hands its queue slot back after 30s.
* `sweepNftTokenMetadata` — once a collection has an audience (`priority >= 2`, set by the `activateTerminal` path every terminal endpoint already calls), each sync pass queues the next \~600 uncovered token ids. Background fill, not a burst.

**Rarity without OpenSea.** `refreshTraitCounts` fell back to nothing when the provider returned nothing, and `refreshTokenRanks` returns early on an empty `NftTraitCount` — so ranks never computed while the stream was down, no matter how much metadata we held. It now derives trait frequencies from our own `NftTokenMetadata` rows when the provider has none. Frequencies are over the tokens we have characterised, which is precisely what `metadataCoveragePct` and `rankVarietyPct` report beside them.

### The stat strip degrades tile by tile

`/collections/:slug/sales` ran its ten queries in one `Promise.all`, so one rejection took out all eleven tiles — which is exactly what the missing table did. They now settle independently; a failed leg reports `null` (rendered `—`) and is logged. This is a defence against the next schema drift, **not** a substitute for running the migration.

Also: `rankVariety` was `findMany({ distinct: ["rank"] })` — one row loaded per distinct rank, on an endpoint polled every 20s — now a `COUNT(DISTINCT rank)`. And `emptySaleStats()` returned `0` for the sales/cancel/sweep counts of an untracked collection; it returns `null`, because we have not measured them.

### Advanced Charts on the candle view

`/candles` already accepted `from`, `to`, `cursor`, `limit` and returned `nextCursor`, so **no API change was needed**.

* `apps/web/src/lib/advanced-chart/nft-datafeed.ts` — resolutions `1/15/30/60` → `1m/15m/30m/1h`, matching the route's enum exactly; pages backwards on `cursor`; realtime off the existing `/collections/:slug/stream` socket with the 15s poll only while disconnected.
* Two symbol ids from one datafeed — `<SLUG>` for sale candles, `<SLUG>.FLOOR` for the floor series — the same trick `.MCAP` uses on the token side.
* `apps/web/src/components/nft/NftAdvancedChart.tsx` — floor overlay as an `Overlay` study on the `.FLOOR` symbol, entry price as a locked `horizontal_line`, order price as a draggable one reporting its new price back through `drawing_event`, last sale as a datafeed bar mark.
* Shared with the token chart rather than copied: `price-scale.ts` (`priceScaleFor`) and `theme.ts` (overrides, studies overrides, disabled features, loading screen).

**The dots view is untouched.** `NftSaleScatter` keeps the canvas overlay — the library cannot render 2,000 image-hover points at arbitrary prices, and the hover card is the whole point of that view.

**Fallbacks.** A collection with no candles, or a library that fails to load, both render the existing `NftFloorChart`. The second is what keeps a deploy that lost the vendored assets from losing the chart with it.

## Checks after deploy

1. `curl -s '.../collections/proofmachines/sales?chain=robinhood&limit=5' | head -c 200` → JSON, not `P2021`. The eleven blank tiles fill in.
2. `curl -s '.../collections/<slug>/health?chain=robinhood'` → `onchainLagSeconds` present and under \~30. **Re-check after 10 minutes: it must not be climbing.** That is the one number that says whether the throughput fix took.
3. Worker logs should show the abandoned-gap line **once** per scanner on first start after the cursor delete, then nothing: `"On-chain NFT scan cursor is beyond the maximum lag"`. (If step 2 of the deploy ran, they should not appear at all.)
4. Find a contract that minted in the last two minutes on `rh-scan.com`. It should be in `/v1/nft-market/discover` without anyone having opened it.
5. `curl -s '.../collections/<slug>/activity?chain=robinhood&limit=5'` — a row with `"source":"onchain_mint_scan"` should now carry a non-null `imageUrl` and a non-empty `traits`. Give it a few minutes: the metadata job runs behind the scan.
6. `metadata fetched` and `rank variety` climb off 0% over an hour on an actively-trading collection. They will not reach 100% and are not meant to.
7. Candles mode on a collection terminal: TradingView toolbar renders, resolutions 1m/15m/30m/1h each load, the `floor` toggle adds and removes the overlay, the order line drags and the trade panel's price follows it, the last sale is marked on its bar.
8. Dots mode: unchanged scatter, hover card shows image, `#tokenId`, rank and traits.
9. Rename `apps/web/public/charting_library` locally and reload — the old `NftFloorChart` appears rather than an empty box.

## Known limits

* The \~30h gap between 2026-09-01T00:20Z and the deploy is **gone**, by choice. Candles and the scatter have a hole there.
* `openSeaConnected: false` chain-wide is unresolved. This round removes the *dependency* on that stream for facts and cosmetics; why the credential or env will not connect is still open.
* `1h cancels` still needs OpenSea `cancel` events. Seaport's `OrderCancelled` is on chain, but attributing an `orderHash` to a collection needs a matching `NftOrderSnapshot` row, so on-chain coverage would be partial in a way the other tiles are not.
* The sweep assumes token ids are contiguous from 1. Ids outside that range still arrive through the traffic-driven path, so the assumption costs coverage, not correctness.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.trover.tech/engineering/verify-2026-09-02-nft-chain-head.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.