> For the complete documentation index, see [llms.txt](https://docs.trover.tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.trover.tech/engineering/verify-2026-09-07-discovery-live.md).

# Verify 2026-09-07 (b) — leaderboard chain, 15-minute sold-out, instant discovery, live deltas

**Migration:** none. One new env key with a default (`NFT_DISCOVERY_CHAINS`, extra chains beside `OPENSEA_CHAIN`; leave unset).

## What was wrong

* **Ink collection drawn as Robinhood.** The API already reported `inkbrokers-nft` as `ink` (the `normalized` row carries the chain). The marketplace rows are the raw OpenSea objects, whose chain lives only on `contracts[].chain`; `collectionChainFor()` never read it, defaulted to `robinhood`, and then also failed to find the row's enrichment because the lookup key was `robinhood:inkbrokers-nft`.
* **"Sold out" forever.** The 10-minute presentation rule from round (a) was not live yet when it was checked (that build was still exporting images). The window is now 15 minutes, as asked.
* **"Newly launched" an hour late.** `/v1/nft-market/collections?mode=new` called OpenSea's `/collections?order_by=created_date` with the catalogue cache (3,600 s), then up to 24 `collectionStats` calls on a miss.
* **Volume blank.** Rows discovery had never synced had `volume24hNative` null; the read-time fill only patched the floor.

## Changes

* `apps/worker/src/nft-discovery.ts` (new) + scheduler `nft-discovery-lists` every 60 s: fetches trending / top / new per chain (3 calls a minute, the "new" list with a 45 s cache), stores each raw list in Redis (`trover:nft-discovery:v1:<chain>:<mode>`, 15 min), creates a row for every listed collection whose slug is not already held on another chain with a contract, writes 24 h volume and sales from our own sale tape for every row on the page, and fills floor / owners for up to 10 rows per mode per tick from `collectionStats` (marker `trover:nft-discovery-stats:v1:<id>`, 3 min).
* `GET /v1/nft-market/collections`: serves the worker's warm list when present (never calls the provider list on the request path then), caches the whole page 15 s (`trover:nft-discovery-page:v1:…`, `x-trover-cache: hit|miss`), resolves provider identities with no chain via our contract rows (`resolveIdentityChains`), overlays tape volume, and bounds the read-time floor fill to 8 rows. `/search` uses the same chain resolution.
* `packages/domain`: `nftSoldOutBadgeMs` = 15 min.
* Web:
  * `NftMarketplace`: chain from our index first, then `contracts[].chain`; slug-only enrichment fallback; skeleton rows + "loading newly launched collections" while a tab loads, a thin progress bar while it refreshes; live deltas on floor / volume / owners / listed; minting chip with `minted/max (pct)`.
  * `NftTrendingBoard`: rows animate to their new positions (framer-motion `layout`), deltas on holders / listed / volume / floor / sales, `MintPulse` reads "MINTING 500/5,000 (10%) · 12/h" with the minted count floating each increase.
  * `NftLifecycleChip`: `minted` / `max` props → "minting 500/5,000 (10%)".
  * Terminal: every stat tile floats its change; the minting button carries the same progress text.
  * `LiveDelta.tsx` (new): `LiveNumber` — keeps the previous value, floats `+Δ` / `−Δ` for 2.4 s, green up / rose down (`invert` for the opposite).

## Deploy

Frontend: Vercel on push. Backend — **after** `deploy-20260907-a` has finished (one deploy at a time):

```
ssh trevor-server 'cd ~/trover && nohup bash -c "set -x; git pull --ff-only && docker compose -f docker-compose.yml -f docker-compose.production.yml -f docker-compose.nft-realtime.yml build api worker nft-realtime && docker compose -f docker-compose.yml -f docker-compose.production.yml -f docker-compose.nft-realtime.yml up -d --force-recreate --no-deps api worker nft-realtime; echo DEPLOY_EXIT=\$?" > ~/trover/deploy-20260907-b.log 2>&1 < /dev/null & echo launched'
```

No migrate step.

## Verify

```
# worker job ran and the lists are warm
docker logs --tail 3000 trover-worker-1 | grep -c "NFT discovery lists refreshed"
docker exec trover-redis-1 redis-cli --scan --pattern "trover:nft-discovery:v1:*"
# the route is a cache hit on the second call and never waits on the list
curl -s -D - -o /dev/null "localhost:3001/v1/nft-market/collections?chain=robinhood&mode=new&limit=24" | grep -i "x-trover-cache"
# chain of an Ink row in the trending page
curl -s "localhost:3001/v1/nft-market/collections?chain=robinhood&mode=trending&limit=24" | python3 -c 'import sys,json; print([(r["slug"], r["chain"]) for r in json.load(sys.stdin)["normalized"] if r["slug"]=="inkbrokers-nft"])'
# sold-out presentation: last mint older than 15 min → "secondary"
curl -s 'localhost:3001/v1/nft-market/trending?chain=robinhood&window=24h&limit=40&lifecycle=ended' | python3 -c 'import sys,json; [print(r["slug"], r["lifecycle"]) for r in json.load(sys.stdin)["rows"][:10]]'
```

On the site: the Ink Brokers row wears the Ink badge; switching to "newly launched" shows skeleton rows then the list within a second; a minting row reads `MINTING n/N (p%)`; leave the board open through a sale and the volume / sales cells float `+…`.

## Not done, and why

External data services: none added. Everything above runs on our own chain indexer and the existing OpenSea key. I cannot create accounts on third-party services; if a provider is wanted later (holder analytics at scale, cross-chain metadata), the candidates with free tiers are listed in the reply, and the key goes in the server `.env` only.

## Addendum — cold-miss cost

Measured after deploy (b): page cache hit 12 ms, cold miss 6.0 s (24 × `trackCollection`, a 2.1 s tape aggregation — 87 ms per collection on the 5 GB table — and up to 8 paced stats calls). Changed in (c): the page is kept 90 s and served stale past 15 s while one request rebuilds it in the background (`x-trover-cache: hit|stale|miss`); with the worker's warm list present the route skips tracking, the stats fill and the tape read, since the worker wrote all three into the rows within the last minute. Deploy: api only.

```
ssh trevor-server 'cd ~/trover && nohup bash -c "set -x; git pull --ff-only && docker compose -f docker-compose.yml -f docker-compose.production.yml build api && docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --force-recreate --no-deps api; echo DEPLOY_EXIT=\$?" > ~/trover/deploy-20260907-c.log 2>&1 < /dev/null & echo launched'
```

## Addendum (d) — mints were 30 minutes late; zeros on new rows; freshness line

**Measured 11:20Z:** sales via the sockets: 1,204 fills/hour, median 1.1 s after the block, all accepted. Mints: newest stored mint 20 minutes old, live mints median 1,790 s late. Cause: `runLane()` in both on-chain scanners guarded per *lane* while awaiting every chain together, so the Robinhood head tick (seconds) stayed "in flight" until the Ink head scan settled (30–46 min per run, 7–11k Ink mints each). The mint lag keys had expired from Redis — "not running". Fixed: guard and ceiling per `${lane}:${chain}` in `nft-onchain-mints-indexer.ts` and `nft-onchain-sales-indexer.ts`.

**Zeros on "newly launched":** OpenSea reports `floor_price 0` / `num_owners 0` for a launch it has not counted, and the discovery job stored those and then treated the row as enriched. Now a zero is retried like null, and owners fall back to distinct recipients of accepted mints and sales from our tape.

**Freshness:** `/collections` and `/trending` responses carry `freshness { chain, headBlock, indexedBlock, indexedAt, lagBlocks, listRefreshedAt, servedAt }` from the live sale lane's Redis lag record. `DataFreshness.tsx` renders "block 56,795,430 · 12s ago" (green <1 min, amber <5 min, red after or when the lane is silent) in the marketplace header and in the trending board header, replacing the static "Live" dot.

Deploy: api + worker.

```
ssh trevor-server 'cd ~/trover && nohup bash -c "set -x; git pull --ff-only && docker compose -f docker-compose.yml -f docker-compose.production.yml build api worker && docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --force-recreate --no-deps api worker; echo DEPLOY_EXIT=\$?" > ~/trover/deploy-20260907-d.log 2>&1 < /dev/null & echo launched'
```

Verify after: `redis-cli --scan --pattern "trover:nft-onchain-lag:mints-head:*"` exists and `select max("occurredAt") from nft_market_events where "eventType"='mint'` is within a minute of `now()`.

## Addendum (e) — mint writes batched; the disk is the bottleneck

After (d) the Robinhood head lane still overran its 20 s ceiling every tick: 191 mints took 170 s. `pg_stat_activity` showed 24 mint INSERTs waiting on `LWLock:WALWrite`; `vmstat` 41% I/O wait; two Postgres containers share the one disk (`teztap-db-1` has written 8.8 TB, ours 3.9 TB); our Postgres ran on defaults (shared\_buffers 128 MB for a 38 GB database, wal\_buffers 4 MB, max\_wal\_size 1 GB, checkpoint\_timeout 5 min). Each mint was five commits.

* `recordOnchainMintBatch()` (nft-market-indexer.ts) writes a window's mints per collection as one twin lookup + one `createMany` (skipDuplicates on the same dedupe/canonical keys `storeEvent` uses) + one read-back + one observations `createMany`; live mints publish individually up to 40, then one `snapshot_required`. Cross-source twins still go through the per-event path so promotion of an OpenSea "transfer" row keeps working.
* Backfill lane: the 257,038-block gap (≈03:30–11:29Z) it abandoned under `NFT_ONCHAIN_MAX_LAG_BLOCKS=216000` is recovered by setting that env to 1,500,000 and resetting `nft-onchain-mints:robinhood` to 56542148 after the deploy.
* Postgres, reload-safe, applied with `ALTER SYSTEM` + `pg_reload_conf()`: `wal_compression=on`, `max_wal_size=4GB`, `checkpoint_timeout=15min`. **Recommended but needs a restart (user's call):** `shared_buffers=4GB`, `wal_buffers=64MB`, `effective_cache_size=24GB`.

Deploy: worker only.

```
ssh trevor-server 'cd ~/trover && nohup bash -c "set -x; git pull --ff-only && docker compose -f docker-compose.yml -f docker-compose.production.yml build worker && docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --force-recreate --no-deps worker; echo DEPLOY_EXIT=\$?" > ~/trover/deploy-20260907-e.log 2>&1 < /dev/null & echo launched'
```

Then: `update stabledex_indexer_cursors set "nextBlock"=56542148 where source='nft-onchain-mints:robinhood';` Verify: head lane no longer logs "exceeded its ceiling" for robinhood; newest mint within a minute; the backfill cursor climbs past 56799186.

### (e) result and the gap marker

After (e): head lane 693–1,152 mints per run in 60–70 s (was 191 in 170 s), WAL waiters 24 → 4, newest mint within 3 min while the head lane caught up its own 15k-block backlog. The cursor reset for the gap did **not** hold: the in-flight backfill run committed `toBlock + 1` over it at the end of its window (cursor read 56802814 seconds after the reset to 56542148). Added a gap marker the backfill lane drains before its cursor:

```
docker exec trover-redis-1 redis-cli set trover:nft-mint-gap:v1:robinhood '{"from":"56542148","to":"56799186"}'
```

The lane logs "On-chain NFT mint gap drain" with the span each tick, advances the marker per window, never touches the live cursor, and logs "gap drained" then deletes the key. At the measured \~2,000 blocks per 70 s the 257k-block span takes about 2.5 h. Deploy: worker only (same command as (e), log `-f`).

## Addendum (f) — per-network leaderboards, "last mint x ago"

* `GET /v1/nft-market/collections` gained `scope=chain|all` (default `chain`): provider rows from other chains are dropped, the page is topped up from our own rows on that chain (by volume / last sale, or newest row for `mode=new`), and the raw `data.collections` list the marketplace renders is cut and extended the same way (synthetic rows carry `source: "trover_index"`). Page cache key bumped to `…page:v2:…:<scope>`. The marketplace and the DEX home pass `scope=all` only for "all networks"; the trending board was already chain-only from our index.
* `lastMintAt` on `marketJson` rows and on trending rows; `MintAgo.tsx` renders "last mint 3m ago" (green under five minutes) on both leaderboards.

Deploy: api only, after (f) worker deploy.

```
ssh trevor-server 'cd ~/trover && nohup bash -c "set -x; git pull --ff-only && docker compose -f docker-compose.yml -f docker-compose.production.yml build api && docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --force-recreate --no-deps api; echo DEPLOY_EXIT=\$?" > ~/trover/deploy-20260907-g.log 2>&1 < /dev/null & echo launched'
```

Verify: `curl -s 'localhost:3001/v1/nft-market/collections?chain=robinhood&mode=trending&limit=24' | python3 -c 'import sys,json; d=json.load(sys.stdin); print(sorted({r["chain"] for r in d["normalized"]}), len(d["normalized"]), d["normalized"][0].get("lastMintAt"))'` → `['robinhood'] 24 <iso>`.

## Addendum (g) 2026-09-08 — collections stuck in "queued"

`dex.trover.tech/nfts/robinhood/maxextractors` sat on "queued · waiting for verified sales". Cause: every full sync ran `UPDATE nft_order_snapshots SET status='inactive' WHERE collectionId=… AND generation < …` with no status filter, rewriting every historical row (button-presser: 440,648 rows over 1,247 generations), 39 s per statement, the batch transaction timing out, and the sync queue stalled at eight jobs of six minutes each while opened terminals waited behind discovery syncs.

* `nft-market-indexer.ts`: retire only `status: "active"` rows of earlier generations; prune inactive rows older than 7 days after each sync.
* `nft-market-routes.ts` `trackCollection`: BullMQ `priority: 1` for a terminal-opened collection (5 otherwise).
* `DexNftFloorChart.tsx`: "first index queued · usually a few minutes" instead of "waiting for verified sales".

Migration: none. Deploy: api + worker (frontend via Vercel). Verify: `select "syncStatus", "lastProviderAt" from nft_collection_markets where slug='maxextractors'` moves off `queued` within a few minutes of the deploy; no `nftOrderSnapshot.updateMany` timeouts in the worker log.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.trover.tech/engineering/verify-2026-09-07-discovery-live.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.