> For the complete documentation index, see [llms.txt](https://docs.trover.tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.trover.tech/engineering/verify-2026-09-05-instant-activity.md). # Verify: instant activity, duplicate collections, images, mint stats (2026-09-05) Migration: **none**. Compose change: worker `connection_limit` 10 → 30 (production override). The `nft-realtime` service is part of this deploy — it has been running a 2026-09-02 image, unhealthy, since it started. ## What was wrong | symptom | cause | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | sale on chain, on the app 3–10 min later | the websocket ingestor (`nft-realtime`) hit one startup error on 2026-09-02 while Postgres was in recovery, never retried, and sat "alive" with zero providers for two days; every sale went through the polling scanner | | ponsguy-nft "last sale 1h ago" while OpenSea shows 12 min | 34 contracts had two rows (named slug + bare address). `findFirst` with no order picked either; the process cache pinned it. 17,000 events sat on rows nobody can open | | blank thumbnails everywhere | 27,405 of 35,383 listed tokens on Robinhood had no metadata row; artwork only came from per-token chain reads triggered by trades | | stats skeleton for \~20 s on open | the first loads ran inside `requestAnimationFrame`, which a background tab never fires | | "index failed · retry" on live collections | a provider abort/timeout on a refresh set `syncStatus: failed` regardless of the previous state | | minting collections with no figures | `mintedSupply` only ever came from OpenSea drops/stats; chain-registered minting contracts had nothing | | worker `connection pool timeout` ×312 / 2,000 log lines | 40 concurrent job slots against a pool of 10 | ## Deploy (server, one at a time, foreground) ```bash cd ~/trover && git pull docker compose -f docker-compose.yml -f docker-compose.production.yml -f docker-compose.nft-realtime.yml build api worker nft-realtime docker compose -f docker-compose.yml -f docker-compose.production.yml -f docker-compose.nft-realtime.yml up -d --force-recreate --no-deps api worker nft-realtime docker ps --format '{{.Names}} {{.Status}}' | grep trover- ``` No `run --rm migrate` needed. ## Verify 1. Realtime service ready within a minute, two providers, watched ≈ full catalogue: ```bash docker exec trover-nft-realtime-1 node -e "fetch('http://localhost:3003/health/ready').then(async r=>console.log(r.status, await r.text()))" ``` Expect `ok:true`, `providers` = 2 with `lastSeenAt` recent, `watched` in the thousands. 2. Sales land in seconds: pick any fresh fill on blockscout and query `SELECT "observedAt" - "occurredAt" FROM nft_market_events WHERE "transactionHash" = ''`. Expect single-digit seconds and `source` = `robinhood_wss:*`. 3. Duplicate rows drain: `merge_duplicate_nft_collections` runs every 2 min, 5 pairs per run; worker log line `Merged duplicate NFT collection rows`. After \~15 min: ```sql SELECT count(*) FROM nft_collection_markets a JOIN nft_collection_markets n ON n.chain=a.chain AND lower(n."contractAddress")=lower(a."contractAddress") AND n.id<>a.id AND n.slug NOT LIKE '0x%' WHERE a.slug LIKE '0x%'; ``` Expect 0. ponsguy-nft's last sale then matches OpenSea. 4. Images: open a collection, wait one sync tick; worker log `NFT image crawl complete` for it. Listed-token coverage: the "missing\_meta" count in the query below falls toward 0 for opened collections. ```sql SELECT count(*) FILTER (WHERE m.id IS NULL) missing_meta FROM nft_order_snapshots o LEFT JOIN nft_token_metadata m ON m."collectionId"=o."collectionId" AND m."tokenId"=o."tokenId" WHERE o.side='listing' AND o.status='active'; ``` 5. Terminal: open a collection in a **background** tab, switch to it after 10 s — it is already loaded. 6. Minting: `SELECT slug, "mintedSupply", "totalSupply" FROM nft_collection_markets WHERE lifecycle='minting' ORDER BY "lastMintAt" DESC LIMIT 10;` — `mintedSupply` populated within 2 min; `totalSupply` where the contract exposes a cap. 7. Pool: `docker logs --tail 2000 trover-worker-1 | grep -c "connection pool"` trends to 0. ## Addendum (same day, after the first deploy) Three more causes found while verifying, all shipped in follow-up commits: | symptom | cause | fix | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ingestor up but every live fill "receipt could not be found" | receipts were read from the HTTP archive lane, which trails the websocket provider by seconds | receipts and blocks read from the delivering socket first, six short retries, archive as fallback | | every scheduled NFT job 5 min late | `trover-nft-market` had all 8 slots in 2.5–6 min collection syncs, 198 waiting; 186 of those were reconcile hints from the stale canary container | new `trover-nft-scheduled` queue (concurrency 4) for the backfill scanners, recent-sales poll, tracked round-robin and merges; every canary hint dropped; `trover-opensea-browser-canary-1` stopped | | first data request at 12 s on the terminal | Vercel Live toolbar (signed-in team members only) loaded at 3.3 s, its iframe at 11.8 s, app requests at 12.0 s | `x-vercel-skip-toolbar: 1` on every response | | offers / activity / holders tabs blank | the three panel branches were deleted by the grid rewrite in 18c1580 | restored | Deploy for the addendum (already run): `build worker nft-realtime` + `up -d --force-recreate --no-deps worker nft-realtime`, and `docker stop trover-opensea-browser-canary-1`. Measured after: `robinhood_wss:alchemy` sale latency avg **1 s**, max **3 s** (85 sales / 8 min); zero receipt rejections. ## Addendum 2 (evening) | item | finding | fix | | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | "13 s gap" before the first data request | Only in the user's Chrome: two wallet extensions (Leap + a provider proxy) recurse on `window.ethereum.request` until stack overflow. Clean headless Chromium: first API request at **3.7 s**, DOM ready 3.5 s | Not an app defect. Disable one of the two extensions, or verify in incognito. The Vercel toolbar header stays (harmless). | | Ethereum collections labelled Robinhood | `/nfts/robinhood/` created a robinhood row for any slug; 35 rows (BAYC, Pudgy, CryptoPunks, …) carried the wrong chain | worker re-files a row under the chain the provider's contracts name (or merges into the right-chain twin); api never creates a row for a slug known elsewhere and serves the right-chain row; web redirects to `/nfts//` | | no "processing" state while a buy is in flight | nothing published between quote and confirmation | worker publishes `action_state` (executing / submitted / confirmed / failed); terminal shows an amber "processing · about to be sold" on the row and card, instantly for its own action, via the stream for everyone else's. This chain has no public mempool, so third-party buys are only visible once mined (\~1 s). | | worker never redeployed by the earlier background run | the ssh session died before the build ran | re-run in the foreground; queue split live: `trover-nft-market wait=0`, `trover-nft-scheduled` holds exactly its four jobs | Measured after the queue split: duplicate pairs **0**, ponsguy-nft newest sale **38 s** old, `robinhood_wss` sale latency **2 s** avg over 378 sales / 10 min, worker pool timeouts **0**. Re-chain check: `SELECT slug, chain FROM nft_collection_markets WHERE slug IN ('pudgypenguins','boredapeyachtclub','cryptopunks');` — expect `ethereum` after their next sync (open the page once to trigger it). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.trover.tech/engineering/verify-2026-09-05-instant-activity.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.