> For the complete documentation index, see [llms.txt](https://docs.trover.tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.trover.tech/engineering/verify-2026-09-06-rejected-sales-and-oom.md).

# Verify — 2026-09-06: rejected live sales, worker OOM, chain resolution, extension notice

## What was wrong (measured)

**Live sales were being thrown away.** Every Seaport fill decoded by the websocket lanes (`robinhood_wss:alchemy`, `robinhood_wss:robinhood-public`) and the on-chain Seaport scanner was re-verified through `verifyTransaction`, whose RPC list was the rate-limited public endpoint plus an archive node budgeted at 0.5 rps (and whose `|rps=0.5` suffix travelled into the URL, so that entry never worked at all). A failed lookup was stored as `accepted=false, rejectionReason=onchain_asset_verification_failed`.

Per 30-minute bucket over the last 12 hours, websocket-lane sales:

| bucket      | accepted | rejected (verification) |
| ----------- | -------- | ----------------------- |
| 09-05 23:00 | 137      | 1,511                   |
| 09-06 03:00 | 146      | 1,390                   |
| 09-06 09:00 | 391      | 1,059                   |
| 09-06 10:30 | 54       | 517                     |

Roughly 85% of live fills were hidden. That is why `ponsguy-nft` read "last sale 2h ago" while OpenSea showed one at 10:37: our websocket row for that exact tx (`0xa5dc9e34…`) existed, rejected, and the sale only surfaced when the 5-minute OpenSea poll re-fetched it.

**The worker was dying of heap exhaustion.** 16 restarts since 2026-09-05 19:36 (`FATAL ERROR: Ineffective mark-compacts near heap limit`). Each crash killed every sync in flight, which is why `alpha-gardeners-pass` sat `queued` for five hours. Cause: the on-chain sale/mint scanners release their `laneInFlight` mark when a run exceeds its wall-clock ceiling, but the run itself keeps going (viem offers no cancel). The next tick (every 5–15 s) then started another full run beside it. The log showed "exceeded its ceiling; releasing the slot" every \~20 s, i.e. dozens of concurrent scans each holding its own log arrays.

**Wrong-chain rows kept coming back.** `GET /collections/:slug` trusted the URL's chain; `trackCollection` only refused to create a row when a twin on another chain had already synced a contract. So while the Ethereum row for `alpha-gardeners-pass` waited for its first sync, every poll from the open tab re-created the Robinhood row the worker had just merged away.

**nft-realtime Prisma pool.** Default limit 17, timeout 10 s: under a few hundred fills an hour it logged "Timed out fetching a new connection" and dropped the fill.

## Changes

* `apps/worker/src/nft-market-indexer.ts`
  * `storeEvent` accepts `onchainVerified`; `recordVerifiedOnchainEvent` passes `true` (its callers decoded a chain log and checked the receipt themselves).
  * `verifyTransaction` returns `null` (unverifiable → accepted) when no RPC gives a definitive answer; `false` only when a node answered and the tx reverted / did not touch the contract, or two healthy nodes both report it missing.
  * `publicRpcs` strips `|rps=…|lanes=…` budgets and adds the basic lane first.
* `apps/worker/src/nft-rejected-repair.ts` (new) + `JOBS.repairRejectedOnchainSales`, scheduler `nft-rejected-sale-repair` every 2 min on `trover-nft-scheduled`: re-accepts rejected websocket/scanner fills (10 collections per run, one row per tx+token, never where OpenSea already stored the same fill as accepted), marks the listings they closed as filled, refreshes `lastSaleAt`, rebuilds candles, publishes `snapshot_required`. Leftover duplicates are relabelled `duplicate_of_accepted_row`. Exits immediately once nothing is pending.
* `apps/worker/src/nft-onchain-sales-indexer.ts`, `nft-onchain-mints-indexer.ts`: an abandoned run keeps its lane marked in flight until it actually settles; the queue slot is still released at the ceiling. At most one run per lane, ever.
* `apps/worker/src/index.ts`: `worker memory` log line every 60 s (rss, heap, which memory-heavy job holds the slot).
* `docker-compose.nft-realtime.yml`: `connection_limit=30&pool_timeout=20`.
* `apps/api/src/nft-market-routes.ts`: `resolveCollectionChain(slug, requested)` asks OpenSea (4 s cap, Redis-cached 6 h) which chain a never-synced slug lives on, and the route serves/tracks that chain so `market.chain` triggers the terminal's redirect on first paint. `trackCollection` treats a twin as known when it has a contract *or* a provider payload listing contracts.
* `apps/web/src/components/site/ExtensionConflictNotice.tsx` (new), mounted in `app/layout.tsx`: shows once when an error from a `chrome-extension://` script (or the "Maximum call stack size exceeded" overflow attributed to one) is observed, or when ≥2 EIP-6963 wallets announced and the page took >6 s to become interactive. Names known extensions (Compass Wallet for Sei, Revoke.cash, MetaMask, …). Dismissal is stored in `localStorage` per set of extensions.

## Migration

None. No schema change.

## Deploy (backend — one at a time, never in parallel)

```bash
ssh trevor-server
cd ~/trover && git pull
docker compose -f docker-compose.yml -f docker-compose.production.yml -f docker-compose.nft-realtime.yml build api worker nft-realtime
docker compose -f docker-compose.yml -f docker-compose.production.yml -f docker-compose.nft-realtime.yml up -d --force-recreate --no-deps api worker nft-realtime
docker ps --format '{{.Names}} {{.Status}}' | grep trover
```

Frontend deploys from the push to `main` (Vercel).

## Verify

Websocket-lane sales are accepted now (expect the rejected column at \~0 for buckets after the deploy):

```sql
select date_trunc('hour', "createdAt") h,
       count(*) filter (where accepted) ok,
       count(*) filter (where "rejectionReason"='onchain_asset_verification_failed') rej
from nft_market_events
where source like 'robinhood_wss%' and "eventType"='sale' and "createdAt" > now() - interval '3 hours'
group by 1 order by 1;
```

Repair drains (should reach 0 within \~30–60 min; the job logs "Re-accepted on-chain sales rejected by the redundant RPC verification"):

```sql
select count(*) from nft_market_events
where accepted=false and "rejectionReason"='onchain_asset_verification_failed'
  and (source like 'robinhood_wss%' or source in ('onchain_seaport_scan','robinhood_rpc','onchain_mint_scan'));
```

ponsguy: `lastSaleAt` within minutes of OpenSea's newest sale, and the API body's `lastSale.source` is a websocket lane:

```bash
curl -s 'https://api.trover.tech/v1/nft-market/collections/ponsguy-nft?chain=robinhood' | jq '.lastSale | {occurredAt, source}'
```

Worker memory: `docker logs --tail 2000 trover-worker-1 | grep '"worker memory"'` should show `heapUsedMb` plateauing, and "exceeded its ceiling" must not repeat every tick — it is followed by "Abandoned on-chain NFT … scan settled" before the next one. `docker inspect trover-worker-1 --format '{{.RestartCount}}'` stops climbing.

Chain: `curl -s 'https://api.trover.tech/v1/nft-market/collections/alpha-gardeners-pass?chain=robinhood' | jq .market.chain` returns `"ethereum"`, and the old URL redirects in the browser.

Extension notice: in the affected Chrome profile, open any collection page; the amber notice names the extensions; "Got it" hides it for that set permanently.

## Addendum (same day, second round)

Measured after the first deploy: every websocket-lane sale stored after the realtime container restarted was accepted (46/46), `alpha-gardeners-pass` reports `ethereum`, but the repair backlog was 58,862 rows — including 21,398 rejected **mint** events from the mint scanner and \~6k OpenSea rows — and `ponsguy-nft.lastSaleAt` still read 08:01 with an accepted 11:00 fill on disk.

* Repair job now covers every source, runs on a 100 s time budget per 2-minute tick (40 collections per query) instead of 10 collections per tick.
* `recordVerifiedOnchainEvent` stamps `lastSaleAt` on every live sale.
* Header on trover.tech gained a `dex` entry (plain anchor to <https://dex.trover.tech>).
* Clipboard chip in the DEX header now also fires on a copied collection **name** (2–48 chars, exact name/slug match after folding) and shows the collection logo.
* Trovernomics progression rewritten as two roads to every level (verified actions 10/40/120/300 or cumulative burns 100k/300k/700k), five verified actions per UTC day cap, in en/ru/az. **No backend progression registry exists in this repo**; the page copy and levels table are the only "logic" here.

Deploy: worker and nft-realtime (same three-file compose commands as above). Frontend via Vercel on push. No migration.

## Addendum 3 — the actual heap leak

After the scan pile-up fix the worker still climbed \~90 MB/min (443 → 1,604 MB heap in 14 minutes) with `sync_tracked_nft_markets` holding the memory-heavy slot. Cause: `OpenSeaClient`'s in-process `cache` Map had no eviction — every orderbook page, item page and stats body was retained for the life of the process, and Redis hits were copied into it as well. Fix: `remember()` caps it at 300 entries (expired first, then oldest). Server `.env` also gained `TROVER_WORKER_MEMORY_LIMIT=6g` and `TROVER_WORKER_MAX_OLD_SPACE_MB=4096` (backup taken first) as headroom.

Verify: `docker logs --tail 3000 trover-worker-1 | grep '"worker memory"'` — `heapUsedMb` should plateau well under 1 GB across a full round robin instead of climbing every minute. Deploy api, worker and nft-realtime (all three build `@trover/integrations`).

## Addendum 4 — Alchemy spend (24.4M of 26.7M CU in two days)

Measured from the worker's own `RPC usage` meter, one 15-minute window:

| host                                   | calls | CU      | getLogs |
| -------------------------------------- | ----- | ------- | ------- |
| robinhood-mainnet.g.alchemy.com        | 4,100 | 258,574 | 3,268   |
| rpc.mainnet.chain.robinhood.com (free) | 192   | 4,755   | 17      |
| chainstack (free tier)                 | 268   | 6,786   | 48      |

Projected 745M CU/month. Cause: `pooledTransport` ranked every endpoint of the LOG pool by stability; a paid node never 429s, so Alchemy ranked first and took \~98% of the `eth_getLogs` from the backfill scanners and the per-collection ownership walks, plus 664 `eth_getBlockByNumber` and 117 `net_listening` ranking probes per window. The `cu=200M` ceiling on that entry was far above the plan.

Changes:

* `apps/worker/src/rpc-transport.ts`: two tiers — free endpoints ranked among themselves, budgeted (paid) endpoints unranked and only reached when the free tier fails.
* `apps/worker/src/nft-realtime.ts`: both websocket lanes see every fill; only the first lane fetches the receipt and block, the second reuses them (halves per-fill RPC, and the Alchemy lane's share of it goes to \~0).
* Server `.env`: `ROBINHOOD_LOG_RPC_URLS` Alchemy entry now `cu=15M` (backup taken). The meter's month-to-date (18.9M) is already above it, so Alchemy getLogs are refused for the rest of this cycle and capped at 15M/month afterwards.

Verify after deploy: `docker logs --tail 20000 trover-worker-1 | grep '"RPC usage"' | grep alchemy` should show calls in the tens per window, `net_listening` absent; the free hosts carry the getLogs. Alchemy dashboard: getLogs/getBlockByNumber fall to near zero.

Not covered here: Alchemy's own charge for websocket subscription notifications on the realtime lane (not visible to our meter). If the dashboard still shows spend after this, swap the Alchemy `wss` out of `ROBINHOOD_NFT_WS_URLS` for the publicnode one.

## Addendum 5 — the heap, measured

A 240-second sampling heap profile through the inspector on a fresh worker (heap 106 → 882 MB during the sample) put the live allocations here:

| live MB | site                                                  |
| ------- | ----------------------------------------------------- |
| 166.7   | `persistRobinhoodSwaps` (dex-chain-indexer)           |
| 125.4   | viem http `request` (responses retained by the above) |
| 39.8    | undici JSON parse (same responses)                    |
| 19.9    | viem `getLogs`                                        |

The DEX swap tape read every block from its cursor to the head in one tick, held every swap in one array, and wrote once at the end. After downtime that range was hours of a 10-block-a-second chain; a tick that died before the write (lock renewal failures, the OOM itself) left the cursor behind, so the next tick re-read it all. Fixed: per-window writes with the cursor following each window, and at most 6,000 blocks per tick.

Verify: `"worker memory"` lines flat across a round robin; `stabledex_indexer_cursors` row `robinhood-swaps:live` advancing every tick; no `FATAL` in the worker log; the Alchemy line in `"RPC usage"` in the tens of calls per window.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.trover.tech/engineering/verify-2026-09-06-rejected-sales-and-oom.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.