> For the complete documentation index, see [llms.txt](https://docs.trover.tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.trover.tech/engineering/runbook.md).

# Operator runbook

## Deploy the backend

The frontend (`apps/web`) ships from Vercel on every push to `main`; never build the `web` image on the server. The backend (`api`, `worker`, `nft-realtime`) is built and recreated on the server at `~/trover` (`/home/teztap/trover`, host alias `trevor-server`).

Rules:

* **One deploy at a time.** Two `docker compose` runs racing each other corrupted Docker's name index and took the API down for about 50 minutes. Wait for `DEPLOY_EXIT=` in the previous log before launching the next.
* **Detached.** A build takes around ten minutes on this box; launch it under `nohup` with a log file and check the log later. The `< /dev/null` is required or ssh keeps the session open.
* **Always pass both base files**, `-f docker-compose.yml -f docker-compose.production.yml`. The production file is an override and fails alone ("service migrate has neither an image nor a build context").
* **Add `-f docker-compose.nft-realtime.yml` and the `nft-realtime` service** whenever the worker or `@trover/integrations` changed; that service lives only in its own compose file and the ordinary deploy never rebuilds it. It sat dead for two days once for exactly this reason.
* `api`, `worker` and `migrate` share one build stage: list them in one `build` so the backend compiles once.
* **Migrations:** build before `run --rm migrate`. The migrate service runs the migrations baked into its image, so on a stale image it reports "No pending migrations" while the newest one is simply absent.
* `docker compose logs --since` returns zero lines at any window on this host. Measure with `--tail N | grep -c`, and check the denominator (`| wc -l`) before reporting a zero.

Backend without migration (api, worker, realtime):

```bash
ssh trevor-server 'cd ~/trover && nohup bash -c "set -x; git pull --ff-only && docker compose -f docker-compose.yml -f docker-compose.production.yml -f docker-compose.nft-realtime.yml build api worker nft-realtime && docker compose -f docker-compose.yml -f docker-compose.production.yml -f docker-compose.nft-realtime.yml up -d --force-recreate --no-deps api worker nft-realtime; echo DEPLOY_EXIT=\$?" > ~/trover/deploy-$(date +%Y%m%d)-a.log 2>&1 < /dev/null & echo launched'
```

Then, once:

```bash
ssh trevor-server 'until grep -q DEPLOY_EXIT= ~/trover/deploy-<date>-a.log; do sleep 20; done; tail -3 ~/trover/deploy-<date>-a.log; docker ps --format "{{.Names}} {{.Status}}" | grep trover'
```

With a migration, the inner command becomes `build migrate api worker [nft-realtime] && docker compose ... run --rm migrate && docker compose ... up -d --force-recreate --no-deps api worker [nft-realtime]`. API only: drop the realtime file and list only `api`.

After the deploy: `docker ps` shows every `trover-*` container `(healthy)`; `trover-nft-realtime-1` readiness (`:3003/health/ready` inside the container) reports two providers and `watched` in the thousands; `curl --fail https://api.trover.tech/health/ready`.

## Postgres on the server

Reload-safe settings applied with `ALTER SYSTEM` and `pg_reload_conf()` on 2026-09-07 (they persist in `postgresql.auto.conf` across container restarts):

```
wal_compression = on
max_wal_size = 4GB
checkpoint_timeout = 15min
```

Recommended, but they need a Postgres restart and are the operator's call:

```
shared_buffers = 4GB          # was the 128 MB default for a 38 GB database
wal_buffers = 64MB            # was 4 MB
effective_cache_size = 24GB
```

Access: the role and database are both `meow` (`TROVER_DB_USER` / `TROVER_DB_NAME`), not `trover`. Every `docker compose exec` inside an ssh heredoc needs `< /dev/null` or it swallows the rest of the script:

```bash
docker compose -f docker-compose.yml -f docker-compose.production.yml exec -T postgres psql -U meow -d meow -Atc "select 1;" < /dev/null
```

Per-service guardrails from `docker-compose.production.yml`: the API runs with `statement_timeout=15000` and `connection_limit=15`, the worker with `statement_timeout=120000` and `connection_limit=30`, the realtime service with `connection_limit=30&pool_timeout=20`; the migrate service has no statement timeout.

## Known server condition

The box (8 vCPU) also hosts another application's Postgres (`teztap-db-1`) on the same disk. That database has written 8.8 TB against our 3.9 TB, and `vmstat` shows 20 to 40% I/O wait under normal load. Consequences:

* backend builds take around ten minutes, and `docker compose up` of a large image can take minutes more;
* the API's 15 s statement timeout fires on queries that are fine on an idle disk, and worker inserts wait on `LWLock:WALWrite` (24 mint INSERTs were queued on it before the batched writes);
* a cold discovery page miss measured 6 s at load 20.

Per-event commits are the thing to avoid in worker code; write per window and move the cursor with the window. The Postgres settings above are the mitigation short of moving one of the two databases off the disk.

## Reconnect X

1. Set `SAFE_MODE=true` and `DRY_RUN=true`.
2. Rotate/reissue OAuth 2.0 user tokens with read/write/offline scopes and OAuth 1.0a user credentials for Account Activity.
3. Update secrets in the deployment provider and restart API/worker services.
4. Register or validate the webhook, then subscribe the bot account.
5. Confirm CRC, signed delivery, polling checkpoints, and a dry-run reply before considering live mode.

## Rotate keys

1. Keep safe mode enabled.
2. Revoke the old provider key first when compromise is suspected; otherwise create the replacement before revocation.
3. Update secret storage, roll API and worker instances, and confirm readiness/provider metrics.
4. Revoke the old credential and record the rotation in the operations log.

Never place a real credential in `.env.example`, source control, command history, or an admin audit metadata field.

On the dedicated host, edit `/home/teztap/trover/.env`, keep it mode `0600`, then recreate the affected Compose services. Vercel environment variables do not propagate to the API or worker on that host. For X webhooks, use the selected X Client App's Consumer/API Secret for CRC and signature validation; do not generate an unrelated webhook secret.

## Change the canonical token

1. Keep posting disabled.
2. Verify the new address independently on Robinhood Chain and Blockscout.
3. Update the registry through the protected admin API or a reviewed migration. Mark the former address `old`; do not silently convert it into an equivalent alias.
4. Run manual analyses for both addresses and verify the conflict wording.
5. Update `CANONICAL_TROVER_TOKEN` and redeploy only after registry and environment values agree.

## Recover missed X events

1. Leave `ENABLE_POLLING_FALLBACK=true` and inspect polling checkpoints/dead letters.
2. Requeue relevant dead letters with `pnpm admin -- requeue ID`.
3. Use X webhook replay when the account tier supports it, then let fingerprints discard duplicates.
4. Confirm `x_events`, `analysis_runs`, and queue-depth metrics converge.

## Disable autopost immediately

Set both `SAFE_MODE=true` and `DRY_RUN=true` in the deployment environment, then restart API and worker services. These environment hard stops override all feature flags and approvals. If necessary, also revoke the X user access token while preserving the app bearer token for read-only diagnosis.

## Disable Relay bridging immediately

Set `ENABLE_RELAY_BRIDGING=false` and restart API and worker services. This blocks API, mention, and DM bridge execution even if profile settings remain enabled. For a social-only stop use `ENABLE_X_RELAY_BRIDGING=false`. For a single network add it to `RELAY_DENIED_CHAIN_IDS`; the denylist overrides Relay metadata and database operator settings. Reconcile already-submitted Relay request IDs before changing or retrying them.

## Incident checks

* `/health/ready` for PostgreSQL/Redis readiness.
* `/metrics` for provider failures, X rate limits, queue depth, skipped replies, latency, and OpenAI usage/cost estimates.
* `/v1/admin/jobs` for dead letters.
* `/v1/admin/replies?status=pending` for approval backlog.
* `source_conflicts` for unresolved identity mismatches.
* `docker inspect trover-worker-1 --format '{{.RestartCount}}'` and `docker logs --tail 3000 trover-worker-1 | grep -c FATAL`: a climbing restart count with `Ineffective mark-compacts near heap limit` is the worker running out of heap. The `"worker memory"` line (every 60 s: rss, heap, which memory-heavy job holds the slot) should plateau well under 1 GB across a round robin. Headroom on the server is `TROVER_WORKER_MEMORY_LIMIT=6g`, `TROVER_WORKER_MAX_OLD_SPACE_MB=4096`.
* `docker logs --tail 20000 trover-worker-1 | grep '"RPC usage"'`: the Alchemy row in the tens of calls per window, no `net_listening`. See `RPC-COST-CONTROL.md`.
* `docker logs --tail 50000 trover-api-1 | grep -c '"statusCode":429'`: our own rate limiter, not OpenSea. Remote addresses in the log must be real client addresses, not `172.19.0.1`; if they are all the bridge address, `trustProxy` is off.
* NFT freshness: `redis-cli --scan --pattern "trover:nft-onchain-lag:*"` must list the head lanes for every chain, and `select max("occurredAt") from nft_market_events where "eventType"='mint'` must be within a minute of `now()`. "exceeded its ceiling" in the worker log must not repeat every tick; it is followed by "Abandoned on-chain NFT ... scan settled" before the next one.
* Rejected fills: `select count(*) from nft_market_events where accepted=false and "rejectionReason"='onchain_asset_verification_failed'` should be zero; the repair job drains it. Check `accepted` in the database, not the realtime log.
* Scanner gap: never edit the `stabledex_indexer_cursors` row. Set `trover:nft-mint-gap:v1:<chain>` to `{"from":"…","to":"…"}` and watch for "On-chain NFT mint gap drain" then "gap drained". Measured drain rate is about 2,000 blocks per 70 s.
* Discovery lists: `redis-cli --scan --pattern "trover:nft-discovery:v1:*"` lists three keys per chain and `"NFT discovery lists refreshed"` appears every minute; a second `curl` of `/v1/nft-market/collections` answers with `x-trover-cache: hit`.
* A backlog on `trover-nft-market` from a container older than the last deploy (`docker ps --format "{{.Names}} {{.CreatedAt}}"`) is the stale canary; it is stopped and the worker drops every `nft-canary-reconcile*` job.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.trover.tech/engineering/runbook.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.