# Troubleshooting (https://condensate.dev/operations/troubleshooting/)

> Find the failing part and fix it, starting from read-only checks.

This page helps you find which part is failing (browser, application server, runtime, engine or provider) and fix it. Most checks only read state through the runtime's loopback HTTP API. Find the failing part before you restart anything, because a restart interrupts active tasks and hides the cause.

## Find the boundary

| Boundary | Default address | Check first |
|---|---|---|
| Dex app | `http://127.0.0.1:5189` | The server is running, and `CONDENSATE_AGENTS_URL` points at the runtime |
| Runtime (agents-host) | `http://127.0.0.1:4741` | `GET /health` |
| Review hub (gt-review) | `http://127.0.0.1:4700` | The hub is listening, the runtime it reads is up, and `CONDENSATE_HUB_URL` on the Dex app server points at it |
| Dojo | `http://127.0.0.1:4800` | The Dojo container is running, published on that port |
| Engine | Inside the runtime process | The thread's events and the failure class |
| Provider | Outside Condensate | The provider error recorded in the thread |

## The page does not load

Confirm the application's startup command and port. Opening `index.html` from disk does not start its backend.

The Dex app forwards its `/runtime/*` requests to `CONDENSATE_AGENTS_URL`. Setting that variable alone does not fix file paths. If the runtime runs on another machine or in a container, set `DEX_APP_GTR_MCP` to the path of the hub's MCP entry (it ends in `apps/gt-review/src/mcp.ts`) as that host sees it.

Dojo runs in Docker Compose and publishes port 4800 on loopback. If the page does not load, check that the container is up before you look at the app.

## The page loads but tasks do not

```sh
curl -i 'http://127.0.0.1:4741/health'
curl -i 'http://127.0.0.1:4741/threads?limit=5'
curl -i http://127.0.0.1:4741/workspaces
curl -i http://127.0.0.1:4741/live
```

`/health` returns the build identity and any open retain alert. `/live` lists the sessions the runtime holds and any account pool that is out of quota right now.

Distinguish a successful empty page from a transport error. A list can look empty or incomplete for ordinary reasons:

- A thread list returns 50 threads by default and 200 at most. Follow `nextAfter` while `hasMore` is true.
- Retired threads are hidden unless you pass `archived=archived` or `archived=all`.
- `seats=open`, the default, hides finished reviewer and fixer seats. Pass `seats=all` to see them.
- A `429` with code `thread-list-rate-limited` means the first page was requested too often. Follow `/workspaces/:name/changes` and read only the threads that changed.

Also check `CONDENSATE_AGENTS_URL` on the application server, the selected workspace, and pagination. A loaded page is not a complete inventory.

## A model turn fails

Read the thread's events with `GET /threads/:id/tail`, and check whether a turn is still running with `GET /threads/:id/turn`. Each failure is recorded with a class. The class tells you whether the runtime tried another model and what to change:

| Class | What it means | What to do |
|---|---|---|
| `quota` | The account's allowance is spent | Wait for it to reset, or add an account. The runtime may move the turn to the next model in the chain |
| `auth` | The login is missing, expired, or revoked | Sign in again for that engine. The runtime may move the turn to another engine |
| `model-not-found` | The model id is not in the engine's catalog | Fix the model id in routing config. The runtime does not fail over, because that would hide a typo |
| `unsupported-model` | The provider refuses this model for this login | Choose another model. The runtime does not retry the same login |
| `engine-unavailable` | The engine cannot run on this machine, for example a missing Codex CLI or one that is not codex-cli 0.160.0 | Check the `engines` list on `GET /accounts`, which names the binary, the version found and the version needed. Install that release or set `CONDENSATE_CODEX_BIN` to it, then send again |
| `tool` | A tool server would not connect | Fix the tool server. Another model would see the same broken tools |
| `permission` | A permission gate refused the action | Answer the approval, or change the policy |
| `session` | A protocol fault, such as a message sent to a running or closed session | Check the thread's state before you resend |
| `closed` | The session was closed on purpose, such as a seat removed by an operator | Nothing to fix. The runtime never retries this, by design |
| `interrupted` | The turn was aborted | Send the message again if you still want the work done |
| `stalled` | The CLI ended a turn after a tool result without answering it | The runtime continues the session. Check the thread if it does not |
| `unknown` | Anything unclassified, including a real bug | Read the error text in the event. Treat it as a bug |

To see which failures repeat across threads, use `GET /incidents?since=<epoch ms>`. It groups failed turns by class and by message fingerprint, and it counts the threads each one hit. The window covers the last 24 hours and resets when the runtime restarts.

A working list endpoint does not prove that model credentials work. Verify the engine and model pair, the account binding, the workspace policy, and provider availability.

The credential proxy is on by default, so sessions hold only placeholder tokens and the proxy adds the real credential on the way out. An auth error inside a session can then mean the proxy cannot reach the provider. Check `CONDENSATE_PROXY_URL` and `CONDENSATE_PROXY_CA` in the runtime's launch agent environment, and that the proxy container is running.

## Streaming pauses

Check the durable event log and the current turn status before you assume the task stopped.

- `GET /threads/:id/stream` sends live deltas only. Nothing is replayed, and a slow reader loses its oldest frames. Treat deltas as a preview. A `429` here means the thread already has as many stream subscribers as it allows. Close idle tabs, or read `follow` instead.
- To recover after a gap, read `GET /threads/:id/tail?after=<last seq>` first, then reopen the stream. `GET /threads/:id/follow?after=<seq>` also accepts a `Last-Event-ID` header as its cursor.
- `tail?wait=<ms>` holds an empty page open for up to 30 seconds, waiting for new events.
- An idle stream sends a heartbeat every 15 seconds by default. A proxy that buffers responses, or closes idle connections early, looks like a stall. The runtime sends `cache-control: no-cache, no-transform` and `x-accel-buffering: no`. Keep the proxy from overriding them.

## Context reductions repeat

The runtime shrinks a thread's context when it nears the model's limit. It records each step in the event log:

| Event | Meaning |
|---|---|
| `window` | A reduction ran. The record names the kind (`microcompact`, `collapse`, or `summary`), what was cut, and the model that wrote the summary |
| `window-pending` | The context crossed its line. With `forced` false it is only a warning. With `forced` true the next call will not fit, and the runtime waits `deadlineMs` for an answer to `POST /threads/:id/compact` (`GET` on the same path shows the state). `options` lists the modes you can choose |
| `compaction-disposition` | How a forced compaction request was settled |

When a window fills and the runtime has to recover, the log gets a system message that starts `window full, recovering (<stage>)`. The stage is `resume-skipped`, `recut`, or `restarted`. It is not an error, and the turn stays open.

When a reduction is refused, the runtime says why:

- **WindowThrash:** reductions ran too often in recent turns, so the runtime stopped reducing and recorded a system line.
- **WindowNoHeadroom:** the system prompt and skills leave no usable room under the ceiling.
- **WindowBlocked:** a delegated child thread has not joined yet, so the cut waits until the child merges.

Summaries are written by the curator model, with a fallback. Set them with `CONDENSATE_CURATOR_MODEL` and `CONDENSATE_CURATOR_FALLBACK_MODEL`. A summary record with no `summaryModel` means no model wrote it, which happens for the cheap kinds and for a forced window whose curator never answered.

Inspect the window records and the curator's answers first. Keep the durable log. Deleting history is not a diagnosis of a model-facing context problem.

## Work is refused with 503

Every refusal carries a `code`:

| Code | Cause | What to do |
|---|---|---|
| `draining` | An operator armed the drain for a restart. New spawns, messages, forks, loops, and seat joins are refused. Reads, interrupts, and approvals still work | Wait. The response includes `retry-after: 30` and the active thread ids. `GET /drain` shows the count |
| `draining` after a failed bounce | A reload stopped before it disarmed the drain | Run `curl -X DELETE http://127.0.0.1:4741/drain`. A `409` means an owner has claimed the drain: `GET /drain` names it, and only that owner can release it with `DELETE /drain/claim`, or a restart clears the drain |
| `spawns-paused` | The data volume is under 10 GB free, so the keeper paused new spawns | Free disk space. Spawns resume when the volume is back over 30 GB. An operator can clear the pause sooner with `POST /keeper/spawn-pause/clear` and a `reason`. `GET /keeper/spawn-pause` shows the state. Running threads are not touched |
| `thread-degraded` | The store could not save a body from an earlier turn, so the log has a gap and the thread refuses further turns. The response names the missing body (`key`, `index`, `tag`) | Fix the cause first. The mark survives a restart. To accept the gap and reopen the thread, call `POST /threads/:id/recover` with `{"acknowledgeHistoryGap": true, "reason": "..."}`. Recovery does not restore the missing history |

A `429` with code `session-cap` means the runtime holds as many live sessions as it allows, and none has been idle long enough to be evicted. Retry after the `retry-after` interval.

The drain is held in memory. A restart clears it, so a restart also fixes a drain left armed. The `/drain` routes accept calls from anything on loopback.

## Lost turns after a restart

A restart ends any turn that was running. On boot, the runtime finds each thread that was mid-turn and writes a system error message that begins `This turn was lost to a host restart`.

The message marks the turn as failed, and nothing retries it. Use Resume, or send a new message, to run it again. A thread that still shows as running with no engine behind it, and has no such message, has not been swept yet; the keeper's lost-turn rule flags those.

## Update a running installation

Use the reload script rather than restarting the service by hand:

```sh
scripts/bounce-runtime.sh "reason for the reload"
```

The script does the following in order:

1. Fast-forward pulls the checkout and runs `bun install --frozen-lockfile`.
2. Recreates the credential proxy from the updated config, unless `BOUNCE_SKIP_PROXY=1` is set.
3. Arms the drain, so new work is refused and running turns finish.
4. Waits until no thread is mid-turn. The wait is `BOUNCE_WAIT_SECONDS`, 5400 by default. If the wait runs out, the drain is disarmed and nothing restarts.
5. Restarts the `com.condensate.agents-host` launch agent and waits for the new process to answer.

The update steps run before the drain is armed, so the runtime keeps serving while they run. Nothing they do takes effect until the restart.

Back up the runtime's state before you change it. It lives in three places:

| Data | Location | Notes |
|---|---|---|
| Thread records and events | `threads.db` in the data directory (`CONDENSATE_AGENTS_DATA_DIR`, default `~/.condensate/agents-host`) | SQLite in WAL mode |
| Compacted event bodies | `threads-archive.db` beside `threads.db`, when present | Bodies moved out of the main log live only here. Back it up with `.backup` too |
| Large event bodies | `blobs/` beside `threads.db` | Bodies over 64 KiB (`CONDENSATE_AGENTS_BLOB_THRESHOLD`) are stored here. The database keeps only their hashes, so back up both |
| Engine resume state | `~/.condensate/engine` by default | Lets a resumed turn continue from its last state |

Use SQLite's online backup while the runtime runs:

```sh
sqlite3 ~/.condensate/agents-host/threads.db ".backup '/path/to/backup/threads.db'"
```

Copying `threads.db` alone while WAL is active is not a complete backup. Back up `threads-archive.db`, `blobs/`, and the engine state with it.

This page does not restart or migrate an existing deployment. Use the reload script for that, and only in a quiet window or behind the drain.

## Where to look next

| Question | Endpoint |
|---|---|
| Is the runtime up, and which build is it? | `GET /health` |
| What sessions are running right now? | `GET /live` |
| Is a drain armed? | `GET /drain` |
| Which failures repeat across threads? | `GET /incidents?since=<epoch ms>` |
| What happened in a thread? | `GET /threads/:id/tail` |
| Is a turn running right now? | `GET /threads/:id/turn` |