Troubleshooting

This page helps you find which part is failing (browser, application server, runtime, engine or provider) and fix it. Most checks only read state through the runtime's loopback HTTP API. Find the failing part before you restart anything, because a restart interrupts active tasks and hides the cause.

Find the boundary

The page does not load

Confirm the application's startup command and port. Opening index.html from disk does not start its backend.

The Dex app forwards its /runtime/* requests to CONDENSATE_AGENTS_URL. Setting that variable alone does not fix file paths. If the runtime runs on another machine or in a container, set DEX_APP_GTR_MCP to the path of the hub's MCP entry (it ends in apps/gt-review/src/mcp.ts) as that host sees it.

Dojo runs in Docker Compose and publishes port 4800 on loopback. If the page does not load, check that the container is up before you look at the app.

The page loads but tasks do not

curl -i 'http://127.0.0.1:4741/health'
curl -i 'http://127.0.0.1:4741/threads?limit=5'
curl -i http://127.0.0.1:4741/workspaces
curl -i http://127.0.0.1:4741/live

/health returns the build identity and any open retain alert. /live lists the sessions the runtime holds and any account pool that is out of quota right now.

Distinguish a successful empty page from a transport error. A list can look empty or incomplete for ordinary reasons:

• A thread list returns 50 threads by default and 200 at most. Follow nextAfter while hasMore is true.
• Retired threads are hidden unless you pass archived=archived or archived=all.
• seats=open, the default, hides finished reviewer and fixer seats. Pass seats=all to see them.
• A 429 with code thread-list-rate-limited means the first page was requested too often. Follow /workspaces/:name/changes and read only the threads that changed.

Also check CONDENSATE_AGENTS_URL on the application server, the selected workspace, and pagination. A loaded page is not a complete inventory.

A model turn fails

Read the thread's events with GET /threads/:id/tail, and check whether a turn is still running with GET /threads/:id/turn. Each failure is recorded with a class. The class tells you whether the runtime tried another model and what to change:

To see which failures repeat across threads, use GET /incidents?since=<epoch ms>. It groups failed turns by class and by message fingerprint, and it counts the threads each one hit. The window covers the last 24 hours and resets when the runtime restarts.

A working list endpoint does not prove that model credentials work. Verify the engine and model pair, the account binding, the workspace policy, and provider availability.

The credential proxy is on by default, so sessions hold only placeholder tokens and the proxy adds the real credential on the way out. An auth error inside a session can then mean the proxy cannot reach the provider. Check CONDENSATE_PROXY_URL and CONDENSATE_PROXY_CA in the runtime's launch agent environment, and that the proxy container is running.

Streaming pauses

Check the durable event log and the current turn status before you assume the task stopped.

• GET /threads/:id/stream sends live deltas only. Nothing is replayed, and a slow reader loses its oldest frames. Treat deltas as a preview. A 429 here means the thread already has as many stream subscribers as it allows. Close idle tabs, or read follow instead.
• To recover after a gap, read GET /threads/:id/tail?after=<last seq> first, then reopen the stream. GET /threads/:id/follow?after=<seq> also accepts a Last-Event-ID header as its cursor.
• tail?wait=<ms> holds an empty page open for up to 30 seconds, waiting for new events.
• An idle stream sends a heartbeat every 15 seconds by default. A proxy that buffers responses, or closes idle connections early, looks like a stall. The runtime sends cache-control: no-cache, no-transform and x-accel-buffering: no. Keep the proxy from overriding them.

Context reductions repeat

The runtime shrinks a thread's context when it nears the model's limit. It records each step in the event log:

When a window fills and the runtime has to recover, the log gets a system message that starts window full, recovering (<stage>). The stage is resume-skipped, recut, or restarted. It is not an error, and the turn stays open.

When a reduction is refused, the runtime says why:

• WindowThrash: reductions ran too often in recent turns, so the runtime stopped reducing and recorded a system line.
• WindowNoHeadroom: the system prompt and skills leave no usable room under the ceiling.
• WindowBlocked: a delegated child thread has not joined yet, so the cut waits until the child merges.

Summaries are written by the curator model, with a fallback. Set them with CONDENSATE_CURATOR_MODEL and CONDENSATE_CURATOR_FALLBACK_MODEL. A summary record with no summaryModel means no model wrote it, which happens for the cheap kinds and for a forced window whose curator never answered.

Inspect the window records and the curator's answers first. Keep the durable log. Deleting history is not a diagnosis of a model-facing context problem.

Work is refused with 503

Every refusal carries a code:

A 429 with code session-cap means the runtime holds as many live sessions as it allows, and none has been idle long enough to be evicted. Retry after the retry-after interval.

The drain is held in memory. A restart clears it, so a restart also fixes a drain left armed. The /drain routes accept calls from anything on loopback.

Lost turns after a restart

A restart ends any turn that was running. On boot, the runtime finds each thread that was mid-turn and writes a system error message that begins This turn was lost to a host restart.

The message marks the turn as failed, and nothing retries it. Use Resume, or send a new message, to run it again. A thread that still shows as running with no engine behind it, and has no such message, has not been swept yet; the keeper's lost-turn rule flags those.

Update a running installation

Use the reload script rather than restarting the service by hand:

scripts/bounce-runtime.sh "reason for the reload"

The script does the following in order:

1. Fast-forward pulls the checkout and runs bun install --frozen-lockfile.
2. Recreates the credential proxy from the updated config, unless BOUNCE_SKIP_PROXY=1 is set.
3. Arms the drain, so new work is refused and running turns finish.
4. Waits until no thread is mid-turn. The wait is BOUNCE_WAIT_SECONDS, 5400 by default. If the wait runs out, the drain is disarmed and nothing restarts.
5. Restarts the com.condensate.agents-host launch agent and waits for the new process to answer.

The update steps run before the drain is armed, so the runtime keeps serving while they run. Nothing they do takes effect until the restart.

Back up the runtime's state before you change it. It lives in three places:

Use SQLite's online backup while the runtime runs:

sqlite3 ~/.condensate/agents-host/threads.db ".backup '/path/to/backup/threads.db'"

Copying threads.db alone while WAL is active is not a complete backup. Back up threads-archive.db, blobs/, and the engine state with it.

This page does not restart or migrate an existing deployment. Use the reload script for that, and only in a quiet window or behind the drain.

Where to look next