Fouria Operator Guide
On this page
1. Welcome
Fouria is a Cloudflare Worker that runs OpenClaw (the AI agent gateway) inside a Cloudflare Sandbox container. Each customer gets an isolated container; the Worker starts it, proxies chat traffic to it, snapshots its state to R2, and keeps it healthy.
Most of the time the system runs itself. This guide is for the moments when it does not: when a container is stuck, a backup will not restore, the AI gateway rejects requests, or the gateway will not start.
The three surfaces you operate
| Surface | Entry point | When to use |
|---|---|---|
| Health snapshot | GET /api/health (and ?passive=1) | Is the gateway up? What is its status? |
| Debug routes | /debug/* (only when DEBUG_ROUTES=true) | Deep inspection + emergency actions (stop, destroy, rescue, restart) |
| Diagnostics module | src/diagnostics.ts | The code that powers container rescue; gateway restart lives in src/gateway/process.ts + src/routes/debug.ts |
Mental model: the lifecycle
Request / cron / health probe
│
▼
ensureGateway() ── finds or starts the container
│
▼
start-openclaw.sh (onboard → moltlazy patch → plugins → gateway run)
│
▼
OpenClaw gateway (port 18789, Control UI + WebSocket)
│
▼
createSnapshot() → R2 (state preserved between restarts)
When any link in this chain breaks, the tools in this guide are your recovery path.
2. Getting Started
2.1 Prerequisites
You should be comfortable with:
- Reading a JSON health response (
curl … | jq). - Cloudflare concepts: Workers, Containers, R2, AI Gateway.
- That the gateway runs inside a container — most “server is down” issues are really “container is down or the gateway process inside it is down”.
2.2 Enabling diagnostics — how to turn on debug options
Debug tooling is off by default in production for safety. There are two layers:
Layer A — DEBUG_ROUTES (the /debug/* HTTP surface)
The /debug/* routes are gated by the DEBUG_ROUTES environment variable
(apps/fouria/src/index.ts:379, src/types.ts:34).
# In .dev.vars (local) or as a Worker var (production)
DEBUG_ROUTES=true
DEBUG_ROUTES !== 'true'→ every/debug/*call returns 404 immediately (index.ts:292).- Setting it is non-destructive: it only exposes the inspection/action endpoints; it does not change runtime behavior.
⚠️ Never leave
DEBUG_ROUTES=trueon a tenant-facing deployment. It exposes/debug/gateway-token,/debug/stop-gateway, and/debug/destroy-container. Use it on operator/ROOT deployments only.
Layer B — passive health probes (no extra config)
The public health endpoint GET /api/health is always on. Append ?passive=1 to inspect
a down instance without warming/cold-starting it (routes/public.ts:163):
curl "https://<worker>/api/health?passive=1" | jq .
Passive probes use a short 3-second bound (PASSIVE_PROBE_TIMEOUT_MS, config.ts:24) and
never trigger ensureGateway, so a dead container fails fast instead of spinning up for
minutes.
2.3 Your first diagnostic call
# 1. Is the gateway healthy? (fast, no warm-up)
curl "https://<worker>/api/health?passive=1" | jq '.status, .degraded, .containerError'
# 2. Full active snapshot (warms the container, schedules a backup)
curl "https://<worker>/api/health" | jq .
# 3. Deep dive (requires DEBUG_ROUTES=true)
curl "https://<worker>/debug/version" | jq .
curl "https://<worker>/debug/processes?logs=true" | jq .
Read the status field — it is one of running | starting | stopped
(health-probes.ts:67). Read degraded — when true, a permanent container error
(e.g. deleted image) requires a rescue (see §6.1).
2.4 Manual snapshots
POST /api/admin/storage/sync queues a durable snapshot by default when the
ACTIVITY Durable Object is configured. It returns 202 with a jobId (or 409
when a backup/restore job already owns the slot). Use ?inline=1 only when the
caller needs the completed snapshot response immediately, such as an E2E checkpoint;
?async=1 retains the accepted-then-background behavior. Without ACTIVITY, the
endpoint falls back to inline work.
The shared R2 lease is valid for 12 minutes (STEP_DEADLINE_MS), renewed every 3
minutes and at slow snapshot/restore phase boundaries. A dead lease can consequently
take longer to recover than under the former 60-second TTL; this favors preventing
false expiry while a durable archive step is active. Watch for structured
persistence.lock.renewal_late and persistence.lock.renewal_failed events.
2.5 Rollout park (why admin writes answer 409 during an upgrade)
A container replacement always starts from a fresh disk, so the replacement
restores the tenant from the checkpoint taken just before the rollout. To make
that lossless the control plane parks the tenant first
(POST /control → X-FourIA-Control: checkpoint-park): the gateway is drained,
the stopped filesystem is snapshotted, and the tenant stays parked — every
mutating /api/admin/* or /debug/* request answers
409 { code: "TENANT_PARKED" }, and container exec is refused outright,
including background/spawned tasks. Read-only requests keep working.
The condition is durable (maintenance/replacement.json in the tenant’s
BACKUP_BUCKET) and expires after 15 minutes, so a controller that dies mid-park
cannot wedge a tenant. A normal upgrade releases it at the :resume step
(X-FourIA-Control: resume, issued right after the container is replaced); the
replacement container then restores the checkpoint and boots the gateway. If an
operator sees 409 TENANT_PARKED long after an upgrade finished:
POST /controlwithX-FourIA-Control: resume(Lerma does this automatically on success and on failure); or- wait out the 15-minute window —
getMaintenancediscards it andensureGatewayfrees itself; or - as a last resort, delete
maintenance/replacement.jsonfrom the tenant’sBACKUP_BUCKET(only after confirming no rollout/snapshot is in flight).
Do not delete it while a checkpoint is running: the park is what stops a write from landing after the archive was produced.
While parked you will also see:
- Live agent sessions in
read-only. The park switches them before it stops the gateway and waits (up to two minutes) for their current turn to finish, so a turn is not interrupted mid-write. Their previous mode is recorded inmaintenance/session-modes.jsonand put back on resume / after the restored gateway is up. New sessions are unaffected. If a session is stillread-onlyafter an upgrade, check that file first — a failed restore keeps it for the next boot (it is applied bylaunchRestoredState). - Telegram replies from the ingress. A message sent during the park gets the “I’m being upgraded, please resend” notice from the ingress itself (no AI turn, no gateway), and the update is acked rather than queued.
2.6 Sandbox v1 migration gate (staging only)
The Sandbox SDK 1.0 cutover starts with an authority gate (contract C1/C2/C10).
POST /control with X-FourIA-Control: migrate is accepted only when the
trusted WORKER_PLATFORM binding is exactly staging and the tenant slug
resolves from trusted Worker env and the slug is not the reserved client.
Refusals happen before the sandbox stub is acquired:
| Condition | Response |
|---|---|
WORKER_PLATFORM absent/not staging | 403 { code: "MIGRATION_PLATFORM_FORBIDDEN" } |
tenant slug unresolved (default, missing) | 409 { code: "MIGRATION_SLUG_UNRESOLVED" } |
reserved client tenant | 409 { code: "MIGRATION_RESERVED_CLIENT" } |
| allowed staging tenant | 202 { state: "queued" | "running" }, 200 { state: "committed", recordId }, or 502 { code: "SANDBOX_V1_CONVERSION_FAILED" } |
migrate is queue-only: it enqueues a durable job on the SandboxV1 Durable
Object and answers immediately — the conversion (legacy archive → v1 container →
DirectoryBackup → catalog + pointer) runs on the DO alarm(), so a tenant is
never converted by a request that could be cancelled. Poll with the separate
read-only action X-FourIA-Control: migration-status (Dispatch Worker:
POST /_control/:slug/migration-status), which only reads
v1ConversionState(): it never enqueues, never starts work and never wakes the
container, answers 200 { state: "absent" | "queued" | "running" | "committed" | "failed" }
and never 202. A legacy tenant reports absent without addressing a v1 DO.
Every other control action (freeze, unfreeze, backup, checkpoint,
checkpoint-park, resume) is not gated and keeps its existing behavior on
every platform, including client. Operators drive a conversion through Lerma
(POST/GET /api/provisioning/migrate/:slug, or
mix lerma.sandbox_v1.migrate), never by hand. WORKER_PLATFORM is provisioned
by Lerma from its own cf_environment; never derive it from the request host or
a header.
3. Operation Guidelines — Debug Options & How to Activate Them
3.1 The debug endpoint catalog
All under /debug/*, active only when DEBUG_ROUTES=true. Each maps to logic in
src/diagnostics.ts or gateway/process.ts.
| Method & Path | What it does | Backed by |
|---|---|---|
GET /debug/version | FourIA Worker version + git commit, plus OpenClaw + Node versions in the container | __FOURIA_VERSION__ / __FOURIA_COMMIT__ + runShell (sandbox.exec) |
GET /debug/processes?logs=true | List container processes (+ logs); returns 503 { status: "platform_updating" } while the sandbox runtime is being updated (deploy) | sandbox.listProcesses |
GET /debug/gateway-api | Probe the OpenClaw HTTP API | GATEWAY_API_TIMEOUT_MS |
GET /debug/cli | Run an OpenClaw CLI command | startShell (sandbox.exec) |
GET /debug/logs | Container stdout logs | sandbox |
GET /debug/env | Sanitized environment (secrets masked) | maskEnvValues() |
GET /debug/container-config | OpenClaw config from inside container | sandbox.readFile |
GET /debug/gateway-logs | Gateway stdout (/tmp/gateway-stdout.log) | sandbox.exec |
POST /debug/sqlite-integrity | Read-only PRAGMA integrity_check over every *.sqlite under /root/openclaw/.openclaw (e.g. state/openclaw.sqlite after a restore). Opens each file readOnly; never checkpoints, repairs, or removes WAL/SHM sidecars. Returns ok, per-file messages, and sidecars. Results are a snapshot while the gateway is writing — stop the gateway first for an authoritative check. Runs as a durable maintenance job (ActivityScheduler DO) because a large check outlives a Worker request; returns 202 { accepted, jobId } and is polled via GET /debug/sqlite-integrity/jobs/:id or streamed over GET /debug/sqlite-integrity/jobs/:id/ws (runs inline when the ACTIVITY binding is absent) | sandbox.exec (node:sqlite, readOnly) |
GET /debug/gateway-token | The gateway auth token | resolveSecret |
GET /debug/gateway-lock | OpenClaw native lock state | gateway/lock.ts |
POST /debug/stop-gateway | Kill gateway without restarting | killGateway() |
POST /debug/restart-openclaw | Hard gateway restart (kill → clear startup cache → flag R2 restore → start-openclaw.sh reprovision) | killGateway() + resetStartupCache() + signalRestoreNeeded() |
POST /debug/reprovision | Force a container reprovision (checkpoint → 409 unless restorable backup or force → destroy → clear startup/persistence caches → reset orphan streak); never touches the gateway | createSnapshot() + getBackupHistory() + sandbox.destroy() |
POST /debug/destroy-container | Destroy the container entirely | sandbox.destroy() |
POST /debug/sleep | Force the idle-sleep deprovision path (E2E seam): enqueue the durable backup-then-stop job the idle hook enqueues, so a container is never slept without a committed snapshot. 202 {jobId} | getActivityScheduler() → enqueueBackupJob({operation:'deprovision',mode:'sleep'}) |
POST /debug/rescue-container | Diagnose + rescue a stuck container | rescueContainer() |
POST /debug/backup/:mode | Trigger a checkpoint on demand (mode = sync | async): sync blocks and returns the result (502 on failure); async acknowledges 202 and runs the snapshot via waitUntil. Idle container is skipped, never woken | ensureGateway() + scheduleBackupIfNeeded() |
POST /debug/r2-put?key= | Write raw content to R2 (tests) | BACKUP_BUCKET.put |
Secret safety:
GET /debug/envruns every value throughmaskEnvValues()(diagnostics.ts:31). Any key matchingtoken,api_key,password,secret,credential,auth, orprivate_keyis replaced with••••••••. Never paste raw/debug/envoutput into tickets.
3.2 Reading the gateway lock (the most common “ghost” outage)
OpenClaw enforces single-instance ownership via three lock layers — a state lock, a
config lock, and an exclusive TCP bind on port 18789 (gateway/lock.ts:1,
docs.openclaw.ai/gateway/gateway-lock). A stale lock is the #1 cause of
"gateway already running (pid …)" / EADDRINUSE on restart.
curl "https://<worker>/debug/gateway-lock" | jq '.state'
state is one of (lock.ts:27):
| State | Meaning | Operator action |
|---|---|---|
free | No lock held — safe to start | None |
held-live | A live gateway owns the port | Adopt it; do not kill |
stale-reclaimable | Lock files present but owner PID gone | Auto-reclaimed on next start; restart is safe |
conflict | Lock error seen in the log | Wait/restart; avoid parallel starts |
unknown | Probe failed (container off) | Check container health first |
The hard restart path checkpoints state before killing the gateway, then waits for the next activation to restore that committed snapshot. OpenClaw’s native lock protects gateway ownership, but does not serialize filesystem mutations performed by onboarding, plugins, configuration, or backup operations.
3.3 Emergency actions — when to use each
| Action | Use when | Risk |
|---|---|---|
POST /debug/stop-gateway | Gateway wedged but config is fine; you want to manually start fresh | Low — does not destroy data |
POST /debug/restart-openclaw | Config valid but gateway process dead; you want a clean in-place restart | Low |
POST /debug/reprovision | Container lost/orphaned (e.g. destroyed from the dashboard) but the DO is alive; gateway RPCs are timing out | Low–Medium — checkpoints first, 409 if no restorable backup (override force); R2 restore keeps state |
POST /debug/destroy-container | You want the platform to provision a brand-new container (fresh state) | High — loses in-memory + unsnapshotted state |
POST /debug/rescue-container | Container is in Error mode / image deleted (see §6.1) | Medium — destroys stuck instance, keeps R2 backups |
3.4 The two diagnostic operations you will script most
Both live in src/diagnostics.ts and accept a dryRun option — always run dry first.
POST /debug/restart-openclaw — checkpointed hard gateway restart
Exposed as the diagnostic dashboard button (ROOT, DEBUG_ROUTES=true). It is the hard
restart path (the opposite of the Admin API’s soft reload):
- Find the live gateway process via
findExistingGatewayProcess(sandbox, { probeLive: true }). - Refuse with
409if the tenant is busy: a restore is in flight (or its outcome is still unknown after a timeout), or a maintenance window is open. Nothing is stopped. - Under the R2 lease, open a
restartmaintenance window andkillGateway(sandbox)—SIGTERMwithGATEWAY_KILL_GRACE_MS(5s) grace,SIGKILLonly if still active; uses the[o]penclawpkill bracket trick (no blanket"openclaw"pkill, no manual lock-file deletion — respects OpenClaw’s own single-instance lock). The gateway is stopped before the archive is cut, so the checkpoint is consistent. - Create and commit a snapshot of
/home/openclaw. If a restore is pending over a committed recovery point, the checkpoint is refused (409) before the gateway is stopped. resetStartupCache()— clears the startup health cache.signalRestoreNeeded(BACKUP_BUCKET)— flags R2 restore so the next request reprovisions a fresh gateway viastart-openclaw.sh. The maintenance window closes only after this marker is set, so no request can relaunch the gateway on pre-restart state in between.
If the checkpoint fails before the gateway is stopped, the gateway is not touched. If it fails
after the stop, the gateway is restarted in place on the untouched filesystem and no restore is
armed. There is no direct binary launch; reprovisioning always goes through start-openclaw.sh.
The Admin API
POST /api/admin/gateway/restartis the soft reload: it sendsSIGUSR2to the liveopenclaw gateway runprocess (softReloadGateway), thenresetStartupCache(). When no live gateway is found it starts the gateway through the normal cold start (ensureGateway) and does not armrestore-needed: the live filesystem is kept, and the restore barrier still restores a container that has no state. It returns409while a maintenance window is open.
SIGUSR1is not a restart signal: in current OpenClaw it starts Node’s inspector on127.0.0.1:9229and leaves the gateway running. The signal must reach the gateway server, not theopenclaw gateway runlauncher: the launcher merely supervises the server, soSIGUSR2to the launcher kills it and orphans the server (PPID 1) without restarting anything, while the endpoint still answersreloaded: true.buildSoftReloadScripttherefore matches the server’s exactcomm(openclaw-gatewa, Linux’s 15-char truncation). Inside the container OpenClaw always takes the in-process restart path —isContainerEnvironment()reportsdisabled/unmanagedeven withoutOPENCLAW_NO_RESPAWN=1— so soft reload never spawns an orphan either way. See../../docs/plans/sandbox-memory-usage.mdfor the #463 decision record.
POST /debug/reprovision — force a container reprovision
The escape hatch for an orphaned/dead container whose Sandbox Durable Object is still alive — most commonly after the instance was destroyed from the Cloudflare Containers dashboard, which leaves the DO pointing at a placement that no longer exists.
- Busy guard — refused with
409while a restore is in flight (or its outcome is still unknown) or a maintenance window is open. Unlikeforce, this cannot be overridden: destroying the container mid-apply leaves a half-replaced filesystem. Re-checked after the checkpoint, in case a restore started while it ran. - Checkpoint first (
createSnapshot) — only whenisRuntimeActive()reports a live runtime (an idle container is never woken just to checkpoint it), and bounded byREPROVISION_CHECKPOINT_TIMEOUT_MSso an orphaned container’s hanging RPC cannot block the route. If the container is gone or the snapshot times out, the route falls back to an existing restorable backup (verified via HEAD on the newest catalog entries’meta.json, not just the entry count). A snapshot abandoned by the timeout can finish after the destroy, but it only archives state that already existed, so it cannot corrupt anything. - Fail closed: if neither the checkpoint nor an existing backup is restorable, the route
returns
409and does not destroy the container. Destroying it would permanently lose tenant state. clearPersistenceCache()/resetStartupCache()— drop the per-isolate “already restored / startup ok” facts about the old container.clearProvisioningFailures(sandbox)— reset the orphaned-instance failure streak.sandbox.destroy()— tear down the instance so the next RPC provisions a fresh one.
Unlike restart-openclaw, this never talks to the gateway, so it still works when the container
is completely gone and gateway RPCs are timing out.
curl -X POST "https://<worker>/debug/reprovision" | jq .
When the container is already gone and no backup exists (state is unrecoverable), pass the explicit override:
curl -X POST "https://<worker>/debug/reprovision" \
-H 'content-type: application/json' -d '{"force":true}' | jq .rescueContainer(sandbox, env, options) — diagnostics.ts:296
Recovers a container stuck in Error mode (e.g. the fouria:unstable-<sha> image it
references was deleted by the stale-image cleanup cron). Sequence:
- Probe container health (
probeContainerHealth, shortPASSIVE_PROBE_TIMEOUT_MSbound). Healthy →{ action: 'none', ok: true }. - Repair image via the Cloudflare Containers API (
createContainersApiClient→rolloutImagewithstrategy: 'full_auto') if credentials are present and the current image ≠ expected (diagnostics.ts:339). - Destroy the stuck container (
sandbox.destroy()) so the platform provisions a fresh instance against the repaired image.
# Diagnose first (dry run — does nothing)
curl -X POST "https://<worker>/debug/rescue-container" \
-H 'content-type: application/json' -d '{"dryRun":true}' | jq .
# Actually rescue (requires CLOUDFLARE_API_TOKEN + CONTAINER_APPLICATION_ID)
curl -X POST "https://<worker>/debug/rescue-container" \
-H 'content-type: application/json' -d '{"dryRun":false}' | jq .
The result includes action ∈
none | destroy | repair-image | repair-image-and-destroy | error, a diagnosis
(health + errorClass), and the image report (current, expected, repaired,
rolloutId).
Error classification (what “permanent” vs “transient” means)
classifyContainerError() (diagnostics.ts:166) decides whether retrying helps:
- Permanent (no point retrying):
no such image,no matching app,no namespace configured,did not call start,out of memory,resource exhaustion,pid limit. → a rescue (image repair + destroy) is required. - Transient (recovers on its own):
connection refused,econnrefused,etimedout,timed out,not mapped,no container instance available. → just wait / warm up. - unknown → treat as transient unless health stays red.
This classification drives the degraded flag on the health snapshot (health-probes.ts:281):
a permanent error is what flips degraded:true and tells you a rescue is needed.
4. Working with Backups
State lives in the container at /home/openclaw and is preserved as squashfs snapshots
in R2 (persistence.ts). The gateway restarts from the latest snapshot on boot.
4.1 The persistence model
| Concept | Where | Notes |
|---|---|---|
| Backup store handle | backup-handle.json in R2 | Tracks up to maxVersions (default 3) |
| Backup objects | backups/<id>/data.sqsh[.enc], meta.json | .enc when encrypted at rest |
| Config versions | configs/<timestamp>.json | Kept 30 days (CONFIG_RETENTION_DAYS) |
| Restore marker | restore-needed key | Durable restore-in-progress/retry latch; absent while idle or after success |
| Lock | backup-lock (60s TTL, 20s heartbeat) | Shared cross-isolate lease: serializes manual + scheduled snapshots |
4.2 Encryption at rest (F4E1)
When BACKUP_ENCRYPTION_KEY is set, every snapshot is post-encrypted with AES-256-GCM
before it lands in R2 (persistence.ts:405, createSnapshot). Config versions under
configs/ are encrypted with customMetadata.encrypted='true'.
# Generate a key (store it as a secret_text binding, NEVER in .dev.vars for prod)
openssl rand -base64 32
BACKUP_ENCRYPTION_KEYunset → backups are stored UNENCRYPTED (warned in logs).BACKUP_ENCRYPTION_REQUIRED=true→ backups fail closed if the key is missing (persistence.ts:419). Recommended in production.
Legacy plaintext backups remain restorable; decryption is transient and deleted in a
finallyblock (persistence.ts:223). Seedocs/plans/r2-encryption-at-rest.md.
4.3 How backups happen
- Dispatch cron (running instances only) — the Dispatch Worker’s cron (every
15 min production / hourly staging) fans a
backupcontrol action out to every non-frozen tenant. The tenant assertsisRuntimeActive()(a non-waking bridge read) and skips a sleeping container — an idle instance has no workload and no filesystem changes to capture, and waking it would restart billing. This is the backstop for tenants the lermaHealthMonitordoes not poll. Before snapshotting, the persistence layer refuses to run while arestore-neededmarker is present, rechecked under the shared lease, so a wake that lands mid-restore cannot snapshot and promote a partial filesystem over the committed recovery point. - On hard restart —
POST /debug/restart-openclawfirst creates a checkpoint, then clears the startup cache and flags R2 restore. - Manual —
POST /api/admin/storage/synccallscreateSnapshot()(explicit admin action). - Async admin —
POST /api/admin/storage/sync?async=1andPOST /api/admin/storage/restore/:id?async=1commit a durable job to theActivitySchedulerDurable Object and return202 { accepted: true, jobId }. A DO alarm runs the job and retries it with a bounded lease, so the workflow survives the request ending, isolate eviction, and Worker redeploys (the oldwaitUntil()path was only guaranteed ~30s after the response). PollGET /api/admin/storage/jobs/:idforqueued | running | retrying | succeeded | failed | skipped. - Health probes are observability-only:
GET /api/healthNEVER writes the backup catalog. POST /debug/r2-putonly writes test objects; it does not create a snapshot.
Stable recovery point. A snapshot becomes the recovery point only once its
archive, optional F4E1 encryption, and catalog entry are all committed
(stable-backup.json). A cold restore prefers that pointer and refuses to
restore a newer archive whose promotion was interrupted, and retention always
keeps it — so a snapshot that failed partway can never evict the last known-good
recovery point. GET /api/admin/storage/backups reports it as
stableBackupId. This is a checkpoint guarantee, not zero-RPO replication:
writes acknowledged after the last committed snapshot are outside it (see
issue #377). The pointer only advances forward in time by default, so a stale
snapshot can never demote a newer committed one; a targeted restore is the one
exception (see §4.4).
Two guards keep the pointer safe in the other direction:
- Restore pending. While a
restore-neededmarker is present and a committed recovery point exists, a snapshot is still archived but does not move the pointer, and a pre-replacement checkpoint is refused (409). The live filesystem is not authoritative until the pending restore has applied. - Restore-only hold. A targeted restore with
setCurrent=0(§4.4) writes arecovery-hold.jsonmarker. Scheduled snapshots keep archiving but do not move the pointer onto the restored state, so the restore is not silently undone within one backup interval. Any explicit snapshot (manual, checkpoint, hard restart) or a restore that sets the recovery point clears the hold.GET /api/admin/storage/backupsreports it asrecoveryHeld.
All three snapshot paths share one tenant-scoped R2 backup-lock lease, so only
one isolate ever stages the fixed legacy stage directory (.fouria-legacy-stage) or
calls sandbox.createBackup at a time. The lease has a 60s TTL and is renewed every
20s while a snapshot is in flight, and it is released by compare-and-swap to a
tombstone (never by deleting the lock object, which would race a concurrent takeover).
A live lease left behind when an isolate is killed is reclaimed after the 60s TTL;
the tombstone ({acquired:0}) is the normal released state, not a stuck lock.
Crash-consistency (issue #344). A Worker isolate can be killed by its CPU budget after
sandbox.createBackup()has uploadedbackups/<id>/…but before the handle is written — leaving an orphaned, apparently-unrecoverable snapshot and a missingbackup-handle.json. Two guards prevent/repair this:
- The catalog entry is committed immediately after the archive is created (after encryption, before the slower config-version capture and retention prune), so an interrupted run still leaves a catalogued, restorable archive.
- The catalog is reconciled explicitly from
backups/*/meta.jsonbyreconcileBackupCatalog(the durablerepairstep,POST /api/admin/storage/reconcile,restoreIfNeeded,scheduleBackupIfNeeded, and the reprovision guard) whenever an archive is missing from the handle, so pre-existing orphans self-heal.getBackupHistorystays a pure read: a dashboard poll never lists, prunes, sweeps, or writes. Retention is enforced on that reconcile path too, so an orphan adopted after a run of failed snapshots cannot push the catalog pastmaxVersions.The scheduled
backupcontrol action additionally runs under a dedicated CPU budget (the Dispatch Worker’sBACKUP_CONTROL_CPU_MS) so the full orchestration can finish instead of being killed mid-flight. Freeze now enqueues a durable backup-then-destroy job and acknowledges immediately (202 { jobId }) under the default budget; the caller pollsGET /api/health/backup/:jobId(through the health Access tunnel) until the job reaches a terminal status instead of holding the request open. Unfreeze keeps the default.Legacy-stage timeout (issue #344 follow-up). The pre-snapshot legacy-state staging shell can exceed its bound on a large tenant (observed killing the
tarjeterostage at exactly 300s). The bound is nowLEGACY_STAGE_TIMEOUT_MS(default600000, i.e. 10 min), the idempotent stage is retried once on a transient timeout/transport drop, and a stage failure logs its exit code, timeout flag and bounded stderr. The Sandbox process is also reported to Worker Logs with itsprocessIdso the failure can be joined to the Containersprocess.completeevent.The stage wait itself is transport-resilient (
runShellResilient): a Sandbox DO↔container drop mid-stage reattaches to the process by id and resumes (bounded by the stage deadline) instead of ending the request. The DispatchbackupCPU budget (BACKUP_CONTROL_CPU_MS) is aligned to the same 10 minute bound.Asynchronous control acknowledgment (issue #344). Even with the larger budget, holding the
POST /controlrequest open for the whole snapshot outlived the dispatch/Worker request lifecycle, which cut the tenant request right aftercreateBackupuploaded the archive and before the catalog write. The scheduledbackupaction now acknowledges immediately (202 { ok: true, accepted: true, jobId }) and enqueues the snapshot as a durable job on the per-tenantActivitySchedulerDO, so the dispatch cron and operators never block on it. AwaitUntiltask is cancelled ~30s after the response (platform cap) and was observed dropping a snapshot that was still cold-booting, so the backup must not rely on it. The crash-consistent catalog commit + reconciliation above keep the archive durable if the isolate is still replaced mid-snapshot. To trigger either mode on demand (ROOT,DEBUG_ROUTES=true), usePOST /debug/backup/asyncorPOST /debug/backup/sync.
The two paths behave differently when the lease is busy:
- Scheduled acquisition is non-blocking: if another isolate holds the lease, this tick writes no backup and returns (it does not wait).
- Manual (
POST /api/admin/storage/sync) and the hard-restart checkpoint wait for the lease with bounded exponential backoff (up to 120s), then fail with an error rather than silently reporting success. A contended manual sync can therefore take up to 120s and then return an error; it never bypasses the lock.
A lease that cannot be proven free — held live, malformed, or unreadable — is treated fail-closed: manual callers keep waiting and then time out, and scheduled callers skip the tick. (Contention is bounded, not instant.)
Automatic backup trigger state machine
There is one automatic snapshot trigger: the Dispatch Worker cron, which
fans a backup control action out to every non-frozen tenant. The tenant only
proceeds when isRuntimeActive() reports a live runtime — an idle container is
never woken — after which the interval/lock gates below decide whether an object
lands in R2. Health probes play no part: GET /api/health is observability-only
and must not alter the catalog.
stateDiagram-v2
direction TB
[*] --> Idle
state "Idle (no snapshot scheduled)" as Idle
state "Runtime active? (isRuntimeActive)" as RuntimeGate
state "Interval elapsed?" as IntervalGate
state "backup-lock free?" as LockGate
state "Acquire shared lease (wait ≤120s)" as Acquire
state "Snapshotting (sandbox.createBackup)" as Snapshot
state "Encrypting at rest (F4E1)" as Encrypt
state "Committing catalog + pruning" as Commit
state "No backup written" as NoBackup
state "Reprovision guard" as ReproGuard
state "Dispatch cron (backup control)" as DispatchCron
Idle --> Acquire : hard restart checkpoint (POST /debug/restart-openclaw)
Idle --> Acquire : manual sync (POST /api/admin/storage/sync)
Idle --> ReproGuard : reprovision (POST /debug/reprovision)
Idle --> DispatchCron : Dispatch Worker cron (15 min)
DispatchCron --> RuntimeGate : backup control action
RuntimeGate --> Idle : not running (never wake)
RuntimeGate --> IntervalGate : running
ReproGuard --> Snapshot : checkpoint (container alive)
ReproGuard --> Idle : 409 — no checkpoint and no restorable backup
IntervalGate --> Idle : < effective interval since last
note right of IntervalGate : max(BACKUP_INTERVAL_MINUTES, 2 × last snapshot duration)
IntervalGate --> LockGate : interval elapsed and BACKUP_BUCKET bound
LockGate --> Idle : lease held (skip this tick)
LockGate --> Snapshot : lease acquired (60s TTL, 20s heartbeat)
Acquire --> Snapshot : lease acquired (60s TTL, 20s heartbeat)
Acquire --> NoBackup : lease held > 120s (bounded error)
Snapshot --> NoBackup : InvalidBackupConfigError / R2 error
Snapshot --> Encrypt : archive created
Encrypt --> NoBackup : BACKUP_ENCRYPTION_REQUIRED and key missing
Encrypt --> Commit : encrypted (.enc) or plaintext (warned)
Commit --> Idle : backup-handle.json written, oldest pruned
NoBackup --> Idle : lease released (CAS tombstone)
The same state machine as an Eraser
flowchart. Render it by POSTing the DSL to the Eraser render API (see the
eraser-diagrams skill); it is kept inline so the diagram can be regenerated
without an API key.
direction down
Start [shape: oval]
Idle [shape: rectangle, label: "Idle - no snapshot scheduled"]
RuntimeGate [shape: diamond, label: "Runtime active? (isRuntimeActive)"]
HardRestart [shape: rectangle, label: "Hard restart checkpoint"]
ManualSync [shape: rectangle, label: "Manual sync"]
ReproGuard [shape: diamond, label: "Reprovision guard"]
DispatchCron [shape: rectangle, label: "Dispatch cron (backup control)"]
NotRunning [shape: oval, color: gray, label: "Skip (never wake)"]
IntervalGate [shape: diamond, label: "Interval elapsed?"]
LockGate [shape: diamond, label: "backup-lock free?"]
Acquire [shape: rectangle, label: "Acquire shared lease (wait ≤120s)"]
Snapshot [shape: rectangle, label: "Snapshotting (sandbox.createBackup)"]
Encrypt [shape: rectangle, label: "Encrypting at rest (F4E1)"]
Commit [shape: rectangle, label: "Commit catalog + prune"]
NoBackup [shape: oval, color: red, label: "No backup written"]
Start > Idle
Idle > HardRestart
Idle > ManualSync
Idle > ReproGuard : "POST /debug/reprovision"
Idle > DispatchCron : "Dispatch Worker cron (15 min)"
DispatchCron > RuntimeGate : "backup control action"
RuntimeGate > NotRunning : "not running"
RuntimeGate > IntervalGate : "running"
ReproGuard > Snapshot : "checkpoint (container alive)"
ReproGuard > Idle : "409 - no checkpoint, no backup"
IntervalGate > Idle : "too soon"
IntervalGate > LockGate : "yes + BACKUP_BUCKET bound"
LockGate > Idle : "lease held (skip tick)"
LockGate > Snapshot : "acquired (60s TTL, 20s heartbeat)"
HardRestart > Acquire
ManualSync > Acquire
Acquire > Snapshot : "acquired (60s TTL, 20s heartbeat)"
Acquire > NoBackup : "lease held >120s (bounded error)"
Snapshot > NoBackup : "InvalidBackupConfigError / R2 error"
Snapshot > Encrypt : "archive created"
Encrypt > NoBackup : "encryption required, no key"
Encrypt > Commit : "encrypted or plaintext warned"
Commit > Idle : "catalog written, oldest pruned"
NoBackup > Idle : "lease released (CAS tombstone)"| Automatic trigger | Entry point | Gates that must pass |
|---|---|---|
| Dispatch cron checkpoint | POST /control (X-FourIA-Control: backup) via the Dispatch Worker cron | tenant not frozen; runtime active (isRuntimeActive(), never wakes an idle container); BACKUP_BUCKET bound; effective interval elapsed (max(BACKUP_INTERVAL_MINUTES, 2 × last snapshot duration)); 60s lock free; SDK presigned creds present |
| Hard-restart checkpoint | POST /debug/restart-openclaw | none (unconditional checkpoint before kill) |
| Reprovision guard | POST /debug/reprovision | runtime-active checkpoint (time-bounded) or a restorable backup exists; otherwise 409 (override with {"force":true}) |
| Manual snapshot | POST /api/admin/storage/sync | caller authorized (ADMIN/ROOT) — explicit admin action |
Operator traps that leave the bucket empty even though the worker is healthy.
First, the Dispatch cron is the only automatic trigger. A tenant missing from
TENANT_INDEX(or marked frozen) is skipped. The lermaHealthMonitoris observability-only and never backs up.Second, the Sandbox SDK (>= 0.12.9)
createBackup()on the remote path callsrequirePresignedURLSupport(), which throwsInvalidBackupConfigErrorunlessR2_ACCESS_KEY_ID,R2_SECRET_ACCESS_KEY,BACKUP_BUCKET_NAME, andCLOUDFLARE_ACCOUNT_IDare all set on the tenant worker. The failure is caught byscheduleBackupIfNeeded()and logged as[persistence] Scheduled backup failed:— the R2 bucket then contains onlybackup-handle.json(or nothing at all).Third, snapshots only run against an active runtime. A sleeping container is skipped (never woken — it has no workload and no filesystem changes to capture), and
POST /debug/reprovisionreturns409when the runtime is idle and no restorable backup exists.Verify all three before assuming backups are running.
To verify the R2 backup bucket is reachable without triggering a snapshot, use
POST /api/admin/storage/test-connectivity (ADMIN or ROOT). It HEADs the backup catalog
handle; a 404 (no backup written yet) still counts as connected. ROOT users can run it from
the Diagnostic Dashboard’s Test Backup Connectivity button even though the Backups page
is ADMIN-gated.
The number of retained versions is BACKUP_MAX_VERSIONS (default 3); the oldest
beyond that are pruned. Retention is enforced on the snapshot path and by the
explicit catalog reconcile (reconcileBackupCatalog, run by the durable repair
step before every scheduled snapshot and by restore), so a run of failed snapshots
cannot leave the catalog or R2 over the cap. GET /api/admin/storage/backups is a
pure read; run POST /api/admin/storage/reconcile (ADMIN or ROOT) to adopt
orphaned archives, sweep dead ones, re-encrypt plaintext ones, widen an expired
archive TTL, and prune on demand.
Retention is a count, never a clock. The archive TTL (SNAPSHOT_ARCHIVE_TTL_SECONDS,
10 years) is not a storage-lifecycle hint: the SDK refuses to restore an archive
past createdAt + ttl. A short TTL therefore turns a retained recovery point into
a billed-but-unusable object, which is why the durable repair step widens it for
existing tenants (§5.4d).
Archives are full copies of the container by default. BACKUP_ARCHIVE_EXCLUDES=true
opts an instance into dropping regenerable paths before mksquashfs (.npm,
*.log, *.tmp, .cache, and workspace/*node_modules*), which shrinks the
archive but makes an archive-restored tenant differ from its pre-eviction tree
(and from a Tier 1 workspace snapshot). It is off by default; enable it per
instance only when you know those paths regenerate. Plugin dependencies, SQLite
databases and session transcripts are never excluded: a restore has no reinstall
step of its own.
4.3b Change-aware backups
Scheduled snapshots skip tenants whose state has not changed, and a per-tenant
watch can snapshot within minutes of a change instead of waiting for the next
15-minute dispatch tick. Detection is on by default; the watch loop is
opt-in. See docs/WORKER-DEVELOPMENT.md (Change-Aware Backups) for the mechanism.
- Detection is already on. A snapshot re-uploads the whole archive, so an
unnecessary one dominates the backup bill: measured on
tenant-tarjetero-dev, two snapshots 37 minutes apart differed by 0.08 % while both were 428 MB. Only an explicitBACKUP_CHANGE_DETECTION=false(lerma provisioning flag) opts out. - Enable the watch: set
BACKUP_CHANGE_WATCH=truefor that lerma environment, then redeploy the affected tenants. It also binds theBACKUP_WATCHDurable Object. - Roll back: set the flag to
falseand redeploy. Both flags are always emitted explicitly, so the rollback takes effect on redeploy. - Observe:
[persistence] Scheduled backup skipped: tenant state unchangedmeans the gate saved a snapshot.[backup-watch] scan failed(orrequest failed) means the watch fell back to a snapshot or retry. A scan failure is never a skipped backup. - Watch for cost: each armed tenant runs one short
findper check. The watch disarms after 10 idle minutes, so a quiet tenant is not kept awake.
4.4 Restoring
- Automatic —
restoreIfNeeded()runs at container startup and restores the committed recovery point when the container has no tenant state. Therestore-neededmarker is not required to trigger a cold restore: its absence is normal while idle and after a successful restore. A cold restore writes the marker before applying the archive and clears it after restore plus legacy state materialization succeed. If candidate scanning finds no restorable archive and no stable recovery point exists, it clears the temporary marker for the permitted clean start. A recorded stable recovery point with no restorable archive instead fails closed and keeps the marker. - Targeted —
restoreBackupById(sandbox, bucket, backupId, env, ctx, options)restores a specific snapshot (persistence.ts). Use the backup history + config version list to pick one. A successful targeted restore force-promotes the restored archive to the recovery point (stable-backup.json) even when it predates the current one, so the next cold boot restores the same archive. Without this,restoreIfNeededkeeps preferring the newer pointer — which advances on every committed snapshot — and silently re-restores the wrong state (e.g. a post-deploy snapshot taken from a not-yet-materialized config, observed ontenant-tarjetero-dev). - Restore without moving the recovery point —
POST /api/admin/storage/restore/:id?setCurrent=0(also?async=1&setCurrent=0) passes{ setRecoveryPoint: false }through torestoreBackupById, which applies the archive but leavesstable-backup.jsonunchanged. Use this to inspect or recover an older snapshot with the next fresh container reverting to the current recovery point. The flag defaults to set-current (?setCurrent=1/ omitted), matching the targeted-restore behavior above. The Backups page exposes it as the Set as recovery point checkbox (checked by default); unchecking maps tosetCurrent=0. The restored state is then protected by a restore-only hold (§4.3), so scheduled snapshots cannot move the recovery point onto it. - After a restore the gateway is launched and the config is re-captured — every restore path
(the durable boot job, the durable restore job, and the synchronous admin route) finishes through
launchRestoredState(src/post-restore.ts): it brings the restored state up (ensureGateway({ waitForReady: true })) and captures the live config as a newconfigs/<ts>.jsonplusstable/config.json, so the restored configuration becomes the latest and stable config version. A launch failure does not fail the restore (the restore already succeeded): the sync route returnsgatewayReady: falseand the durable job succeeds withdetail: "gateway-launch-pending", leaving the durablerestore-config-pending.jsonmarker for the next boot/maintenance tick. Adoctor --fixrepair is deferred (without consuming attempts) while a restore is queued/applying or the restored gateway is not up yet — otherwise its fix would be silently overwritten by the next re-apply of the recovery point (observed ontenant-tarjetero-dev). - Timeouts never start a second restore —
sandbox.restoreBackupis bounded byRESTORE_BACKUP_TIMEOUT_MS. The bound only stops the wait: the container call keeps running. A timeout therefore records the restore as outcome unknown, and every other restore is refused (409,RestoreInFlightError) until that record is older than the restore phase budget (GATEWAY_RESTORE_PHASE_BUDGET_MS, 11 min). A restore that fails definitively does not block a retry. The restore-needed marker stays set, so the next restore after the window re-applies the archive. - The recovery point’s restorable TTL is healed on the restore path — the SDK
refuses an archive past
createdAt + meta.json.ttlbefore it touches the container, so an archive whose TTL lapsed fails every boot in ~250 ms even though its bytes are intact.ensureArchiveRestorablewidens the TTL right beforesandbox.restoreBackup(both the automatic and the targeted path), so a tenant whose catalog predatesSNAPSHOT_ARCHIVE_TTL_SECONDScan still boot. See §5.4e for the operator view. - Writer quiesce + maintenance window —
restoreBackupByIdopens a cross-isolate maintenance window (maintenance/replacement.json, operationrestore) and stops the gateway immediately after it takes the shared restore lease, then closes the window in afinally. A live gateway (or automations/channels) therefore cannot keep writing while the archive replaces the writable layer, and a concurrent health/proxy refetch cannot restart it mid-restore.restoreIfNeededrefuses to run while any maintenance window is active, which breaks the restore → stop-gateway → refetch → remount-the-latest loop; the window self-heals on its TTL if the controller dies. - Recovery visibility — the Backups page marks the committed recovery point in
the snapshot timeline (
GET /api/admin/storage/backups→stableBackupId) with a recovery point badge and a header line (A cold boot restores <id> — <date>). The loading page names the backup + date being loaded and shows a progress bar synchronized with the durable restore’s phase events: while a restore is in flight it reads the durablerestore-target.json(the archive being applied) and otherwise the committedstable-backup.json, surfaced asrecovery.restoring/recovery.committed, plusrestoreProgress({ phase, label, fraction }) from theActivitySchedulerDO, on/api/health(non-passive) and/api/startup/ws. So an operator can always tell which snapshot a cold start is restoring, when it was taken, and how far the restore has progressed, without opening logs.GET /api/admin/boot-statereturns the samerestoreProgressfor inspection.
Restore and scheduled-backup safety rules:
- The pre-restore
umountis best-effort:umount … || trueonly guards the shell command, not the exec RPC. A cold-start/platform interruption on that exec is caught and logged instead of aborting the restore beforerestoreBackupruns — the previous fatal behaviour keptrestore-neededand failed the startup restore barrier on every boot. - An empty catalog read does not clear
restore-needed. The read can be transient (a reconcilelistfailure, or an archive not yet visible); a pending marker is kept so an operator-requested restore is never silently dropped. Restore failures keep the marker. If candidate scanning finds no restorable archive, a recorded stable recovery point fails closed and keeps the marker; with no stable pointer, the clean-start fallback removes it. - If restore fails after writing partial state,
restore-neededremains in R2 and forces a retry even when a config file is already present. Scheduled snapshots skip while that marker exists, rechecked under the backup lease, so a cron tick cannot overwrite the stable snapshot during a cold boot. The gate is keyed on the durable R2 marker (not a container-state probe) so an inconclusive probe can never silently disable scheduled backups.
Therefore, an absent restore-needed object by itself is not evidence that R2
lost a restore request. Check stable-backup.json and its referenced
backups/<id>/meta.json; those identify the committed recovery point. A fresh
container restores it based on the state probe even when the marker was absent.
Each entry in backup-handle.json records whether it was created with the SDK’s
localBucket mode (persistence.ts). That flag is forwarded to restoreBackup() so a
local-binding backup restores through the local R2 binding instead of the
remote presigned-URL path — a backup created locally only exists in the local bucket, so a
remote restore 404s with Presigned URL download failed (exit code 22).
Local-bucket mode is the explicit BACKUP_LOCAL_BUCKET=true opt-in; when unset it falls
back to DEV_MODE for local development. Do not rely on the DEV_MODE fallback for a real
deployment (including the cloud E2E, which sets DEV_MODE=true only to bypass Access): set
BACKUP_LOCAL_BUCKET=false there so backups land in the remote BACKUP_BUCKET the Worker
restores from.
Entries created before the flag was persisted carry no localBucket; on restore it is
defaulted from the same signal used at create time, so legacy local backups
still restore via the local binding. A restore that 404s at the presigned URL (archive
gone / unreachable) is treated like BACKUP_NOT_FOUND — the stale entry is dropped from
the catalog so one dead backup cannot block every subsequent boot.
Operator workflow for “I need to roll back to a known-good config”:
- List config versions:
GET /api/admin/storage/config(or inspectconfigs/in R2). - Confirm the snapshot you want exists in
backup-handle.json’sbackups[]. - If the container is up, stop the gateway (
POST /debug/stop-gateway), then call a targeted restore, then restart. If the container is down, the next boot auto-restores the newest.
⚠️ The current implementation uses the Sandbox backup API and the
BACKUP_BUCKETR2 binding. It does not mount R2 with s3fs. Never delete tenant backup objects outside the retention or decommission procedure.
4.5 Setting up R2 (one-time)
Scheduled backups run when the Worker has its BACKUP_BUCKET binding and the
interval/lock guards pass (scheduleBackupIfNeeded() in persistence.ts).
The bound bucket is provisioned per tenant, for example
fouria-backup-{slug}{-environment}{-platform}.
The Sandbox SDK remote path requires R2 S3 credentials. In
@cloudflare/sandbox >= 0.12.9,createBackup()(andrestoreBackup()) on the non-localBucketpath callsrequirePresignedURLSupport(), which throwsInvalidBackupConfigErrorunless all ofR2_ACCESS_KEY_ID,R2_SECRET_ACCESS_KEY,CLOUDFLARE_ACCOUNT_ID, andBACKUP_BUCKET_NAMEare present on the tenant Worker.hasBackupCredentials()only gates onBACKUP_BUCKET, so a misconfigured worker still runs the scheduler and then fails at snapshot time (logged as[persistence] Scheduled backup failed:) with an empty bucket. Only local-bucket mode (BACKUP_LOCAL_BUCKET=true, or theDEV_MODEfallback for local development) avoids the credential requirement.
-
Bind the tenant bucket — the Lerma provisioner creates and binds the tenant R2 bucket as
BACKUP_BUCKETand setsBACKUP_BUCKET_NAME. -
Let lerma resolve it (it is now mandatory) — the provisioner resolves a credential per instance and fails provisioning (rolling back) when none can be produced; it no longer warns and deploys a tenant that silently cannot back up. Strategy is
R2_TOKEN_STRATEGY(auto|shared|per_instance):auto(default): usesR2_TOKEN_MAX_PER_ACCOUNT; a known limit of at most 250 selectsshared, a larger or unknown limit selectsper_instance. Cloudflare exposes no maximum-token endpoint, so the limit defaults to unknown → per-instance token minted at instance creation, scoped to the tenant bucket.shared: setR2_ACCESS_KEY_ID/R2_SECRET_ACCESS_KEYin the lerma environment (generate withmix lerma.r2_token --all-buckets).
Generated per-instance credentials are persisted on the instance record and reused across redeploys; a token minted for a failed brand-new provisioning is deleted on rollback.
CLOUDFLARE_ACCOUNT_IDis a plain var and is always injected. -
Ensure the client is being probed — the lerma
HealthMonitoris the only automatic trigger and only polls clients inproduction/error. A client indraftgets no scheduled snapshots regardless of credentials.# For a standalone/legacy Worker (not provisioned via lerma): npx wrangler secret put R2_ACCESS_KEY_ID npx wrangler secret put R2_SECRET_ACCESS_KEY
Sandbox snapshot behavior differs between deployed and local environments; local tests use the SDK’s
localBucketmode whenBACKUP_LOCAL_BUCKET=true(or theDEV_MODEfallback) inpersistence.ts. The flag is persisted in the backup catalog so restores route to the same mode — a local backup restores via the local R2 binding, and a remote (presigned-URL) backup restores viaR2_ACCESS_KEY_ID/R2_SECRET_ACCESS_KEY/CLOUDFLARE_ACCOUNT_ID/BACKUP_BUCKET_NAME. Pre-persist entries default to the current mode. There is no s3fs mount.
5. Troubleshooting — Critical Issues
This section maps symptoms → root cause → the exact recovery from this guide.
5.1 Container stuck in Error / image deleted → RESCUE
Symptom: /api/health?passive=1 returns degraded:true,
containerError:"no such image" (or no matching app). The instance can never boot.
Why: the stale-image cleanup cron deleted the fouria:unstable-<sha> image the
application references (diagnostics.ts:42). The cleanup script now protects every
tag still referenced by a container application — including the versions of an
in-progress rollout — and aborts (fail closed) when that in-use set cannot be
resolved (.github/scripts/cleanup-stale-images.sh). Production deploys are also
restricted to stable release tags, so a production application can no longer be
pinned to the unstable-<sha> channel (Lerma.Cloudflare.ImageRegistry.stable?/1).
Fix: rescueContainer (§3.4):
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":true}' | jq .
# confirm action would be repair-image-and-destroy, then:
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":false}' | jq .
Credentials required: CLOUDFLARE_API_TOKEN (Containers:Edit),
CONTAINER_APPLICATION_ID, and either CONTAINER_IMAGE_TAG or an explicit
expectedImage (diagnostics.ts:304, .dev.vars.example:43). Without them, the image
cannot be repaired — only the container is destroyed (you then must re-deploy the
correct image tag).
NEVER set these on tenant workers — the rescue tool re-points the application image
and is operator/ROOT-only (.dev.vars.example:40).
5.2 Gateway “already running” / EADDRINUSE on restart
Symptom: restart fails with gateway already running (pid …) or failed to bind gateway socket.
Why: a previous gateway still holds the native lock / TCP bind when the new one
starts (gateway/process.ts).
Fix: for a clean state use killGateway() (the POST /debug/stop-gateway route), which
SIGTERMs with the GATEWAY_KILL_GRACE_MS (5s) grace and only SIGKILLs if still active,
then wait for the port to close before reprovisioning. POST /debug/restart-openclaw (hard
restart) does this then reprovisions via start-openclaw.sh. Inspect lock state with
GET /debug/gateway-lock.
5.3 Gateway process invisible but container live
Symptom: listProcesses() shows nothing, yet /health answers.
Why: the gateway runs as a detached child of start-openclaw.sh
(process.ts:427). The SDK does not track it, but it is healthy.
Fix: the live-gateway fallback (findExistingGatewayProcess with probeLive:true)
detects it via a bounded /health probe. Prefer restart/rescue APIs that use this path
rather than destroy-container (which would throw away a healthy instance).
5.4 Backup restore loops / “hadBackup but not restored”
Symptom: health startup.ok=false permanently with
Backup found but not restored.
Why: restoreIfNeeded found an entry but every backup expired/was missing
(persistence.ts:212). The code explicitly reports a clean start in this case so it
does not gate startup forever (persistence.ts:233).
Fix: confirm real backups exist (backup-handle.json → backups[]); if none, the
instance boots fresh (expected). If backups exist but restore fails, check
BACKUP_ENCRYPTION_KEY (encrypted backups cannot be read without it).
Manual-restore loop (restore → stop-gateway → refetch → remount the latest).
A manual restore of an older archive kills the gateway so no writer mutates the
layer being replaced. Without a guard, the next health/proxy refetch calls
ensureGateway → restoreIfNeeded, which prefers the committed recovery point
(the newer archive) and silently undoes the restore. Two guards prevent this:
restoreBackupById holds a cross-isolate maintenance/replacement.json window
(operation restore) for the whole restore, during which ensureGateway refuses
to launch, and restoreIfNeeded itself refuses to run under any active window.
A targeted restore that should survive cold boots also force-promotes the archive
(the default); a setCurrent=0 restore deliberately does not, and is reverted on
the next fresh container.
5.4b Stuck gateway start (“already in progress” forever)
Symptom: logs repeat [Gateway] Gateway start already in progress for openclaw , waiting... for tens of minutes while listProcesses returned N processes keeps climbing and the port stays closed (health probes curl exit 7
every ~5s). The gateway never binds 18789.
Why: the shared ensureGateway start promise spans restore → supervised
start-openclaw.sh start → port wait. If the wrapper never execs the gateway
(or the container control channel drops), the promise does not settle and every
request waits on it. The raw listProcesses().length cannot distinguish this
from a busy boot — it counts exited processes too and only grows.
Automatic recovery: the Worker now tracks the start phase and force-resets a
phase that overran its budget, logging a structured gateway.start.stuck
event and killing the stuck process; after repeated resets it reprovisions the
container. Before resetting a port-wait, it makes a fresh /health check: a
gateway that became healthy after the original readiness deadline is adopted
(gateway.start.late-ready) instead of killed. If that recheck cannot reach the
container, health is unknown and reset is deferred until a later request can
probe again. A start-openclaw.sh start wrapper older than the cold-start budget
with the port closed is also killed immediately.
Manual recovery (if needed): POST /debug/reprovision (ROOT, DEBUG_ROUTES=true)
to destroy + reprovision the container, or POST /debug/restart-openclaw to
hard-restart the gateway in place; a freeze/unfreeze from lerma also forces a
fresh cold start.
Investigate: filter the cloudflare-workers dataset to
event = "gateway.start.stuck", event = "gateway.start.phase",
event = "gateway.start.late-ready", or
event = "gateway.start.health-recheck-failed" — records carry the phase,
attempt, elapsed time, and process id needed to distinguish a real stuck start
from a late healthy gateway or an inconclusive probe.
5.4c Repeated same-size snapshots and a container that never sleeps
Symptom: R2 keeps receiving a full archive of the same size every few
minutes/hours (e.g. 422 MB at 02:35/02:39/02:44 and 05:39/05:43/05:49 UTC), the
container instance shows running with a started_at days old, and
backup-handle.json has a stale lastScheduledBackup even though new entries
appear. Often paired with a cost spike at a quiet hour.
Why: the idle-sleep deprovision job snapshots before stopping. On a
large tenant that archive takes minutes, and its own RPCs renew the Sandbox DO
activity window, so the stop-time stopIfIdle() re-check refuses
(active:activity). The SDK re-arms, the next idle expiry enqueues another
deprovision, and each attempt uploads the full archive again. Deprovision
archives used to be unstamped, so the scheduled cadence did not advance either.
Fix (current bundle): TenantSandbox.onActivityExpired() first proves that an
existing archive already covers the current filesystem, and stops inline without
a new archive when it does:
- a committed snapshot inside the idle window (+5 min slack), or
- with change detection on (the default),
unchangedfrom the same change scan the scheduled path uses — a quiet tenant takes no scheduled snapshot, so its newest archive can be older than the idle window even though nothing changed.
A deprovision archive also stamps lastScheduledBackup. If an older tenant still
loops, redeploy it to the current bundle. Diagnose with:
npx wrangler r2 object get fouria-backup-<slug>/backup-handle.json --remote --file -
npx wrangler r2 object get fouria-backup-<slug>/backups/<id>/meta.json --remote --file -
Related — 30-minute dev heartbeats. OpenClaw 2026.10.5 retired
agents.entries.*.heartbeat: the heartbeat is an ordinary automation job and
config reloads never rewrite it, so a migrated dev instance can keep the default
30-minute cadence (a full agent turn on the main session each time). The
dev-only ensure_dev_heartbeat_cadence in start-openclaw.sh reconciles it on
post-start; to fix a warm instance immediately:
openclaw automations edit heartbeat-fouria-builder \
--every 12h --session isolated --light-context5.4d “The gateway rebooted — was it memory?”
Symptom: the gateway cycled (2–5 boots in a day), the dashboard dipped to
starting/503, and someone asks whether it hit the container memory limit.
Answer it from the Worker first — no SSH needed. /api/health (and
/api/health-snapshot / /api/health-ws) carries a gateway block for a running
gateway:
"gateway": { "pid": 4242, "rssBytes": 2234000000, "threads": 29 }
RSS is sampled from /proc/<pid>/status at most once a minute, so it is cheap to
poll. The same numbers land in Analytics Engine as a gateway-resources row
(double1 = RSS bytes, double2 = threads, blob1 = gateway-resources, blob2 =
customer) — chart it to tell a monotonic leak from a transient peak:
SELECT toStartOfHour(timestamp) AS hour,
max(double1) / 1048576 AS rss_mib, max(double2) AS threads
FROM fourai_cost_metrics
WHERE blob1 = 'gateway-resources' AND blob2 = 'tarjetero'
AND timestamp > NOW() - INTERVAL 1 DAY
GROUP BY hour ORDER BY hour
Reference points on a standard-1 container (4 GiB, no swap):
| Reading | Meaning |
|---|---|
| ~1.1–1.4 GiB | normal baseline right after a hard restart |
| ~2.1 GiB (peak ~2.6 GiB) | observed steady state of a busy dev gateway; still healthy |
| ≥ 1.5 GiB | OpenClaw’s own rss_threshold warning — expected to fire at this level; it is not an action item |
| ≥ 3 GiB | the dev watchdog threshold — the gateway gets replaced |
Before blaming memory, rule it out properly. An OOM kill leaves fingerprints that a graceful restart does not:
# inside the container (wrangler containers ssh)
grep oom_kill /proc/vmstat # must stay 0
dmesg | grep -i "killed process" # empty
grep -ciE "heap out of memory|Allocation failed" /tmp/gateway-stdout.log
grep -c "restart-signal fence" /tmp/gateway-stdout.log # graceful self-restarts
grep -c "host timing gap detected" /tmp/gateway-stdout.log
A non-zero restart-signal fence / received SIGTERM count means the gateway
restarted itself on purpose (doctor repair, config reload, update) — not OOM. A
host timing gap detected: process was frozen ~Ns line means the platform
froze the microVM; a freeze long enough to blow past startup deadlines is followed
by the supervisor replacing the process, which looks like a reboot but is not a
memory event. Neither of those is an OOM.
Dev-only memory watchdog. On a dev instance (-dev hostname /
OPENCLAW_DEV_MODE) the Worker replaces the gateway when RSS reaches 75% of
the container’s memory limit, at most once per 30 minutes, and logs:
{
"event": "gateway.memory.pressure",
"rssMib": 3072,
"threads": 31,
"limitMib": 3072,
"containerLimitMib": 4096,
}
The threshold scales with the instance: 3 GiB on a 4 GiB standard-1, 4.5 GiB on
a 6 GiB standard-2, 6 GiB on an 8 GiB standard-3, 9 GiB on a 12 GiB
standard-4. It is derived from the container’s cgroup limit on every sample, so
an instance-type rollout changes it without a Worker deploy. Instances below
3 GiB (basic is 1 GiB) are not restarted — they cannot hold a healthy
gateway, so the watchdog would only restart-loop; standard-1 is the practical
floor for a tenant that runs agents.
If you see that event, the watchdog fired before the kernel OOM killer could — that is the intended behaviour, not an outage. It is deliberately gated to dev: a replacement interrupts live agent sessions. Production keeps today’s behaviour (no automatic replacement).
Fix the cause, not the symptom. RSS growth is driven by oversized persisted
transcripts and accumulated worker_thread V8 heaps that an in-process restart
never reclaims; see
../../docs/plans/sandbox-memory-usage.md
for the mitigation set and what is still open.
5.4e A recovery point that is billed but cannot be restored (BACKUP_EXPIRED)
Symptom: backup-handle.json and the Backups page list an archive, the R2
object is present and billed, but a restore logs
Backup <id> has expired (created: … TTL: 604800s) → BackupExpiredError →
[persistence] Backup <id> expired/gone, removing from store, and if it was the
recovery point the boot fails closed with TenantStateUnavailableError.
Why: the SDK treats meta.json.ttl as a hard restorability gate
(createdAt + ttl), not a storage-lifecycle hint. Retention prunes by count,
so an idle, frozen, or repeatedly failing tenant could keep maxVersions
archives that no boot could ever apply — while the dashboard still showed them as
the recovery point.
Fix (current bundle): new archives are written with
SNAPSHOT_ARCHIVE_TTL_SECONDS (10 years), and the durable repair step heals
existing tenants: extendArchiveTtl rewrites every meta.json below the minimum
(preserving dir, createdAt, encrypted, encryptionVersion) and reports
ttlExtended / ttlFailed from POST /api/admin/storage/reconcile. The restore
path also heals its own archive (ensureArchiveRestorable, right before
sandbox.restoreBackup), because the repair step only runs inside a snapshot
job — a tenant that cannot boot can never reach the repair that would make it
bootable.
Verify with:
npx wrangler r2 object get fouria-backup-<slug>/backups/<id>/meta.json --remote --file -
# ttl should be 315360000 (10 years), not 604800
Unblock a tenant that is already wedged (no deploy needed — this is R2-only
and works with the gateway down). Note the -X POST: GET on that path does
not match the route and falls through to the gateway proxy, which answers
503 {"error":"Gateway not ready","details":"Tenant boot pending …"}:
curl -X POST https://<tenant-host>/api/admin/storage/reconcile \
-H "cookie: CF_Authorization=<access-jwt>"
# → { "success": true, "ttlExtended": ["<id>", …], … }
Then reload the tenant page: the next boot heals the archive and applies it. The
bytes are unaffected — only the SDK’s restore guard reads meta.json.ttl.
Symptom you will see instead if you skip this: the loading page sits on
Restore attempt failed during "Restoring the backup" — retrying — no progress for N min. That is a fast failure, not a slow restore: the SDK rejects the
expired archive in ~250 ms (one meta.json read), the durable boot job retries
three times 30 s apart (~70 s total) and then goes terminal failed, and the page
keeps rendering the last attempt’s record. Read the bucket to confirm —
restore-target.json with "outcome":"failed", restore-needed still present,
and a backup-lock tombstone ("acquired":0) with nothing written afterwards.
Current bundles replace that stale label with Restore failed — <reason> and
Boot failed after N attempts: <reason>. Refresh the page to retry.
5.4f OTLP billing and promotion show no effect (bridge / R2 transports)
Neither feature has an internet route any more, so “reachability” is not the
first question. There is no /api/otel/* and no /api/admin/releases/import:
diagnose them from the inside.
| Symptom | Cause | Fix |
|---|---|---|
| no traces at all | OTEL_TRACES_ENABLED is not true | set it on the tenant Worker; diagnostics.otel.traces is off by default |
exporter gets 401 | OTEL_SHARED_SECRET is unbound (or differs from the one you replay with) | bind it on the tenant Worker (lerma platform secret) |
403 TENANT_MISMATCH | the Worker is not bound to the tenant slug | verify CUSTOMER_SLUG / WORKER_NAME on the tenant Worker |
no billable-action rows | the batch reached the bridge but had no billable span (check the response’s emitted/nonBillable) | confirm the span name/channel and the plan’s action_type |
| promotion says “staged” and nothing happens | production has not booted/applied yet, or is parked / inside a maintenance window | cold-start it, or POST /api/admin/releases/apply; check GET /api/admin/releases/status (available, release) |
| production never sees the release | PROMOTION_SOURCE_BUCKET missing (older tenant) or targeting the wrong tenant/environment | re-provision/upgrade the production instance so the sibling import bucket binding is added |
| release applied but old agents remain | production booted from an older snapshot with an older container pointer | the next boot re-reads pending.json; force POST /apply |
| promotion “succeeded” but instances never start | the promoted image had no runtime artifact: instances sit in starting, then failed with ImagePullError (10-minute pull budget) | re-run the promotion (lerma prepares the image before rolling and gates on live health); check wrangler containers images list and the app’s health.instances |
Verify from inside the container (the bridge is a virtual host, not a public URL):
# 401 = secret missing/wrong, 200 = bridge reachable and ingest accepted
curl -sS -o /dev/null -w '%{http_code}\n' -X POST http://otel.internal/v1/traces \
-H "authorization: Bearer $OTEL_SHARED_SECRET" -H 'content-type: application/json' -d '{}'Where promotion runs (Worker API, not the container)
Promotion is an operator API action on the tenant Worker. Nothing runs inside the container or through the OpenClaw CLI: the container is only the source (dev) and the target (production) of the file copy.
| Step | Instance | Runs in | Who |
|---|---|---|---|
Stage — POST /api/admin/releases/promote | dev | the dev Worker: exports the dev container’s /home/openclaw (config + workspace) and writes promotion/staged/<id>.tar.gz + promotion/pending.json to the dev bucket | operator session |
History — GET /api/admin/releases/history | both | the Worker: dev reads its own staged entries; production merges the sibling dev bucket with its own applied/failed entries | operator session |
Inspect — GET /api/admin/releases/status | production | the production Worker (container pointer + the sibling bucket) | operator session |
Apply — POST /api/admin/releases/apply, or a production cold boot | production | the production Worker: pulls the bundle, writes it into the production container, re-applies the gateway section and soft-reloads | operator, or the boot job automatically |
promoteis deliberately dev-only: on a production instance it answers409 NOT_DEV_INSTANCE.applyon the dev instance finds nothing (wrong-environment), because the staged pointer targetsproduction.- An agent inside the dev container (e.g. the Fouria Builder) cannot perform
the promotion:
/api/admin/*sits behind the host Access app and requires a resolved ADMIN/ROOT role, which a container process does not have. It prepares the artifact and hands the operator the commands. - The only container-adjacent step is the production boot-time apply, and that is
still driven by the Worker (
ensureGateway) before the gateway launches — never by OpenClaw itself.
# dev instance: stage with a commit message (202 { staged: true })
curl -sS -X POST https://<dev-host>/api/admin/releases/promote \
-H 'content-type: application/json' \
-d '{"message":"Ship the new intake flow"}'
# both instances: push history (short hash, message, author, status)
curl -sS https://<dev-host>/api/admin/releases/history
curl -sS https://<prod-host>/api/admin/releases/history
# production instance: inspect, then apply (or just cold-start it)
curl -sS https://<prod-host>/api/admin/releases/status
curl -sS -X POST https://<prod-host>/api/admin/releases/apply
Every push is content-hashed (SHA-256 of the bundle) and carries the deployer’s
commit message plus the verified Access identity as the author. The history on
the dev instance lists the pushes it staged; the production instance sees those
staged entries through its read-only sibling binding and records its own
applied/failed outcome, so both sides can follow the same sequence without
either bucket being writable by the other.
Promotion from the dashboard (/_admin/releases)
The same operations are available in the admin SPA — Governance → Releases
(ADMIN-only) — once the operator has the same Access session the curl calls
need. Open the tenant’s dev host first and press Stage release for
production; then open the production host and press Apply release now.
The page shows the release waiting in the sibling bucket, the current release
pointer (status: applied is the only proof), the promoted/excluded agents, and
the post-apply verification checklist.
Two states the page surfaces explicitly, both worth checking before an apply:
- “No
PROMOTION_SOURCE_BUCKETbinding” — production cannot see the staged release. Re-provision/upgrade production (lerma creates the dev bucket if missing); staging again will not help. - A skipped apply with a reason —
parked(rollout in progress: resume it),busy:<phase>(backup/restore/maintenance window: apply later),nothing-pending/already-applied/bundle-missing. The API is idempotent, so re-pressing Apply after the blocker clears is safe.
The page is Worker-local (no gateway dependency), so it works while the container is asleep.
Promotion runbook — the real prerequisites
Every operator route on both tenant hosts sits behind the host-level Access
application. GET/POST without a credential answers 302 to
<team>.cloudflareaccess.com with www-authenticate: Cloudflare-Access, and
there is no path bypass to provision (the /import machine path no longer
exists). Check only these:
| Prerequisite | Detail |
|---|---|
| Access credential | A browser/session identity whose JWT resolves a role: ADMIN on production, and on staging a Paso4 Root session (Dispatch’s staging gate is ROOT-only). An Access service token (CF-Access-Client-Id/Secret) clears the edge but resolves no role — the Worker answers 403, so it cannot drive these routes. |
| Confirmed target | Stage on the tenant’s dev host (https://<slug>-dev…) and apply on its production host. For the staging tenant that is https://staging-client.fouria.io; never https://client.fouria.io. |
PROMOTION_SOURCE_BUCKET | The production instance must carry the read-only binding to its sibling dev bucket. A tenant provisioned before the binding existed stages forever without applying — re-provision/upgrade production to add it (lerma creates the dev bucket if missing). |
| Applied pointer | GET /api/admin/releases/status must show available before POST /apply; after applying it shows the release with status: "applied". Nothing to do on the dev side after 202 { staged: true }. |
PROMOTION_PUSH_TOKEN is not part of this model: there is no Worker→Worker
credential to provision, and no /api/admin/releases/import to protect.
Analytics Engine is never touched from the container. The sandbox exports OTLP to the bridge; the Worker writes the
billable-actionrow with itsANALYTICSbinding — no credential involved — and the platform reconciles credits from its own AE credentials. No Analytics Engine credential is forwarded into the sandbox, and no agent, skill or software-manifest entry should expect one. If a build step asks for an AE token inside the container, it is asking for the wrong thing.
ANALYTICS_ENGINE_API_TOKENis not needed for emission and is no longer provisioned by Lerma: it only powers the optional dashboard read-back, and without it the widget reportsavailable: false(usage is platform-aggregated).
5.4g The launch loops on a corrupt state database (“Starting OpenClaw Gateway” forever)
Symptom: every boot attempt ends within seconds with
[openclaw] Reason: database disk image is malformed (stack:
readSqliteUserVersion → inspectStateDatabaseSchema in
openclaw-state-schema-preflight), and the loading page keeps saying
Still working on "Starting OpenClaw Gateway" (NNNs) — the elapsed counter keeps
growing because the container’s startup-progress.json still names the launch
step from the last attempt.
Why: a restored recovery point can carry a damaged
state/openclaw.sqlite (or an agent openclaw-agent.sqlite). OpenClaw’s schema
preflight reads PRAGMA user_version before anything else, so the Gateway exits
immediately; relaunching cannot help, and before this change each attempt also
burned the full ~5-minute cold-start budget waiting for a port that would never
open.
Automatic recovery: the launch now detects the crash (a non-zero exit of the
supervised start-openclaw.sh start process before the port opens) and matches
the diagnostics against the corruption signatures. It queues the durable
doctor-repair job (reason: gateway-launch-state-database), which:
- runs
openclaw doctor --fix --non-interactivewith the Gateway stopped (the repair preserves the original databases before changing anything) without waiting for a gateway the corruption prevents from launching, and - re-queues the durable boot when it succeeds, so the Gateway relaunches on the repaired state.
While that runs, /api/startup/ws reports repair and the loading page shows
“Repairing the OpenClaw state database (openclaw doctor –fix, attempt N)…”.
Repeated failed launches fold into the same job (409-equivalent
“maintenance job busy”), so nothing piles up.
Investigate: filter the cloudflare-workers dataset for
event = "gateway.launch.state-database-repair" (outcome queued / busy /
no-scheduler / failed, with the matched signature and the job id) and
event = "sandbox.process.failed" with failureReason = "state-database-corrupt" (plus exitCode). The job itself is visible at
GET /api/admin/config/fix/jobs/<id> (or its /ws stream), and its result
carries the doctor summary. If the repair fails, POST /debug/sqlite-integrity
tells you which database is still unhealthy, and restoring an older verified
backup (POST /api/admin/storage/restore/<id>) is the safe fallback.
5.5 Platform & setup troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
npm run dev / deploy: Unauthorized | Cloudflare Containers not enabled for the account | Enable Containers in the Containers dashboard |
[sandbox-do] backup.restore … Presigned URL download failed (exit code 22) … 404 | restore hit the remote presigned-URL path for an archive that only exists in the local R2 binding (a BACKUP_LOCAL_BUCKET/localBucket backup), or BACKUP_BUCKET_NAME doesn’t match the BACKUP_BUCKET binding bucket | redeploy so the backup catalog persists localBucket (§4.4) — local backups restore via the local binding; for remote backups verify R2_ACCESS_KEY_ID/R2_SECRET_ACCESS_KEY/CLOUDFLARE_ACCOUNT_ID/BACKUP_BUCKET_NAME all point at the same bucket as the BACKUP_BUCKET binding (§4.5) |
[sandbox-do] backup.restore error <id> — The operation was aborted (~30 s) repeated on every boot, with sandbox.exec … The operation was aborted for health probes/umount | the archive restore ran under a request execution context, which Cloudflare cancels shortly after the response, so sandbox.restoreBackup was aborted mid-flight (large archives, e.g. 350 MB, never finish) | fixed: request paths defer the restore to the durable ActivityScheduler boot job (RestoreDeferredToBootJobError → GatewayBootPendingError). Verify with grep -c "await restoreIfNeeded" apps/fouria/src/index.ts (must be 0) and look for [Gateway] Restore deferred to the durable boot job; the restore then runs in the DO alarm and shows Durable boot job began/succeeded |
| Gateway won’t start | missing/incorrect secrets | npx wrangler secret list + npx wrangler tail |
| Config changes not taking effect | stale Docker build cache | bump the # Build cache bust: comment in Dockerfile, then redeploy |
| Slow first request | cold start (1–2 min) | expected; later requests are faster |
| R2 not mounting | an R2 secret missing, or running the local Vite dev server | set all four R2 secrets (§4.5); R2 mount is production-only |
Access denied on /_admin or /api | CF_ACCESS_TEAM_DOMAIN / CF_ACCESS_AUD not set | configure Cloudflare Access (§8.1) |
| Devices not appearing in admin UI | device-list CLI takes 10–15 s (WS overhead) | wait and refresh |
| WebSocket issues in local dev | local vite dev WS-proxy limitation | deploy for full WS functionality |
Windows: script exits 126 | CRLF line endings in shell scripts | add .gitattributes with * text=auto eol=lf |
5.6 Deploy-window “Could not list processes” / platform interrupted
Symptom: logs show [Gateway] Could not list processes: followed by a stack trace ending in
createPlatformInterruptedError / translatePlatformInterruption, and/or
/debug/processes returns a 503 with { "status": "platform_updating" }.
Why: the Sandbox SDK throws OperationInterruptedError (“Sandbox operation listProcesses was
interrupted while the platform was updating the sandbox runtime”) when the Durable Object isolate is
replaced — exactly what happens during a deploy / code update. It is a transient platform
lifecycle event, not an application failure, and it self-heals on the next request once the new
isolate is live.
Fix: nothing to fix. In findExistingGatewayProcess this is logged as a single concise
“platform was updating the sandbox runtime” line instead of a raw stack trace; /debug/processes
returns 503 platform_updating so debug tooling doesn’t misreport a crash. If it persists beyond
the deploy window, re-check wrangler tail and the container instance health in the Containers
dashboard.
5.7 Container terminated/destroyed → every request 500s or burns a cold-start budget
Symptom A — platform-terminated: the platform SIGTERMed the container (idle recycle,
maintenance, rescue) and after that all requests fail with [Gateway] Could not list processes:
plus sandbox.exec error … HTTP error! status: 500 — the container is dead and nothing ever
restarts it. The dashboard shows “Could not list processes” and the UI is fully bricked.
Symptom B — out-of-band destroyed (orphaned): the instance was destroyed from the Cloudflare
Containers dashboard (or a failed operator force-stop). The Sandbox DO is still alive but points
at a placement that no longer exists, so every request logs around no container instance available / maximum number of running container instances exceeded and each request burns the
SDK’s full provisioning retry budget (≈20s instance + 90s port) before giving up — CPU is spent
repeatedly trying to reach a container that can never be admitted, and the UI never recovers.
Symptom C — control-protocol mismatch (deploy cutover): during a deploy the new Worker bundle
carries a new @cloudflare/sandbox package while the still-running container predates the new
control handshake, so every request logs
sandbox.exec error … — 'utils.getRuntimeMetadata' is not a function. (or
RuntimeControlProtocolError / unsupported-protocol-version). This is the transient
stable↔1.0-preview skew described in AGENTS.md; it clears once the container rollout replaces
the instance.
Why: the Worker holds a Sandbox stub for a DO whose container instance is gone. Retrying any
RPC against that instance fails forever, and the old code swallowed the error and returned null
instead of reprovisioning — so no fresh container was ever created
(gateway/sandbox-recovery.ts).
Fix (automatic): maybeRecoverSandbox classifies all three failure classes and calls
recoverStaleSandbox (destroy the dead instance + reset the startup/persistence caches +
reset the orphan streak). The next RPC auto-provisions a fresh container and the normal cold-start
flow runs (R2 restore → start-openclaw.sh). Wired into findExistingGatewayProcess,
checkGatewayHealth, and the ensureGateway startShell retry (gateway/process.ts).
- Stale (
HTTP error! status: 500) recovers immediately (isStaleContainerError). - Control-protocol mismatch (
'utils.getRuntimeMetadata' is not a function/RuntimeControlProtocolError) also recovers immediately: the SDK logs thesandbox.exec error …line but re-throws the bareTypeError, soisStaleContainerErrormatches the message and reprovisions from the (now matching) image config instead of erroring until the platform rollout catches up. - Orphaned (
no container instance available) is retryable on its own, so it is only treated as an orphan once the failures spanORPHANED_INSTANCE_GRACE_MS(120s) — deliberately longer than the SDK’s ≈110s provisioning budget so a legitimately slow cold start is never cancelled mid-provision. The streak is cleared whenever a sandbox RPC succeeds (the container is demonstrably alive), so a healthy instance never trips the threshold later.
Recoveries are cooldown-guarded (STALE_RECOVERY_COOLDOWN_MS, 60s) to prevent destroy-storms
during request floods. Look for [SandboxRecovery] and repeated provisioning failures lines in
wrangler tail. Manual fallback: POST /debug/reprovision (works even when the gateway is gone),
or restart the container from the Containers dashboard.
NOT covered: permanent image errors (no such image) still require the rescue tool (§5.1),
and deploy-window OperationInterruptedError is expected and self-heals (§5.6).
5.8 Trace a Worker failure into its Sandbox process
The Tenant Worker writes structured worker.request.failed records for
unhandled request errors and sandbox.process.failed records when a Sandbox
command fails, times out, or cannot be launched. Worker logs also include
worker.request.started / worker.request.completed records with the method,
path, status, duration, and cfRay when provided by Cloudflare. Cloudflare adds
$metadata.requestId and $metadata.traceId to Worker log events automatically;
error records also pass an Error to console.error so $metadata.error and
exception details are populated.
In Workers Observability, filter the cloudflare-workers dataset by
event = "sandbox.process.failed" (or worker.request.failed). For a process
failure, copy its processId, then select the containers dataset and filter
on the same processId plus $container.applicationId. On process.complete,
use exitCode and timedOut as the outcome: the container’s generic message
may read success even when the process exited non-zero. The structured Worker
record intentionally excludes the command and process output to avoid leaking
credentials; use the existing ROOT-only debug/process diagnostics when those
details are needed. Persistence events (persistence.config-version.failed,
persistence.restore.failed, and storage.restore.failed) include backupId
when an archive exists; use it to correlate Worker errors with the container’s
backup.create / backup.restore events. persistence.backup.failed records
failures before the archive is necessarily created.
For CPU-limit errors during a backup or restore, filter Worker logs to
event = "persistence.phase". The paired records share an operationId; use
cfRayId to join request-triggered work to worker.request.*, or jobId for
ActivityScheduler work. The last started phase without a terminal record is
the stage to investigate first. See the Worker development guide
for the phase fields and common stage names.
6. Subsystem Deep-Dives
6.1 Working with Cloudflare Containers
- Image reference is built as
registry.cloudflare.com/<accountId>/<name>:<tag>(buildImageReference,diagnostics.ts:213).namedefaults tofouria(DEFAULT_CONTAINER_IMAGE_NAME). - Rescue needs the Containers API (
/accounts/{account}/containers/applications/...,createContainersApiClient,diagnostics.ts:235). It usesAuthorization: Bearer <CLOUDFLARE_API_TOKEN>and afull_autorollout. - Timeouts that keep a degraded/off container from hanging:
CONTAINER_INSTANCE_TIMEOUT_MS(20s),CONTAINER_PORT_TIMEOUT_MS(90s — kept generous for cold starts),CONTAINER_POLL_INTERVAL_MS(300ms) (config.ts:33). Override viaSANDBOX_INSTANCE_TIMEOUT_MS/SANDBOX_PORT_TIMEOUT_MS/SANDBOX_POLL_INTERVAL_MS(.dev.vars.example:78).
Container cost & lifecycle
The tenant container is a standard-1 instance (½ vCPU, 4 GiB memory, 8 GB disk).
Approximate 24/7 monthly cost (per Cloudflare Containers pricing):
| Resource | Provisioned | Approx. cost |
|---|---|---|
| Memory (4 GiB) | billed 24/7 | ~$26/mo |
| CPU (½ vCPU, ~10% used) | usage-based | ~$2/mo |
| Disk (8 GB) | billed 24/7 | ~$1.50/mo |
| Workers Paid plan | flat | $5/mo |
| Total | ~$34.50/mo |
Notes:
- CPU is billed on active usage, not provisioned capacity. Memory/disk are billed on provisioned capacity for the full uptime.
- Cold starts take 1–2 minutes; the first request after a sleep/wake is slow.
- Cost floor from the derived idle timeout — non-billable containers sleep when idle
instead of running 24/7. The timeout is derived from the instance hostname (staging 15m,
dev 2h). Before a non-billable container idle-sleeps, the Worker snapshots it (a durable
backup-then-stop job), so a sleep never discards uncommitted state. The billable
production environment does not auto-sleep; it is stopped only by an explicit lerma
freeze (which also snapshots first) and wakes on traffic. There is no
SANDBOX_SLEEP_AFTERsecret to set. Waking a sleeping container is theActivitySchedulerDO alarm’s job.
6.2 R2 issues
| Symptom | Likely cause | Action |
|---|---|---|
| Backup stored UNENCRYPTED warning | BACKUP_ENCRYPTION_KEY unset | Set the key (secret_text binding) |
| Backup fails closed | BACKUP_ENCRYPTION_REQUIRED=true, key missing | Provide the key |
| Restore of encrypted backup fails | key mismatch / missing | Ensure BACKUP_ENCRYPTION_KEY matches the one used at write time |
| Scheduled backup never runs | missing R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY / BACKUP_BUCKET_NAME / CLOUDFLARE_ACCOUNT_ID, or the tenant is never probed while healthy | Sandbox SDK presigned-URL path needs all four; hasBackupCredentials (persistence.ts) gates the scheduler, requirePresignedURLSupport throws InvalidBackupConfigError at snapshot time. Scheduled snapshots now fire on any healthy /api/health (passive included), so verify the lerma HealthMonitor actually polls the tenant |
Reprovision returns 409 | no restorable backup and the checkpoint failed | Expected fail-closed behaviour. Restore a checkpoint via POST /api/admin/storage/sync first; only pass {"force":true} when the container is already gone and its state is unrecoverable |
| Old config versions pile up | retention is 30 days; normal | cleanupOldConfigVersions prunes automatically |
More backups than maxVersions | a run of snapshots failed before their retention prune, so reconciled orphans accumulated | self-heals via the explicit reconcile: the durable repair step (and POST /api/admin/storage/reconcile) enforces retention through reconcileBackupCatalog, while GET /storage/backups itself never mutates |
R2 persistence uses the Sandbox SDK backup API; there is no s3fs mount to inspect.
Verify backup state with GET /api/admin/storage/backups and never delete tenant
backup objects outside retention or decommission procedures.
6.3 AI Gateway issues
The only provisioned AI provider path is Cloudflare AI Gateway, configured when all
three are set (start-openclaw.sh:161, config.ts:97):
npx wrangler secret put CLOUDFLARE_AI_GATEWAY_API_KEY # your gateway API key
npx wrangler secret put CF_AI_GATEWAY_ACCOUNT_ID # Cloudflare account id
npx wrangler secret put CF_AI_GATEWAY_GATEWAY_ID # gateway id within the account
Cloudflare AI Gateway gives caching, rate limiting, analytics, and cost tracking, and the
cloudflare-unified-billing plugin extends the catalog to 150+ models (Google Gemini,
Anthropic Claude, OpenAI, xAI Grok, DeepSeek, Alibaba Qwen, MiniMax, Moonshot Kimi, and
more) via the AI Gateway REST API. The plugin — plus the cloudflare-ai-gateway
provider and diagnostics-otel — is pre-installed into the image at build time from
ClawHub (unpinned = latest release; published at
https://clawhub.ai/paso4/plugins/cloudflare-unified-billing) so no network install runs
on the latency-critical boot path. start-openclaw.sh only installs from ClawHub as a
fallback when the extension state is missing. Lerma provisions the AI-scoped
credential to tenant workers as both CLOUDFLARE_AI_GATEWAY_API_KEY and the REST alias
CLOUDFLARE_API_TOKEN.
Premade agents:
Production deployments ship no premade agents — only the main agent written by
openclaw onboard. The dev-only Fouria Builder and Fouria Q&A agent are injected
by moltlazy only on dev instances (the Worker derives OPENCLAW_DEV_MODE from the
instance’s own -dev hostname, or DEV_MODE locally; they can self-install plugins and
read per-tenant client documentation materialized from R2). Neither agent is ever part of
a production instance: their purpose is to model the production agent gateway and
subagent squad (agents, skillsets, plugins, tools, automations) that satisfies the
client’s business logic. Because OpenClaw has no per-agent run timeout, dev instances
raise the global agents.defaults.timeoutSeconds to 1800 s (30 min) so the Builder is not
aborted mid-build; production keeps the 300 s baseline.
| Agent | Environment | Primary model | Purpose |
|---|---|---|---|
fouria-builder | dev only | Claude Sonnet 4.6 | Read client proposal/phases, build a working POC |
fouria-qa | dev only | Claude Sonnet 4.6 | Run cctr corpus validation against the Builder’s POC |
| Symptom | Cause | Action |
|---|---|---|
| Onboard skips AI Gateway auth | one of the three vars missing | set all three; mark NEED_ONBOARD=true |
cloudflare-ai-gateway plugin missing | build-time pre-install missing/removed | it is pre-installed in the image; start-openclaw.sh reinstalls from ClawHub as a fallback (bounded, non-fatal) |
cloudflare-unified-billing plugin missing | build-time pre-install missing/removed | it is pre-installed in the image; start-openclaw.sh reinstalls the latest @paso4/cloudflare-unified-billing from ClawHub as a fallback (unpinned, bounded, non-fatal) |
diagnostics-otel install fails | pre-install missing/removed | it is pre-installed in the image; the fallback installs the latest ClawHub build (unpinned). There are no version pins to bump |
| Model requests fail with auth errors | wrong/expired gateway key | rotate CLOUDFLARE_AI_GATEWAY_API_KEY |
| No embedding / memory search errors | no embedding key (CF_AI_GATEWAY_OPENAI_COMPAT_KEY) | memory search auto-disabled (start-openclaw.sh:422) — expected |
The cloudflare-unified-billing plugin (40+ models via the REST API) is pre-installed in
the image and enabled alongside the gateway provider; enablement is written by a single
config patch (which also enables admin-http-rpc and disables diagnostics-otel unless
its endpoint is set), not a plugins enable CLI call, to avoid a hanging ClawHub
registry refresh.
When migrating existing tenants from the old provisioner, re-provision the tenant and perform a cold restart after the AI credential is configured. Older R2 snapshots may contain a stale native-provider auth profile and the old random AI binding; a warm restore alone does not replace those persisted values.
6.4 Startup process issues
The startup sequence is start-openclaw.sh (STEPS array, start-openclaw.sh:69):
init → onboard → plugins → config-patch → config-validate → tunnel → gateway. Watch it
live via:
curl "https://<worker>/api/health" # status: starting while script runs
curl "https://<worker>/debug/gateway-logs" | tail -n 50| Symptom | Cause | Action |
|---|---|---|
Stuck at config-validate → startup aborts | invalid generated config | moltlazy validate locally; fix feature flags |
Stuck at onboard > 60s | AI Gateway plugin not present | the cloudflare-ai-gateway provider is pre-installed in the image (so its plugin phase is a no-op); if it still times out, check ClawHub/npm reachability — onboard is capped at 60s and continues, gateway starts --allow-unconfigured |
| Gateway never becomes ready (readiness gate times out) | enabled-but-missing provider plugin | the provider plugins are pre-installed in the image, so this should be rare; start-openclaw.sh falls back to installing the latest ClawHub spec when the extension state is missing (look for Installing … from ClawHub in /tmp/startup.log). The enablement config patch must not carry plugins.bundledDiscovery (removed in 2026.9.x — including it fails validation and drops the whole patch) |
| Gateway dies before port opens | gateway binary crash | GET /debug/gateway-logs for the crash; check EADDRINUSE. On Reason: database disk image is malformed the Worker queues the doctor --fix repair automatically and relaunches after it (see §5.4f); if the log shows SQLite integrity_check failed for …/state/openclaw.sqlite, run POST /debug/sqlite-integrity (then poll its job id) to confirm which database is corrupt and consider restoring an older verified backup instead of repairing the only copy |
STARTUP_TIMEOUT_MS (300s) exceeded | fallback plugin installs slow | plugins are pre-installed in the image so the normal boot skips installs; if it stalls, the fallback is doing network installs — check ClawHub reachability |
| Loading page never advances | startup-progress.json not written | check /tmp/startup.log via debug/gateway-logs |
The loading page renders from /tmp/startup-progress.json + /tmp/startup.log
(routes/public.ts:131, read bounded by STARTUP_PROGRESS_TIMEOUT_MS).
8. Cloudflare Platform Setup
One-time operator setup tasks for the current FourIA Worker deployment.
8.1 Cloudflare Access (protects /_admin, /api, /debug)
-
Enable Access on the worker — Workers & Pages dashboard → your Worker → Settings → Domains & Routes →
workers.dev→ Enable Cloudflare Access. Copy the Application Audience (AUD) tag. -
Configure who can access — Zero Trust → Access → Applications → your worker app → add allowed identities (email, Google, GitHub, …).
-
Set the secrets:
npx wrangler secret put CF_ACCESS_TEAM_DOMAIN # e.g. myteam.cloudflareaccess.com npx wrangler secret put CF_ACCESS_PLATFORM_AUD # platform AUD (always required) npx wrangler secret put CF_ACCESS_CLIENT_AUD # optional per-tenant AUDAlternatively create a Self-hosted Access application and protect paths
/_admin/*,/api/*,/debug/*.
/debug/*must be Access-protected. Never expose it publicly — it can stop, destroy, and rescue containers.
8.2 Browser Automation (CDP)
A Chrome DevTools Protocol shim backed by the Worker’s Cloudflare Browser
Rendering binding (BROWSER). Set:
npx wrangler secret put CDP_SECRET # shared secret for CDP auth
npx wrangler secret put WORKER_URL # worker public URL
npm run deploy| Endpoint | Description |
|---|---|
WS /cdp?secret=… | CDP WebSocket channel (commands + events) |
GET /cdp/json/version | CDP discovery (webSocketDebuggerUrl) |
GET /cdp/json/list | List available browser targets |
GET /cdp/json | Alias for /cdp/json/list |
All endpoints require ?secret=<CDP_SECRET>. The endpoint is a remote CDP
surface: point an OpenClaw profile at it with
browser.profiles.<name>.cdpUrl = "https://<worker>/cdp?secret=…" (default
openclaw driver) — see https://docs.openclaw.ai/tools/browser/remote. The
E2E coverage is test/e2e/r_browser_cdp.txt.
Tenant agents do not use this public route. On a tenant hostname the host Cloudflare Access application intercepts
/cdpwith a302login redirect, and no bypass is added. Moltlazy instead points the browser tool at the container-local proxy (http://127.0.0.1:9223), which forwards CDP commands to the Worker’s privatecdp.internalbridge; the browser session lives in the tenant’s Sandbox Durable Object and the Browser Rendering credential never enters the container. See WORKER-DEVELOPMENT.md § Browser Rendering andtest/e2e/r_browser_remote_profile.txt.
8.3 Obsidian Sync
The obsidian-sync skill runs an Obsidian LiveSync-compatible service on port 18790
inside the container. Tenant Cloudflare Tunnel support was removed: the container no
longer starts cloudflared, and Lerma provisions no CLOUDFLARE_TUNNEL_TOKEN binding.
The service is therefore reachable only from inside the container unless an operator
builds their own ingress path.
8.4 Built-in skills
The container ships pre-installed skills in /root/clawd/skills/, including
cloudflare-browser (documents the CDP connection pattern for the Worker’s
Browser Rendering shim; needs CDP_SECRET + WORKER_URL).
9. Requirements & References
Requirements
- Workers Paid plan ($5/mo) — required for Cloudflare Sandbox containers.
- An AI provider key via Cloudflare AI Gateway (API key, account id, gateway id) — the only supported AI path. Unified Billing is available as an alternative (docs).
Free tiers exist for Cloudflare Access, Browser Rendering, AI Gateway, and R2 Storage.
See also
- OpenClaw · OpenClaw Docs
- Cloudflare Sandbox · Cloudflare Access
- Worker deep-dive · Root DEVELOPMENT guide
- Env vars reference
7. Quick Reference
Environment variables (operator-relevant)
| Variable | Purpose | Notes |
|---|---|---|
DEBUG_ROUTES | Enables /debug/* | Set true only on operator/ROOT deploys |
CLOUDFLARE_API_TOKEN | AI Gateway REST auth; Containers:Edit for rescue | AI-scoped tenant alias; rescue token on ROOT/ops |
CONTAINER_APPLICATION_ID | Target app for image rollout | Operator-only |
CONTAINER_IMAGE_TAG / CONTAINER_IMAGE_NAME | Expected image unstable-<sha> / fouria | Operator-only |
CLOUDFLARE_AI_GATEWAY_API_KEY / CF_AI_GATEWAY_ACCOUNT_ID / CF_AI_GATEWAY_GATEWAY_ID | AI Gateway auth | Required for AI provider |
BACKUP_ENCRYPTION_KEY / BACKUP_ENCRYPTION_REQUIRED | R2 encryption at rest | Recommended true in prod |
BACKUP_BUCKET_NAME / R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY | R2 persistence | All four (incl. CLOUDFLARE_ACCOUNT_ID) required for scheduled backups; SDK signs R2 presigned URLs |
BACKUP_INTERVAL_MINUTES / BACKUP_MAX_VERSIONS | Backup cadence floor (auto-stretched to 2× the last snapshot duration) / retention | Defaults 5 / 3 |
BACKUP_CHANGE_DETECTION | Skips scheduled snapshots of unchanged tenant state (on by default) | Set false only to force a full snapshot every tick; the scan fails closed to “changed” |
BACKUP_EXPIRED on a catalogued archive | The SDK refuses to restore past createdAt + ttl; the archive is billed but unrestorable | Healed automatically: the durable repair step widens the TTL to 10 years and reports ttlExtended |
| “not encrypted” badge on a backup | The archive is not encrypted at rest (a legacy archive, or one whose encryption step never ran) | The next backup repairs it in place; it is never promoted to the recovery point |
SANDBOX_INSTANCE_TIMEOUT_MS / SANDBOX_PORT_TIMEOUT_MS / SANDBOX_POLL_INTERVAL_MS | Container fail-fast bounds | See §6.1 |
CLOUDFLARE_ACCOUNT_ID | Cloudflare account id (plain var) | Read by the Sandbox SDK for R2 presigned URL signing; legacy CF_ACCOUNT_ID is deprecated |
CF_ACCESS_TEAM_DOMAIN / CF_ACCESS_PLATFORM_AUD / CF_ACCESS_CLIENT_AUD | Cloudflare Access auth | Protects /_admin, /api, /debug |
CDP_SECRET / WORKER_URL | Browser automation (CDP) | See §8.2 |
STARTUP_LOG_VERBOSITY | steps/detailed/verbose boot log | Secret-free |
One-line recovery cheatsheet
# 1. Is it up? (no warm-up)
curl "https://<worker>/api/health?passive=1" | jq '{status,degraded,containerError}'
# 2. What's the lock state?
curl "https://<worker>/debug/gateway-lock" | jq .state
# 3. Degraded (no such image)? Rescue (dry first):
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":true}' | jq .
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":false}' | jq .
# 4. Gateway wedged but config OK? Stop, wait 8s, restart (via Admin API).
# 5. Lost state? Inspect backups, then restore by id or let next boot auto-restore.
This guide is generated from the source in apps/fouria/src/diagnostics.ts and its
dependencies (gateway/process.ts, gateway/health-probes.ts, gateway/lock.ts,
persistence.ts). When the behavior of
those modules changes, update this document in the same change.