# Fouria Operator Guide

---

## 1. Welcome

Fouria is a Cloudflare Worker that runs **OpenClaw** (the AI agent gateway) inside a
**Cloudflare Sandbox container**. Each customer gets an isolated container; the Worker
starts it, proxies chat traffic to it, snapshots its state to **R2**, and keeps it healthy.

Most of the time the system runs itself. This guide is for the moments when it does not:
when a container is stuck, a backup will not restore, the AI gateway rejects requests, or
the gateway will not start.

### The three surfaces you operate

| Surface                | Entry point                                | When to use                                                                                                      |
| ---------------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| **Health snapshot**    | `GET /api/health` (and `?passive=1`)       | Is the gateway up? What is its status?                                                                           |
| **Debug routes**       | `/debug/*` (only when `DEBUG_ROUTES=true`) | Deep inspection + emergency actions (stop, destroy, rescue, restart)                                             |
| **Diagnostics module** | `src/diagnostics.ts`                       | The code that powers container rescue; gateway restart lives in `src/gateway/process.ts` + `src/routes/debug.ts` |

### Mental model: the lifecycle

```
Request / cron / health probe
        │
        ▼
 ensureGateway()  ── finds or starts the container
        │
        ▼
 start-openclaw.sh  (onboard → moltlazy patch → plugins → gateway run)
        │
        ▼
 OpenClaw gateway  (port 18789, Control UI + WebSocket)
        │
        ▼
 createSnapshot() → R2  (state preserved between restarts)
```

When any link in this chain breaks, the tools in this guide are your recovery path.

---

## 2. Getting Started

### 2.1 Prerequisites

You should be comfortable with:

- Reading a JSON health response (`curl … | jq`).
- Cloudflare concepts: **Workers**, **Containers**, **R2**, **AI Gateway**.
- That the gateway runs **inside a container** — most "server is down" issues are really
  "container is down or the gateway process inside it is down".

### 2.2 Enabling diagnostics — how to turn on debug options

Debug tooling is **off by default in production** for safety. There are two layers:

#### Layer A — `DEBUG_ROUTES` (the /debug/* HTTP surface)

The `/debug/*` routes are gated by the `DEBUG_ROUTES` environment variable
(`apps/fouria/src/index.ts:379`, `src/types.ts:34`).

```bash
# In .dev.vars (local) or as a Worker var (production)
DEBUG_ROUTES=true
```

- `DEBUG_ROUTES !== 'true'` → every `/debug/*` call returns **404** immediately
  (`index.ts:292`).
- Setting it is **non-destructive**: it only exposes the inspection/action endpoints; it
  does not change runtime behavior.

> ⚠️ Never leave `DEBUG_ROUTES=true` on a tenant-facing deployment. It exposes
> `/debug/gateway-token`, `/debug/stop-gateway`, and `/debug/destroy-container`. Use it
> on operator/ROOT deployments only.

#### Layer B — `passive` health probes (no extra config)

The public health endpoint `GET /api/health` is always on. Append `?passive=1` to inspect
a down instance **without warming/cold-starting it** (`routes/public.ts:163`):

```bash
curl "https://<worker>/api/health?passive=1" | jq .
```

Passive probes use a short 3-second bound (`PASSIVE_PROBE_TIMEOUT_MS`, `config.ts:24`) and
never trigger `ensureGateway`, so a dead container fails fast instead of spinning up for
minutes.

### 2.3 Your first diagnostic call

```bash
# 1. Is the gateway healthy? (fast, no warm-up)
curl "https://<worker>/api/health?passive=1" | jq '.status, .degraded, .containerError'

# 2. Full active snapshot (warms the container, schedules a backup)
curl "https://<worker>/api/health" | jq .

# 3. Deep dive (requires DEBUG_ROUTES=true)
curl "https://<worker>/debug/version" | jq .
curl "https://<worker>/debug/processes?logs=true" | jq .
```

Read the **`status`** field — it is one of `running | starting | stopped`
(`health-probes.ts:67`). Read **`degraded`** — when `true`, a permanent container error
(e.g. deleted image) requires a rescue (see §6.1).

### 2.4 Manual snapshots

`POST /api/admin/storage/sync` queues a durable snapshot by default when the
`ACTIVITY` Durable Object is configured. It returns `202` with a `jobId` (or `409`
when a backup/restore job already owns the slot). Use `?inline=1` only when the
caller needs the completed snapshot response immediately, such as an E2E checkpoint;
`?async=1` retains the accepted-then-background behavior. Without `ACTIVITY`, the
endpoint falls back to inline work.

The shared R2 lease is valid for 12 minutes (`STEP_DEADLINE_MS`), renewed every 3
minutes and at slow snapshot/restore phase boundaries. A dead lease can consequently
take longer to recover than under the former 60-second TTL; this favors preventing
false expiry while a durable archive step is active. Watch for structured
`persistence.lock.renewal_late` and `persistence.lock.renewal_failed` events.

### 2.5 Rollout park (why admin writes answer `409` during an upgrade)

A container replacement always starts from a fresh disk, so the replacement
restores the tenant from the checkpoint taken just before the rollout. To make
that lossless the control plane parks the tenant first
(`POST /control` → `X-FourIA-Control: checkpoint-park`): the gateway is drained,
the stopped filesystem is snapshotted, and the tenant stays **parked** — every
mutating `/api/admin/*` or `/debug/*` request answers
`409 { code: "TENANT_PARKED" }`, and container exec is refused outright,
including background/spawned tasks. Read-only requests keep working.

The condition is durable (`maintenance/replacement.json` in the tenant's
`BACKUP_BUCKET`) and expires after 15 minutes, so a controller that dies mid-park
cannot wedge a tenant. A normal upgrade releases it at the `:resume` step
(`X-FourIA-Control: resume`, issued right after the container is replaced); the
replacement container then restores the checkpoint and boots the gateway. If an
operator sees `409 TENANT_PARKED` long after an upgrade finished:

1. `POST /control` with `X-FourIA-Control: resume` (Lerma does this automatically
   on success **and** on failure); or
2. wait out the 15-minute window — `getMaintenance` discards it and
   `ensureGateway` frees itself; or
3. as a last resort, delete `maintenance/replacement.json` from the tenant's
   `BACKUP_BUCKET` (only after confirming no rollout/snapshot is in flight).

Do **not** delete it while a checkpoint is running: the park is what stops a
write from landing after the archive was produced.

While parked you will also see:

- **Live agent sessions in `read-only`.** The park switches them before it stops
  the gateway and waits (up to two minutes) for their current turn to finish, so
  a turn is not interrupted mid-write. Their previous mode is recorded in
  `maintenance/session-modes.json` and put back on resume / after the restored
  gateway is up. New sessions are unaffected. If a session is still `read-only`
  after an upgrade, check that file first — a failed restore keeps it for the
  next boot (it is applied by `launchRestoredState`).
- **Telegram replies from the ingress.** A message sent during the park gets the
  "I'm being upgraded, please resend" notice from the ingress itself (no AI
  turn, no gateway), and the update is acked rather than queued.

### 2.6 Sandbox v1 migration gate (staging only)

The Sandbox SDK 1.0 cutover starts with an authority gate (contract C1/C2/C10).
`POST /control` with `X-FourIA-Control: migrate` is accepted **only** when the
trusted `WORKER_PLATFORM` binding is exactly `staging` **and** the tenant slug
resolves from trusted Worker env **and** the slug is not the reserved `client`.
Refusals happen before the sandbox stub is acquired:

| Condition                                   | Response                                                                                                                          |
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `WORKER_PLATFORM` absent/not `staging`      | `403 { code: "MIGRATION_PLATFORM_FORBIDDEN" }`                                                                                    |
| tenant slug unresolved (`default`, missing) | `409 { code: "MIGRATION_SLUG_UNRESOLVED" }`                                                                                       |
| reserved `client` tenant                    | `409 { code: "MIGRATION_RESERVED_CLIENT" }`                                                                                       |
| allowed staging tenant                      | `202 { state: "queued" \| "running" }`, `200 { state: "committed", recordId }`, or `502 { code: "SANDBOX_V1_CONVERSION_FAILED" }` |

`migrate` is **queue-only**: it enqueues a durable job on the `SandboxV1` Durable
Object and answers immediately — the conversion (legacy archive → v1 container →
`DirectoryBackup` → catalog + pointer) runs on the DO `alarm()`, so a tenant is
never converted by a request that could be cancelled. Poll with the separate
read-only action `X-FourIA-Control: migration-status` (Dispatch Worker:
`POST /_control/:slug/migration-status`), which only reads
`v1ConversionState()`: it never enqueues, never starts work and never wakes the
container, answers `200 { state: "absent" | "queued" | "running" | "committed" | "failed" }`
and never `202`. A `legacy` tenant reports `absent` without addressing a v1 DO.

Every other control action (`freeze`, `unfreeze`, `backup`, `checkpoint`,
`checkpoint-park`, `resume`) is **not** gated and keeps its existing behavior on
every platform, including `client`. Operators drive a conversion through Lerma
(`POST`/`GET /api/provisioning/migrate/:slug`, or
`mix lerma.sandbox_v1.migrate`), never by hand. `WORKER_PLATFORM` is provisioned
by Lerma from its own `cf_environment`; never derive it from the request host or
a header.

---

## 3. Operation Guidelines — Debug Options & How to Activate Them

### 3.1 The debug endpoint catalog

All under `/debug/*`, active only when `DEBUG_ROUTES=true`. Each maps to logic in
`src/diagnostics.ts` or `gateway/process.ts`.

| Method & Path                    | What it does                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Backed by                                                                             |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| `GET /debug/version`             | FourIA Worker version + git commit, plus OpenClaw + Node versions in the container                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | `__FOURIA_VERSION__` / `__FOURIA_COMMIT__` + `runShell` (`sandbox.exec`)              |
| `GET /debug/processes?logs=true` | List container processes (+ logs); returns `503 { status: "platform_updating" }` while the sandbox runtime is being updated (deploy)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `sandbox.listProcesses`                                                               |
| `GET /debug/gateway-api`         | Probe the OpenClaw HTTP API                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | `GATEWAY_API_TIMEOUT_MS`                                                              |
| `GET /debug/cli`                 | Run an OpenClaw CLI command                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | `startShell` (`sandbox.exec`)                                                         |
| `GET /debug/logs`                | Container stdout logs                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `sandbox`                                                                             |
| `GET /debug/env`                 | **Sanitized** environment (secrets masked)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | `maskEnvValues()`                                                                     |
| `GET /debug/container-config`    | OpenClaw config from inside container                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `sandbox.readFile`                                                                    |
| `GET /debug/gateway-logs`        | Gateway stdout (`/tmp/gateway-stdout.log`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | `sandbox.exec`                                                                        |
| `POST /debug/sqlite-integrity`   | Read-only `PRAGMA integrity_check` over every `*.sqlite` under `/root/openclaw/.openclaw` (e.g. `state/openclaw.sqlite` after a restore). Opens each file `readOnly`; never checkpoints, repairs, or removes WAL/SHM sidecars. Returns `ok`, per-file `messages`, and `sidecars`. Results are a snapshot while the gateway is writing — stop the gateway first for an authoritative check. Runs as a **durable maintenance job** (ActivityScheduler DO) because a large check outlives a Worker request; returns `202 { accepted, jobId }` and is polled via `GET /debug/sqlite-integrity/jobs/:id` or streamed over `GET /debug/sqlite-integrity/jobs/:id/ws` (runs inline when the `ACTIVITY` binding is absent) | `sandbox.exec` (`node:sqlite`, `readOnly`)                                            |
| `GET /debug/gateway-token`       | The gateway auth token                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `resolveSecret`                                                                       |
| `GET /debug/gateway-lock`        | OpenClaw native lock state                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | `gateway/lock.ts`                                                                     |
| `POST /debug/stop-gateway`       | Kill gateway **without** restarting                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `killGateway()`                                                                       |
| `POST /debug/restart-openclaw`   | Hard gateway restart (kill → clear startup cache → flag R2 restore → `start-openclaw.sh` reprovision)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `killGateway()` + `resetStartupCache()` + `signalRestoreNeeded()`                     |
| `POST /debug/reprovision`        | Force a container reprovision (checkpoint → **409 unless restorable backup or `force`** → destroy → clear startup/persistence caches → reset orphan streak); never touches the gateway                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `createSnapshot()` + `getBackupHistory()` + `sandbox.destroy()`                       |
| `POST /debug/destroy-container`  | Destroy the container entirely                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | `sandbox.destroy()`                                                                   |
| `POST /debug/sleep`              | Force the idle-sleep **deprovision** path (E2E seam): enqueue the durable backup-then-stop job the idle hook enqueues, so a container is never slept without a committed snapshot. `202 {jobId}`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `getActivityScheduler()` → `enqueueBackupJob({operation:'deprovision',mode:'sleep'})` |
| `POST /debug/rescue-container`   | Diagnose + rescue a stuck container                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `rescueContainer()`                                                                   |
| `POST /debug/backup/:mode`       | Trigger a checkpoint on demand (`mode` = `sync` \| `async`): `sync` blocks and returns the result (502 on failure); `async` acknowledges `202` and runs the snapshot via `waitUntil`. Idle container is skipped, never woken                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `ensureGateway()` + `scheduleBackupIfNeeded()`                                        |
| `POST /debug/r2-put?key=`        | Write raw content to R2 (tests)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `BACKUP_BUCKET.put`                                                                   |

> **Secret safety:** `GET /debug/env` runs every value through `maskEnvValues()`
> (`diagnostics.ts:31`). Any key matching `token`, `api_key`, `password`, `secret`,
> `credential`, `auth`, or `private_key` is replaced with `••••••••`. Never paste raw
> `/debug/env` output into tickets.

### 3.2 Reading the gateway lock (the most common "ghost" outage)

OpenClaw enforces **single-instance ownership** via three lock layers — a state lock, a
config lock, and an exclusive TCP bind on port `18789` (`gateway/lock.ts:1`,
`docs.openclaw.ai/gateway/gateway-lock`). A stale lock is the #1 cause of
`"gateway already running (pid …)"` / `EADDRINUSE` on restart.

```bash
curl "https://<worker>/debug/gateway-lock" | jq '.state'
```

`state` is one of (`lock.ts:27`):

| State               | Meaning                               | Operator action                               |
| ------------------- | ------------------------------------- | --------------------------------------------- |
| `free`              | No lock held — safe to start          | None                                          |
| `held-live`         | A live gateway owns the port          | Adopt it; do **not** kill                     |
| `stale-reclaimable` | Lock files present but owner PID gone | Auto-reclaimed on next start; restart is safe |
| `conflict`          | Lock error seen in the log            | Wait/restart; avoid parallel starts           |
| `unknown`           | Probe failed (container off)          | Check container health first                  |

The hard restart path checkpoints state before killing the gateway, then waits
for the next activation to restore that committed snapshot. OpenClaw's native
lock protects gateway ownership, but does not serialize filesystem mutations
performed by onboarding, plugins, configuration, or backup operations.

### 3.3 Emergency actions — when to use each

| Action                          | Use when                                                                                                     | Risk                                                                                                       |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| `POST /debug/stop-gateway`      | Gateway wedged but config is fine; you want to manually start fresh                                          | Low — does not destroy data                                                                                |
| `POST /debug/restart-openclaw`  | Config valid but gateway process dead; you want a clean in-place restart                                     | Low                                                                                                        |
| `POST /debug/reprovision`       | Container lost/orphaned (e.g. destroyed from the dashboard) but the DO is alive; gateway RPCs are timing out | Low–Medium — checkpoints first, **409 if no restorable backup** (override `force`); R2 restore keeps state |
| `POST /debug/destroy-container` | You want the platform to provision a **brand-new** container (fresh state)                                   | **High** — loses in-memory + unsnapshotted state                                                           |
| `POST /debug/rescue-container`  | Container is in `Error` mode / image deleted (see §6.1)                                                      | Medium — destroys stuck instance, keeps R2 backups                                                         |

### 3.4 The two diagnostic operations you will script most

Both live in `src/diagnostics.ts` and accept a **`dryRun`** option — always run dry first.

#### `POST /debug/restart-openclaw` — checkpointed hard gateway restart

Exposed as the diagnostic dashboard button (ROOT, `DEBUG_ROUTES=true`). It is the **hard**
restart path (the opposite of the Admin API's soft reload):

1. Find the live gateway process via `findExistingGatewayProcess(sandbox, { probeLive: true })`.
2. Refuse with **`409`** if the tenant is busy: a restore is in flight (or its outcome is still
   unknown after a timeout), or a maintenance window is open. Nothing is stopped.
3. Under the R2 lease, open a `restart` maintenance window and `killGateway(sandbox)` — `SIGTERM`
   with `GATEWAY_KILL_GRACE_MS` (5s) grace, `SIGKILL` only if still active; uses the `[o]penclaw`
   pkill bracket trick (no blanket `"openclaw"` pkill, no manual lock-file deletion — respects
   OpenClaw's own single-instance lock). The gateway is stopped **before** the archive is cut, so
   the checkpoint is consistent.
4. Create and commit a snapshot of `/home/openclaw`. If a restore is pending over a committed
   recovery point, the checkpoint is **refused** (`409`) before the gateway is stopped.
5. `resetStartupCache()` — clears the startup health cache.
6. `signalRestoreNeeded(BACKUP_BUCKET)` — flags R2 restore so the next request reprovisions a
   fresh gateway via `start-openclaw.sh`. The maintenance window closes only after this marker is
   set, so no request can relaunch the gateway on pre-restart state in between.

If the checkpoint fails before the gateway is stopped, the gateway is not touched. If it fails
after the stop, the gateway is restarted in place on the untouched filesystem and no restore is
armed. There is no direct binary launch; reprovisioning always goes through `start-openclaw.sh`.

> The **Admin API** `POST /api/admin/gateway/restart` is the **soft** reload: it sends `SIGUSR2`
> to the live `openclaw gateway run` process (`softReloadGateway`), then `resetStartupCache()`.
> When no live gateway is found it **starts** the gateway through the normal cold start
> (`ensureGateway`) and does **not** arm `restore-needed`: the live filesystem is kept, and the
> restore barrier still restores a container that has no state. It returns **`409`** while a
> maintenance window is open.
>
> `SIGUSR1` is **not** a restart signal: in current OpenClaw it starts Node's inspector on
> `127.0.0.1:9229` and leaves the gateway running. The signal must reach the gateway **server**,
> not the `openclaw gateway run` launcher: the launcher merely supervises the server, so
> `SIGUSR2` to the launcher kills it and orphans the server (PPID 1) without restarting anything,
> while the endpoint still answers `reloaded: true`. `buildSoftReloadScript` therefore matches the
> server's exact `comm` (`openclaw-gatewa`, Linux's 15-char truncation). Inside the container
> OpenClaw always takes the in-process restart path — `isContainerEnvironment()` reports
> `disabled` / `unmanaged` even without `OPENCLAW_NO_RESPAWN=1` — so soft reload never spawns an
> orphan either way. See
> [`../../docs/plans/sandbox-memory-usage.md`](../../docs/plans/sandbox-memory-usage.md) for the
> #463 decision record.

#### `POST /debug/reprovision` — force a container reprovision

The escape hatch for an **orphaned/dead container whose Sandbox Durable Object is still alive** —
most commonly after the instance was destroyed from the Cloudflare Containers dashboard, which
leaves the DO pointing at a placement that no longer exists.

0. **Busy guard** — refused with **`409`** while a restore is in flight (or its outcome is still
   unknown) or a maintenance window is open. Unlike `force`, this cannot be overridden: destroying
   the container mid-apply leaves a half-replaced filesystem. Re-checked after the checkpoint, in
   case a restore started while it ran.
1. **Checkpoint first** (`createSnapshot`) — only when `isRuntimeActive()` reports a live
   runtime (an idle container is **never woken** just to checkpoint it), and bounded by
   `REPROVISION_CHECKPOINT_TIMEOUT_MS` so an orphaned container's hanging RPC cannot block the
   route. If the container is gone or the snapshot times out, the route falls back to an
   **existing** restorable backup (verified via HEAD on the newest catalog entries'
   `meta.json`, not just the entry count). A snapshot abandoned by the timeout can finish after
   the destroy, but it only archives state that already existed, so it cannot corrupt anything.
2. **Fail closed**: if neither the checkpoint nor an existing backup is restorable, the route
   returns **`409`** and does _not_ destroy the container. Destroying it would permanently lose
   tenant state.
3. `clearPersistenceCache()` / `resetStartupCache()` — drop the per-isolate "already restored /
   startup ok" facts about the **old** container.
4. `clearProvisioningFailures(sandbox)` — reset the orphaned-instance failure streak.
5. `sandbox.destroy()` — tear down the instance so the next RPC provisions a fresh one.

Unlike `restart-openclaw`, this never talks to the gateway, so it still works when the container
is completely gone and gateway RPCs are timing out.

```bash
curl -X POST "https://<worker>/debug/reprovision" | jq .
```

When the container is already gone _and_ no backup exists (state is unrecoverable), pass the
explicit override:

```bash
curl -X POST "https://<worker>/debug/reprovision" \
  -H 'content-type: application/json' -d '{"force":true}' | jq .
```

#### `rescueContainer(sandbox, env, options)` — `diagnostics.ts:296`

Recovers a container stuck in `Error` mode (e.g. the `fouria:unstable-<sha>` image it
references was deleted by the stale-image cleanup cron). Sequence:

1. **Probe** container health (`probeContainerHealth`, short `PASSIVE_PROBE_TIMEOUT_MS`
   bound). Healthy → `{ action: 'none', ok: true }`.
2. **Repair image** via the Cloudflare Containers API (`createContainersApiClient` →
   `rolloutImage` with `strategy: 'full_auto'`) if credentials are present and the current
   image ≠ expected (`diagnostics.ts:339`).
3. **Destroy** the stuck container (`sandbox.destroy()`) so the platform provisions a fresh
   instance against the repaired image.

```bash
# Diagnose first (dry run — does nothing)
curl -X POST "https://<worker>/debug/rescue-container" \
  -H 'content-type: application/json' -d '{"dryRun":true}' | jq .

# Actually rescue (requires CLOUDFLARE_API_TOKEN + CONTAINER_APPLICATION_ID)
curl -X POST "https://<worker>/debug/rescue-container" \
  -H 'content-type: application/json' -d '{"dryRun":false}' | jq .
```

The result includes `action` ∈
`none | destroy | repair-image | repair-image-and-destroy | error`, a `diagnosis`
(health + `errorClass`), and the `image` report (`current`, `expected`, `repaired`,
`rolloutId`).

#### Error classification (what "permanent" vs "transient" means)

`classifyContainerError()` (`diagnostics.ts:166`) decides whether retrying helps:

- **Permanent** (no point retrying): `no such image`, `no matching app`, `no namespace
  configured`, `did not call start`, `out of memory`, `resource exhaustion`, `pid limit`.
  → a **rescue** (image repair + destroy) is required.
- **Transient** (recovers on its own): `connection refused`, `econnrefused`, `etimedout`,
  `timed out`, `not mapped`, `no container instance available`. → just wait / warm up.
- **unknown** → treat as transient unless health stays red.

This classification drives the `degraded` flag on the health snapshot (`health-probes.ts:281`):
a permanent error is what flips `degraded:true` and tells you a rescue is needed.

---

## 4. Working with Backups

State lives in the container at `/home/openclaw` and is preserved as **squashfs snapshots**
in **R2** (`persistence.ts`). The gateway restarts from the latest snapshot on boot.

### 4.1 The persistence model

| Concept             | Where                                       | Notes                                                                       |
| ------------------- | ------------------------------------------- | --------------------------------------------------------------------------- |
| Backup store handle | `backup-handle.json` in R2                  | Tracks up to `maxVersions` (default 3)                                      |
| Backup objects      | `backups/<id>/data.sqsh[.enc]`, `meta.json` | `.enc` when encrypted at rest                                               |
| Config versions     | `configs/<timestamp>.json`                  | Kept 30 days (`CONFIG_RETENTION_DAYS`)                                      |
| Restore marker      | `restore-needed` key                        | Durable restore-in-progress/retry latch; absent while idle or after success |
| Lock                | `backup-lock` (60s TTL, 20s heartbeat)      | Shared cross-isolate lease: serializes manual + scheduled snapshots         |

### 4.2 Encryption at rest (F4E1)

When `BACKUP_ENCRYPTION_KEY` is set, every snapshot is **post-encrypted** with AES-256-GCM
before it lands in R2 (`persistence.ts:405`, `createSnapshot`). Config versions under
`configs/` are encrypted with `customMetadata.encrypted='true'`.

```bash
# Generate a key (store it as a secret_text binding, NEVER in .dev.vars for prod)
openssl rand -base64 32
```

- `BACKUP_ENCRYPTION_KEY` unset → backups are stored **UNENCRYPTED** (warned in logs).
- `BACKUP_ENCRYPTION_REQUIRED=true` → backups **fail closed** if the key is missing
  (`persistence.ts:419`). Recommended in production.

> Legacy plaintext backups remain restorable; decryption is transient and deleted in a
> `finally` block (`persistence.ts:223`). See `docs/plans/r2-encryption-at-rest.md`.

### 4.3 How backups happen

- **Dispatch cron (running instances only)** — the Dispatch Worker's cron (every
  15 min production / hourly staging) fans a `backup` control action out to every
  non-frozen tenant. The tenant asserts `isRuntimeActive()` (a **non-waking**
  bridge read) and skips a sleeping container — an idle instance has no workload
  and no filesystem changes to capture, and waking it would restart billing. This
  is the backstop for tenants the lerma `HealthMonitor` does not poll. Before
  snapshotting, the persistence layer refuses to run while a `restore-needed`
  marker is present, rechecked under the shared lease, so a wake that lands
  mid-restore cannot snapshot and promote a partial filesystem over the
  committed recovery point.
- **On hard restart** — `POST /debug/restart-openclaw` first creates a checkpoint, then clears the startup cache and flags R2 restore.
- **Manual** — `POST /api/admin/storage/sync` calls `createSnapshot()` (explicit admin action).
- **Async admin** — `POST /api/admin/storage/sync?async=1` and
  `POST /api/admin/storage/restore/:id?async=1` commit a durable job to the
  `ActivityScheduler` Durable Object and return `202 { accepted: true, jobId }`.
  A DO alarm runs the job and retries it with a bounded lease, so the workflow
  survives the request ending, isolate eviction, and Worker redeploys (the old
  `waitUntil()` path was only guaranteed ~30s after the response). Poll
  `GET /api/admin/storage/jobs/:id` for `queued | running | retrying | succeeded
  | failed | skipped`.
- Health probes are **observability-only**: `GET /api/health` NEVER writes the backup catalog.
- `POST /debug/r2-put` only writes test objects; it does not create a snapshot.

**Stable recovery point.** A snapshot becomes the recovery point only once its
archive, optional F4E1 encryption, and catalog entry are all committed
(`stable-backup.json`). A cold restore prefers that pointer and refuses to
restore a newer archive whose promotion was interrupted, and retention always
keeps it — so a snapshot that failed partway can never evict the last known-good
recovery point. `GET /api/admin/storage/backups` reports it as
`stableBackupId`. This is a **checkpoint** guarantee, not zero-RPO replication:
writes acknowledged after the last committed snapshot are outside it (see
issue #377). The pointer only advances **forward** in time by default, so a stale
snapshot can never demote a newer committed one; a **targeted restore** is the one
exception (see §4.4).

Two guards keep the pointer safe in the other direction:

- **Restore pending.** While a `restore-needed` marker is present and a committed
  recovery point exists, a snapshot is still archived but does **not** move the
  pointer, and a pre-replacement checkpoint is **refused** (`409`). The live
  filesystem is not authoritative until the pending restore has applied.
- **Restore-only hold.** A targeted restore with `setCurrent=0` (§4.4) writes a
  `recovery-hold.json` marker. Scheduled snapshots keep archiving but do not move
  the pointer onto the restored state, so the restore is not silently undone within
  one backup interval. Any explicit snapshot (manual, checkpoint, hard restart) or a
  restore that sets the recovery point clears the hold. `GET /api/admin/storage/backups`
  reports it as `recoveryHeld`.

All three snapshot paths share **one tenant-scoped R2 `backup-lock` lease**, so only
one isolate ever stages the fixed legacy stage directory (`.fouria-legacy-stage`) or
calls `sandbox.createBackup` at a time. The lease has a 60s TTL and is renewed every
20s while a snapshot is in flight, and it is released by compare-and-swap to a
tombstone (never by deleting the lock object, which would race a concurrent takeover).
A live lease left behind when an isolate is killed is reclaimed after the 60s TTL;
the tombstone (`{acquired:0}`) is the normal released state, not a stuck lock.

> **Crash-consistency (issue #344).** A Worker isolate can be killed by its CPU
> budget after `sandbox.createBackup()` has uploaded `backups/<id>/…` but before
> the handle is written — leaving an orphaned, apparently-unrecoverable snapshot
> and a missing `backup-handle.json`. Two guards prevent/repair this:
>
> 1. The catalog entry is committed **immediately after the archive is created**
>    (after encryption, before the slower config-version capture and retention
>    prune), so an interrupted run still leaves a catalogued, restorable archive.
> 2. The catalog is **reconciled explicitly from `backups/*/meta.json`** by
>    `reconcileBackupCatalog` (the durable `repair` step,
>    `POST /api/admin/storage/reconcile`, `restoreIfNeeded`,
>    `scheduleBackupIfNeeded`, and the reprovision guard) whenever an archive is
>    missing from the handle, so pre-existing orphans self-heal. `getBackupHistory`
>    stays a **pure read**: a dashboard poll never lists, prunes, sweeps, or
>    writes. Retention is enforced on that reconcile path too, so an orphan adopted
>    after a run of failed snapshots cannot push the catalog past `maxVersions`.
>
> The scheduled `backup` control action additionally runs under a dedicated CPU
> budget (the Dispatch Worker's `BACKUP_CONTROL_CPU_MS`) so the full orchestration
> can finish instead of being killed mid-flight. **Freeze** now enqueues a durable
> backup-then-destroy job and acknowledges immediately (`202 { jobId }`) under the
> default budget; the caller polls **`GET /api/health/backup/:jobId`** (through the
> health Access tunnel) until the job reaches a terminal status instead of holding
> the request open. Unfreeze keeps the default.
>
> **Legacy-stage timeout (issue #344 follow-up).** The pre-snapshot legacy-state
> staging shell can exceed its bound on a large tenant (observed killing the
> `tarjetero` stage at exactly 300s). The bound is now `LEGACY_STAGE_TIMEOUT_MS`
> (default `600000`, i.e. 10 min), the idempotent stage is retried **once** on a
> transient timeout/transport drop, and a stage failure logs its exit code,
> timeout flag and bounded stderr. The Sandbox process is also reported to Worker
> Logs with its `processId` so the failure can be joined to the Containers
> `process.complete` event.
>
> The stage wait itself is **transport-resilient** (`runShellResilient`): a
> Sandbox DO↔container drop mid-stage reattaches to the process by id and resumes
> (bounded by the stage deadline) instead of ending the request. The Dispatch
> `backup` CPU budget (`BACKUP_CONTROL_CPU_MS`) is aligned to the same 10 minute
> bound.
>
> **Asynchronous control acknowledgment (issue #344).** Even with the larger
> budget, holding the `POST /control` request open for the whole snapshot
> outlived the dispatch/Worker request lifecycle, which cut the tenant request
> right after `createBackup` uploaded the archive and before the catalog write.
> The scheduled `backup` action now acknowledges immediately (`202
> { ok: true, accepted: true, jobId }`) and enqueues the snapshot as a durable job
> on the per-tenant `ActivityScheduler` DO, so the dispatch cron and operators
> never block on it. A `waitUntil` task is cancelled ~30s after the response
> (platform cap) and was observed dropping a snapshot that was still
> cold-booting, so the backup must not rely on it. The crash-consistent catalog
> commit + reconciliation above keep the archive durable
> if the isolate is still replaced mid-snapshot. To trigger either mode on demand
> (ROOT, `DEBUG_ROUTES=true`), use `POST /debug/backup/async` or
> `POST /debug/backup/sync`.

The two paths behave differently when the lease is busy:

- **Scheduled** acquisition is non-blocking: if another isolate holds the lease, this
  tick writes no backup and returns (it does not wait).
- **Manual** (`POST /api/admin/storage/sync`) and the **hard-restart checkpoint**
  wait for the lease with bounded exponential backoff (up to **120s**), then **fail
  with an error** rather than silently reporting success. A contended manual sync can
  therefore take up to 120s and then return an error; it never bypasses the lock.

A lease that cannot be proven free — held live, malformed, or unreadable — is treated
fail-closed: manual callers keep waiting and then time out, and scheduled callers skip
the tick. (Contention is bounded, not instant.)

#### Automatic backup trigger state machine

There is **one automatic snapshot trigger**: the **Dispatch Worker cron**, which
fans a `backup` control action out to every non-frozen tenant. The tenant only
proceeds when `isRuntimeActive()` reports a live runtime — an idle container is
never woken — after which the interval/lock gates below decide whether an object
lands in R2. Health probes play no part: `GET /api/health` is observability-only
and must not alter the catalog.

```mermaid
stateDiagram-v2
    direction TB

    [*] --> Idle

    state "Idle (no snapshot scheduled)" as Idle
    state "Runtime active? (isRuntimeActive)" as RuntimeGate
    state "Interval elapsed?" as IntervalGate
    state "backup-lock free?" as LockGate
    state "Acquire shared lease (wait ≤120s)" as Acquire
    state "Snapshotting (sandbox.createBackup)" as Snapshot
    state "Encrypting at rest (F4E1)" as Encrypt
    state "Committing catalog + pruning" as Commit
    state "No backup written" as NoBackup
    state "Reprovision guard" as ReproGuard
    state "Dispatch cron (backup control)" as DispatchCron

    Idle --> Acquire : hard restart checkpoint (POST /debug/restart-openclaw)
    Idle --> Acquire : manual sync (POST /api/admin/storage/sync)
    Idle --> ReproGuard : reprovision (POST /debug/reprovision)
    Idle --> DispatchCron : Dispatch Worker cron (15 min)

    DispatchCron --> RuntimeGate : backup control action
    RuntimeGate --> Idle : not running (never wake)
    RuntimeGate --> IntervalGate : running
    ReproGuard --> Snapshot : checkpoint (container alive)
    ReproGuard --> Idle : 409 — no checkpoint and no restorable backup
    IntervalGate --> Idle : < effective interval since last
    note right of IntervalGate : max(BACKUP_INTERVAL_MINUTES, 2 × last snapshot duration)
    IntervalGate --> LockGate : interval elapsed and BACKUP_BUCKET bound
    LockGate --> Idle : lease held (skip this tick)
    LockGate --> Snapshot : lease acquired (60s TTL, 20s heartbeat)

    Acquire --> Snapshot : lease acquired (60s TTL, 20s heartbeat)
    Acquire --> NoBackup : lease held > 120s (bounded error)

    Snapshot --> NoBackup : InvalidBackupConfigError / R2 error
    Snapshot --> Encrypt : archive created
    Encrypt --> NoBackup : BACKUP_ENCRYPTION_REQUIRED and key missing
    Encrypt --> Commit : encrypted (.enc) or plaintext (warned)
    Commit --> Idle : backup-handle.json written, oldest pruned
    NoBackup --> Idle : lease released (CAS tombstone)
```

The same state machine as an [Eraser](https://docs.eraser.io/docs/using-ai-agent-integrations)
flowchart. Render it by POSTing the DSL to the Eraser render API (see the
`eraser-diagrams` skill); it is kept inline so the diagram can be regenerated
without an API key.

```eraser
direction down

Start [shape: oval]
Idle [shape: rectangle, label: "Idle - no snapshot scheduled"]
RuntimeGate [shape: diamond, label: "Runtime active? (isRuntimeActive)"]
HardRestart [shape: rectangle, label: "Hard restart checkpoint"]
ManualSync [shape: rectangle, label: "Manual sync"]
ReproGuard [shape: diamond, label: "Reprovision guard"]
DispatchCron [shape: rectangle, label: "Dispatch cron (backup control)"]
NotRunning [shape: oval, color: gray, label: "Skip (never wake)"]
IntervalGate [shape: diamond, label: "Interval elapsed?"]
LockGate [shape: diamond, label: "backup-lock free?"]
Acquire [shape: rectangle, label: "Acquire shared lease (wait ≤120s)"]
Snapshot [shape: rectangle, label: "Snapshotting (sandbox.createBackup)"]
Encrypt [shape: rectangle, label: "Encrypting at rest (F4E1)"]
Commit [shape: rectangle, label: "Commit catalog + prune"]
NoBackup [shape: oval, color: red, label: "No backup written"]

Start > Idle
Idle > HardRestart
Idle > ManualSync
Idle > ReproGuard : "POST /debug/reprovision"
Idle > DispatchCron : "Dispatch Worker cron (15 min)"
DispatchCron > RuntimeGate : "backup control action"
RuntimeGate > NotRunning : "not running"
RuntimeGate > IntervalGate : "running"
ReproGuard > Snapshot : "checkpoint (container alive)"
ReproGuard > Idle : "409 - no checkpoint, no backup"
IntervalGate > Idle : "too soon"
IntervalGate > LockGate : "yes + BACKUP_BUCKET bound"
LockGate > Idle : "lease held (skip tick)"
LockGate > Snapshot : "acquired (60s TTL, 20s heartbeat)"
HardRestart > Acquire
ManualSync > Acquire
Acquire > Snapshot : "acquired (60s TTL, 20s heartbeat)"
Acquire > NoBackup : "lease held >120s (bounded error)"
Snapshot > NoBackup : "InvalidBackupConfigError / R2 error"
Snapshot > Encrypt : "archive created"
Encrypt > NoBackup : "encryption required, no key"
Encrypt > Commit : "encrypted or plaintext warned"
Commit > Idle : "catalog written, oldest pruned"
NoBackup > Idle : "lease released (CAS tombstone)"
```

| Automatic trigger        | Entry point                                                               | Gates that must pass                                                                                                                                                                                                                                   |
| ------------------------ | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Dispatch cron checkpoint | `POST /control` (`X-FourIA-Control: backup`) via the Dispatch Worker cron | tenant not frozen; **runtime active** (`isRuntimeActive()`, never wakes an idle container); `BACKUP_BUCKET` bound; effective interval elapsed (`max(BACKUP_INTERVAL_MINUTES, 2 × last snapshot duration)`); 60s lock free; SDK presigned creds present |
| Hard-restart checkpoint  | `POST /debug/restart-openclaw`                                            | none (unconditional checkpoint before kill)                                                                                                                                                                                                            |
| Reprovision guard        | `POST /debug/reprovision`                                                 | runtime-active checkpoint (time-bounded) **or** a restorable backup exists; otherwise `409` (override with `{"force":true}`)                                                                                                                           |
| Manual snapshot          | `POST /api/admin/storage/sync`                                            | caller authorized (ADMIN/ROOT) — explicit admin action                                                                                                                                                                                                 |

> **Operator traps that leave the bucket empty even though the worker is healthy.**
>
> First, the **Dispatch cron is the only automatic trigger**. A tenant missing
> from `TENANT_INDEX` (or marked frozen) is skipped. The lerma `HealthMonitor`
> is observability-only and never backs up.
>
> Second, the Sandbox SDK (>= 0.12.9) `createBackup()` on the remote path calls
> `requirePresignedURLSupport()`, which **throws `InvalidBackupConfigError` unless
> `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`, `BACKUP_BUCKET_NAME`, and
> `CLOUDFLARE_ACCOUNT_ID` are all set** on the tenant worker. The failure is caught
> by `scheduleBackupIfNeeded()` and logged as `[persistence] Scheduled backup failed:`
> — the R2 bucket then contains only `backup-handle.json` (or nothing at all).
>
> Third, snapshots only run against an **active runtime**. A sleeping container is
> skipped (never woken — it has no workload and no filesystem changes to capture),
> and `POST /debug/reprovision` returns `409` when the runtime is idle and no
> restorable backup exists.
>
> Verify all three before assuming backups are running.

To verify the R2 backup bucket is reachable without triggering a snapshot, use
`POST /api/admin/storage/test-connectivity` (ADMIN or ROOT). It HEADs the backup catalog
handle; a 404 (no backup written yet) still counts as connected. ROOT users can run it from
the Diagnostic Dashboard's **Test Backup Connectivity** button even though the Backups page
is ADMIN-gated.

The number of retained versions is `BACKUP_MAX_VERSIONS` (default 3); the oldest
beyond that are pruned. Retention is enforced on the snapshot path and by the
explicit catalog reconcile (`reconcileBackupCatalog`, run by the durable `repair`
step before every scheduled snapshot and by restore), so a run of failed snapshots
cannot leave the catalog or R2 over the cap. `GET /api/admin/storage/backups` is a
**pure read**; run `POST /api/admin/storage/reconcile` (ADMIN or ROOT) to adopt
orphaned archives, sweep dead ones, re-encrypt plaintext ones, widen an expired
archive TTL, and prune on demand.

**Retention is a count, never a clock.** The archive TTL (`SNAPSHOT_ARCHIVE_TTL_SECONDS`,
10 years) is not a storage-lifecycle hint: the SDK **refuses to restore** an archive
past `createdAt + ttl`. A short TTL therefore turns a retained recovery point into
a billed-but-unusable object, which is why the durable `repair` step widens it for
existing tenants (§5.4d).

**Archives are full copies of the container by default.** `BACKUP_ARCHIVE_EXCLUDES=true`
opts an instance into dropping regenerable paths before `mksquashfs` (`.npm`,
`*.log`, `*.tmp`, `.cache`, and `workspace/*node_modules*`), which shrinks the
archive but makes an archive-restored tenant differ from its pre-eviction tree
(and from a Tier 1 workspace snapshot). It is **off by default**; enable it per
instance only when you know those paths regenerate. Plugin dependencies, SQLite
databases and session transcripts are never excluded: a restore has no reinstall
step of its own.

### 4.3b Change-aware backups

Scheduled snapshots skip tenants whose state has not changed, and a per-tenant
watch can snapshot within minutes of a change instead of waiting for the next
15-minute dispatch tick. Detection is **on by default**; the watch loop is
opt-in. See `docs/WORKER-DEVELOPMENT.md` (Change-Aware Backups) for the mechanism.

- **Detection is already on.** A snapshot re-uploads the whole archive, so an
  unnecessary one dominates the backup bill: measured on `tenant-tarjetero-dev`,
  two snapshots 37 minutes apart differed by 0.08 % while both were 428 MB. Only
  an explicit `BACKUP_CHANGE_DETECTION=false` (lerma provisioning flag) opts out.
- **Enable the watch:** set `BACKUP_CHANGE_WATCH=true` for that lerma environment,
  then redeploy the affected tenants. It also binds the `BACKUP_WATCH` Durable
  Object.
- **Roll back:** set the flag to `false` and redeploy. Both flags are always emitted explicitly, so the rollback takes effect on redeploy.
- **Observe:** `[persistence] Scheduled backup skipped: tenant state unchanged` means the gate saved a snapshot. `[backup-watch] scan failed` (or `request failed`) means the watch fell back to a snapshot or retry. A scan failure is never a skipped backup.
- **Watch for cost:** each armed tenant runs one short `find` per check. The watch disarms after 10 idle minutes, so a quiet tenant is not kept awake.

### 4.4 Restoring

- **Automatic** — `restoreIfNeeded()` runs at container startup and restores the
  committed recovery point when the container has no tenant state. The
  `restore-needed` marker is not required to trigger a cold restore: its absence
  is normal while idle and after a successful restore. A cold restore writes the
  marker before applying the archive and clears it after restore plus legacy
  state materialization succeed. If candidate scanning finds no restorable
  archive and no stable recovery point exists, it clears the temporary marker
  for the permitted clean start. A recorded stable recovery point with no
  restorable archive instead fails closed and keeps the marker.
- **Targeted** — `restoreBackupById(sandbox, bucket, backupId, env, ctx, options)` restores a specific
  snapshot (`persistence.ts`). Use the backup history + config version list to pick one.
  A successful targeted restore **force-promotes** the restored archive to the recovery
  point (`stable-backup.json`) even when it predates the current one, so the next cold boot
  restores the same archive. Without this, `restoreIfNeeded` keeps preferring the newer
  pointer — which advances on every committed snapshot — and silently re-restores the wrong
  state (e.g. a post-deploy snapshot taken from a not-yet-materialized config, observed on
  `tenant-tarjetero-dev`).
- **Restore without moving the recovery point** — `POST /api/admin/storage/restore/:id?setCurrent=0`
  (also `?async=1&setCurrent=0`) passes `{ setRecoveryPoint: false }` through to
  `restoreBackupById`, which applies the archive but leaves `stable-backup.json`
  unchanged. Use this to inspect or recover an older snapshot with the next fresh
  container reverting to the current recovery point. The flag defaults to set-current
  (`?setCurrent=1` / omitted), matching the targeted-restore behavior above. The
  Backups page exposes it as the **Set as recovery point** checkbox (checked by
  default); unchecking maps to `setCurrent=0`. The restored state is then protected by a
  restore-only hold (§4.3), so scheduled snapshots cannot move the recovery point onto it.
- **After a restore the gateway is launched and the config is re-captured** — every restore path
  (the durable boot job, the durable restore job, and the synchronous admin route) finishes through
  `launchRestoredState` (`src/post-restore.ts`): it brings the restored state up
  (`ensureGateway({ waitForReady: true })`) and captures the live config as a new `configs/<ts>.json`
  plus `stable/config.json`, so the restored configuration becomes the **latest and stable** config
  version. A launch failure does not fail the restore (the restore already succeeded): the sync route
  returns `gatewayReady: false` and the durable job succeeds with `detail: "gateway-launch-pending"`,
  leaving the durable `restore-config-pending.json` marker for the next boot/maintenance tick. A
  `doctor --fix` repair is deferred (without consuming attempts) while a restore is queued/applying or
  the restored gateway is not up yet — otherwise its fix would be silently overwritten by the next
  re-apply of the recovery point (observed on `tenant-tarjetero-dev`).
- **Timeouts never start a second restore** — `sandbox.restoreBackup` is bounded by
  `RESTORE_BACKUP_TIMEOUT_MS`. The bound only stops the wait: the container call keeps
  running. A timeout therefore records the restore as **outcome unknown**, and every other
  restore is refused (`409`, `RestoreInFlightError`) until that record is older than the
  restore phase budget (`GATEWAY_RESTORE_PHASE_BUDGET_MS`, 11 min). A restore that fails
  definitively does not block a retry. The restore-needed marker stays set, so the next
  restore after the window re-applies the archive.
- **The recovery point's restorable TTL is healed on the restore path** — the SDK
  refuses an archive past `createdAt + meta.json.ttl` _before_ it touches the container,
  so an archive whose TTL lapsed fails every boot in ~250 ms even though its bytes are
  intact. `ensureArchiveRestorable` widens the TTL right before `sandbox.restoreBackup`
  (both the automatic and the targeted path), so a tenant whose catalog predates
  `SNAPSHOT_ARCHIVE_TTL_SECONDS` can still boot. See §5.4e for the operator view.
- **Writer quiesce + maintenance window** — `restoreBackupById` opens a
  cross-isolate maintenance window (`maintenance/replacement.json`, operation
  `restore`) and stops the gateway **immediately after it takes the shared restore
  lease**, then closes the window in a `finally`. A live gateway (or
  automations/channels) therefore cannot keep writing while the archive replaces the
  writable layer, and a concurrent health/proxy refetch cannot restart it
  mid-restore. `restoreIfNeeded` refuses to run while any maintenance window is
  active, which breaks the restore → stop-gateway → refetch → remount-the-latest
  loop; the window self-heals on its TTL if the controller dies.
- **Recovery visibility** — the Backups page marks the committed recovery point in
  the snapshot timeline (`GET /api/admin/storage/backups` → `stableBackupId`) with a
  **recovery point** badge and a header line (`A cold boot restores <id> — <date>`).
  The loading page names the backup + date being loaded and shows a **progress bar
  synchronized with the durable restore's phase events**: while a restore is in
  flight it reads the durable `restore-target.json` (the archive being applied) and
  otherwise the committed `stable-backup.json`, surfaced as `recovery.restoring` /
  `recovery.committed`, plus `restoreProgress` (`{ phase, label, fraction }`) from
  the `ActivityScheduler` DO, on `/api/health` (non-passive) and `/api/startup/ws`.
  So an operator can always tell _which_ snapshot a cold start is restoring, _when_
  it was taken, and _how far_ the restore has progressed, without opening logs.
  `GET /api/admin/boot-state` returns the same `restoreProgress` for inspection.

Restore and scheduled-backup safety rules:

- The pre-restore `umount` is **best-effort**: `umount … || true` only guards the shell
  command, not the exec RPC. A cold-start/platform interruption on that exec is caught and
  logged instead of aborting the restore before `restoreBackup` runs — the previous fatal
  behaviour kept `restore-needed` and failed the startup restore barrier on every boot.
- An empty catalog read does **not** clear `restore-needed`. The read can be transient (a
  reconcile `list` failure, or an archive not yet visible); a pending marker is kept so an
  operator-requested restore is never silently dropped. Restore failures keep the marker.
  If candidate scanning finds no restorable archive, a recorded stable recovery point fails
  closed and keeps the marker; with no stable pointer, the clean-start fallback removes it.
- If restore fails after writing partial state, `restore-needed` remains in R2 and
  forces a retry even when a config file is already present. Scheduled snapshots
  skip while that marker exists, rechecked under the backup lease, so a cron tick
  cannot overwrite the stable snapshot during a cold boot. The gate is keyed on
  the durable R2 marker (not a container-state probe) so an inconclusive probe can
  never silently disable scheduled backups.

Therefore, an absent `restore-needed` object by itself is not evidence that R2
lost a restore request. Check `stable-backup.json` and its referenced
`backups/<id>/meta.json`; those identify the committed recovery point. A fresh
container restores it based on the state probe even when the marker was absent.

Each entry in `backup-handle.json` records whether it was created with the SDK's
`localBucket` mode (`persistence.ts`). That flag is forwarded to `restoreBackup()` so a
local-binding backup restores through the local R2 binding instead of the
remote presigned-URL path — a backup created locally only exists in the local bucket, so a
remote restore 404s with `Presigned URL download failed (exit code 22)`.

Local-bucket mode is the explicit `BACKUP_LOCAL_BUCKET=true` opt-in; when unset it falls
back to `DEV_MODE` for local development. Do not rely on the `DEV_MODE` fallback for a real
deployment (including the cloud E2E, which sets `DEV_MODE=true` only to bypass Access): set
`BACKUP_LOCAL_BUCKET=false` there so backups land in the remote `BACKUP_BUCKET` the Worker
restores from.

Entries created before the flag was persisted carry no `localBucket`; on restore it is
defaulted from the same signal used at create time, so legacy local backups
still restore via the local binding. A restore that 404s at the presigned URL (archive
gone / unreachable) is treated like `BACKUP_NOT_FOUND` — the stale entry is dropped from
the catalog so one dead backup cannot block every subsequent boot.

Operator workflow for "I need to roll back to a known-good config":

1. List config versions: `GET /api/admin/storage/config` (or inspect `configs/` in R2).
2. Confirm the snapshot you want exists in `backup-handle.json`'s `backups[]`.
3. If the container is up, **stop the gateway** (`POST /debug/stop-gateway`), then call a
   targeted restore, then restart. If the container is down, the next boot auto-restores the
   newest.

> ⚠️ The current implementation uses the Sandbox backup API and the
> `BACKUP_BUCKET` R2 binding. It does not mount R2 with s3fs. Never delete tenant
> backup objects outside the retention or decommission procedure.

### 4.5 Setting up R2 (one-time)

Scheduled backups run when the Worker has its `BACKUP_BUCKET` binding and the
interval/lock guards pass (`scheduleBackupIfNeeded()` in `persistence.ts`).
The bound bucket is provisioned per tenant, for example
`fouria-backup-{slug}{-environment}{-platform}`.

> **The Sandbox SDK remote path requires R2 S3 credentials.** In
> `@cloudflare/sandbox >= 0.12.9`, `createBackup()` (and `restoreBackup()`) on
> the non-`localBucket` path calls `requirePresignedURLSupport()`, which throws
> `InvalidBackupConfigError` unless **all** of `R2_ACCESS_KEY_ID`,
> `R2_SECRET_ACCESS_KEY`, `CLOUDFLARE_ACCOUNT_ID`, and `BACKUP_BUCKET_NAME` are
> present on the tenant Worker. `hasBackupCredentials()` only gates on
> `BACKUP_BUCKET`, so a misconfigured worker still runs the scheduler and then
> fails at snapshot time (logged as `[persistence] Scheduled backup failed:`)
> with an empty bucket. Only local-bucket mode (`BACKUP_LOCAL_BUCKET=true`, or the
> `DEV_MODE` fallback for local development) avoids the credential requirement.

1. **Bind the tenant bucket** — the Lerma provisioner creates and binds the
   tenant R2 bucket as `BACKUP_BUCKET` and sets `BACKUP_BUCKET_NAME`.
2. **Let lerma resolve it (it is now mandatory)** — the provisioner resolves a
   credential per instance and **fails provisioning** (rolling back) when none
   can be produced; it no longer warns and deploys a tenant that silently cannot
   back up. Strategy is `R2_TOKEN_STRATEGY` (`auto` | `shared` | `per_instance`):

   - `auto` (default): uses `R2_TOKEN_MAX_PER_ACCOUNT`; a known limit of at most
     250 selects `shared`, a larger or unknown limit selects `per_instance`.
     Cloudflare exposes no maximum-token endpoint, so the limit defaults to
     unknown → **per-instance token minted at instance creation**, scoped to the
     tenant bucket.
   - `shared`: set `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` in the lerma
     environment (generate with `mix lerma.r2_token --all-buckets`).

   Generated per-instance credentials are persisted on the instance record and
   reused across redeploys; a token minted for a failed brand-new provisioning
   is deleted on rollback. `CLOUDFLARE_ACCOUNT_ID` is a plain var and is always
   injected.

3. **Ensure the client is being probed** — the lerma `HealthMonitor` is the only
   automatic trigger and only polls clients in `production`/`error`. A client in
   `draft` gets no scheduled snapshots regardless of credentials.

   ```bash
   # For a standalone/legacy Worker (not provisioned via lerma):
   npx wrangler secret put R2_ACCESS_KEY_ID
   npx wrangler secret put R2_SECRET_ACCESS_KEY
   ```

> Sandbox snapshot behavior differs between deployed and local environments;
> local tests use the SDK's `localBucket` mode when `BACKUP_LOCAL_BUCKET=true`
> (or the `DEV_MODE` fallback) in `persistence.ts`. The flag is persisted in the
> backup catalog so restores route to the same mode — a local backup restores via
> the local R2 binding, and a remote (presigned-URL) backup restores via
> `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` / `CLOUDFLARE_ACCOUNT_ID` /
> `BACKUP_BUCKET_NAME`. Pre-persist entries default to the current mode. There is
> no s3fs mount.

---

## 5. Troubleshooting — Critical Issues

This section maps symptoms → root cause → the exact recovery from this guide.

### 5.1 Container stuck in `Error` / image deleted → **RESCUE**

**Symptom:** `/api/health?passive=1` returns `degraded:true`,
`containerError:"no such image"` (or `no matching app`). The instance can never boot.

**Why:** the stale-image cleanup cron deleted the `fouria:unstable-<sha>` image the
application references (`diagnostics.ts:42`). The cleanup script now protects every
tag still referenced by a container application — including the versions of an
in-progress rollout — and aborts (fail closed) when that in-use set cannot be
resolved (`.github/scripts/cleanup-stale-images.sh`). Production deploys are also
restricted to stable release tags, so a production application can no longer be
pinned to the `unstable-<sha>` channel (`Lerma.Cloudflare.ImageRegistry.stable?/1`).

**Fix:** `rescueContainer` (§3.4):

```bash
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":true}' | jq .
# confirm action would be repair-image-and-destroy, then:
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":false}' | jq .
```

**Credentials required:** `CLOUDFLARE_API_TOKEN` (Containers:Edit),
`CONTAINER_APPLICATION_ID`, and either `CONTAINER_IMAGE_TAG` or an explicit
`expectedImage` (`diagnostics.ts:304`, `.dev.vars.example:43`). Without them, the image
**cannot** be repaired — only the container is destroyed (you then must re-deploy the
correct image tag).

**NEVER set these on tenant workers** — the rescue tool re-points the _application_ image
and is operator/ROOT-only (`.dev.vars.example:40`).

### 5.2 Gateway "already running" / `EADDRINUSE` on restart

**Symptom:** restart fails with `gateway already running (pid …)` or `failed to bind gateway
socket`.

**Why:** a previous gateway still holds the native lock / TCP bind when the new one
starts (`gateway/process.ts`).

**Fix:** for a clean state use `killGateway()` (the `POST /debug/stop-gateway` route), which
`SIGTERM`s with the `GATEWAY_KILL_GRACE_MS` (5s) grace and only `SIGKILL`s if still active,
then wait for the port to close before reprovisioning. `POST /debug/restart-openclaw` (hard
restart) does this then reprovisions via `start-openclaw.sh`. Inspect lock state with
`GET /debug/gateway-lock`.

### 5.3 Gateway process invisible but container live

**Symptom:** `listProcesses()` shows nothing, yet `/health` answers.

**Why:** the gateway runs as a **detached child** of `start-openclaw.sh`
(`process.ts:427`). The SDK does not track it, but it is healthy.

**Fix:** the live-gateway fallback (`findExistingGatewayProcess` with `probeLive:true`)
detects it via a bounded `/health` probe. Prefer restart/rescue APIs that use this path
rather than `destroy-container` (which would throw away a healthy instance).

### 5.4 Backup restore loops / "hadBackup but not restored"

**Symptom:** health `startup.ok=false` permanently with
`Backup found but not restored`.

**Why:** `restoreIfNeeded` found an entry but every backup expired/was missing
(`persistence.ts:212`). The code explicitly reports a **clean** start in this case so it
does not gate startup forever (`persistence.ts:233`).

**Fix:** confirm real backups exist (`backup-handle.json` → `backups[]`); if none, the
instance boots fresh (expected). If backups exist but restore fails, check
`BACKUP_ENCRYPTION_KEY` (encrypted backups cannot be read without it).

**Manual-restore loop (`restore → stop-gateway → refetch → remount the latest`).**
A manual restore of an older archive kills the gateway so no writer mutates the
layer being replaced. Without a guard, the next health/proxy refetch calls
`ensureGateway` → `restoreIfNeeded`, which prefers the committed recovery point
(the _newer_ archive) and silently undoes the restore. Two guards prevent this:
`restoreBackupById` holds a cross-isolate `maintenance/replacement.json` window
(operation `restore`) for the whole restore, during which `ensureGateway` refuses
to launch, and `restoreIfNeeded` itself refuses to run under any active window.
A targeted restore that should survive cold boots also force-promotes the archive
(the default); a `setCurrent=0` restore deliberately does not, and is reverted on
the next fresh container.

### 5.4b Stuck gateway start ("already in progress" forever)

**Symptom:** logs repeat `[Gateway] Gateway start already in progress for
openclaw , waiting...` for tens of minutes while `listProcesses returned N
processes` keeps climbing and the port stays closed (health probes `curl` exit 7
every ~5s). The gateway never binds 18789.

**Why:** the shared `ensureGateway` start promise spans restore → supervised
`start-openclaw.sh start` → port wait. If the wrapper never `exec`s the gateway
(or the container control channel drops), the promise does not settle and every
request waits on it. The raw `listProcesses().length` cannot distinguish this
from a busy boot — it counts `exited` processes too and only grows.

**Automatic recovery:** the Worker now tracks the start phase and force-resets a
phase that overran its budget, logging a structured **`gateway.start.stuck`**
event and killing the stuck process; after repeated resets it reprovisions the
container. Before resetting a `port-wait`, it makes a fresh `/health` check: a
gateway that became healthy after the original readiness deadline is adopted
(`gateway.start.late-ready`) instead of killed. If that recheck cannot reach the
container, health is unknown and reset is deferred until a later request can
probe again. A `start-openclaw.sh start` wrapper older than the cold-start budget
with the port closed is also killed immediately.

**Manual recovery (if needed):** `POST /debug/reprovision` (ROOT, `DEBUG_ROUTES=true`)
to destroy + reprovision the container, or `POST /debug/restart-openclaw` to
hard-restart the gateway in place; a freeze/unfreeze from lerma also forces a
fresh cold start.

**Investigate:** filter the `cloudflare-workers` dataset to
`event = "gateway.start.stuck"`, `event = "gateway.start.phase"`,
`event = "gateway.start.late-ready"`, or
`event = "gateway.start.health-recheck-failed"` — records carry the phase,
attempt, elapsed time, and process id needed to distinguish a real stuck start
from a late healthy gateway or an inconclusive probe.

### 5.4c Repeated same-size snapshots and a container that never sleeps

**Symptom:** R2 keeps receiving a full archive of the **same size** every few
minutes/hours (e.g. 422 MB at 02:35/02:39/02:44 and 05:39/05:43/05:49 UTC), the
container instance shows `running` with a `started_at` days old, and
`backup-handle.json` has a stale `lastScheduledBackup` even though new entries
appear. Often paired with a cost spike at a quiet hour.

**Why:** the idle-sleep `deprovision` job snapshots **before** stopping. On a
large tenant that archive takes minutes, and its own RPCs renew the Sandbox DO
activity window, so the stop-time `stopIfIdle()` re-check refuses
(`active:activity`). The SDK re-arms, the next idle expiry enqueues another
deprovision, and each attempt uploads the full archive again. Deprovision
archives used to be unstamped, so the scheduled cadence did not advance either.

**Fix (current bundle):** `TenantSandbox.onActivityExpired()` first proves that an
existing archive already covers the current filesystem, and stops inline without
a new archive when it does:

- a committed snapshot inside the idle window (+5 min slack), or
- with change detection on (the default), `unchanged` from the same change scan
  the scheduled path uses — a quiet tenant takes no scheduled snapshot, so its
  newest archive can be older than the idle window even though nothing changed.

A deprovision archive also stamps `lastScheduledBackup`. If an older tenant still
loops, redeploy it to the current bundle. Diagnose with:

```bash
npx wrangler r2 object get fouria-backup-<slug>/backup-handle.json --remote --file -
npx wrangler r2 object get fouria-backup-<slug>/backups/<id>/meta.json --remote --file -
```

**Related — 30-minute dev heartbeats.** OpenClaw 2026.10.5 retired
`agents.entries.*.heartbeat`: the heartbeat is an ordinary automation job and
config reloads never rewrite it, so a migrated dev instance can keep the default
30-minute cadence (a full agent turn on the **main** session each time). The
dev-only `ensure_dev_heartbeat_cadence` in `start-openclaw.sh` reconciles it on
post-start; to fix a warm instance immediately:

```bash
openclaw automations edit heartbeat-fouria-builder \
  --every 12h --session isolated --light-context
```

### 5.4d "The gateway rebooted — was it memory?"

**Symptom:** the gateway cycled (2–5 boots in a day), the dashboard dipped to
`starting`/`503`, and someone asks whether it hit the container memory limit.

**Answer it from the Worker first — no SSH needed.** `/api/health` (and
`/api/health-snapshot` / `/api/health-ws`) carries a `gateway` block for a running
gateway:

```jsonc
"gateway": { "pid": 4242, "rssBytes": 2234000000, "threads": 29 }
```

RSS is sampled from `/proc/<pid>/status` at most once a minute, so it is cheap to
poll. The same numbers land in Analytics Engine as a `gateway-resources` row
(`double1` = RSS bytes, `double2` = threads, blob1 = `gateway-resources`, blob2 =
customer) — chart it to tell a **monotonic leak** from a **transient peak**:

```sql
SELECT toStartOfHour(timestamp) AS hour,
       max(double1) / 1048576 AS rss_mib, max(double2) AS threads
FROM fourai_cost_metrics
WHERE blob1 = 'gateway-resources' AND blob2 = 'tarjetero'
  AND timestamp > NOW() - INTERVAL 1 DAY
GROUP BY hour ORDER BY hour
```

Reference points on a `standard-1` container (4 GiB, no swap):

| Reading                  | Meaning                                                                                               |
| ------------------------ | ----------------------------------------------------------------------------------------------------- |
| ~1.1–1.4 GiB             | normal baseline right after a hard restart                                                            |
| ~2.1 GiB (peak ~2.6 GiB) | observed steady state of a busy **dev** gateway; still healthy                                        |
| ≥ 1.5 GiB                | OpenClaw's own `rss_threshold` **warning** — expected to fire at this level; it is not an action item |
| ≥ 3 GiB                  | the dev watchdog threshold — the gateway gets replaced                                                |

**Before blaming memory, rule it out properly.** An OOM kill leaves fingerprints
that a graceful restart does not:

```bash
# inside the container (wrangler containers ssh)
grep oom_kill /proc/vmstat            # must stay 0
dmesg | grep -i "killed process"      # empty
grep -ciE "heap out of memory|Allocation failed" /tmp/gateway-stdout.log
grep -c "restart-signal fence" /tmp/gateway-stdout.log   # graceful self-restarts
grep -c "host timing gap detected" /tmp/gateway-stdout.log
```

A **non-zero `restart-signal fence` / `received SIGTERM`** count means the gateway
restarted itself on purpose (doctor repair, config reload, update) — not OOM. A
**`host timing gap detected: process was frozen ~Ns`** line means the platform
froze the microVM; a freeze long enough to blow past startup deadlines is followed
by the supervisor replacing the process, which looks like a reboot but is not a
memory event. Neither of those is an OOM.

**Dev-only memory watchdog.** On a dev instance (`-dev` hostname /
`OPENCLAW_DEV_MODE`) the Worker replaces the gateway when RSS reaches **75% of
the container's memory limit**, at most once per 30 minutes, and logs:

```jsonc
{
  "event": "gateway.memory.pressure",
  "rssMib": 3072,
  "threads": 31,
  "limitMib": 3072,
  "containerLimitMib": 4096,
}
```

The threshold scales with the instance: 3 GiB on a 4 GiB `standard-1`, 4.5 GiB on
a 6 GiB `standard-2`, 6 GiB on an 8 GiB `standard-3`, 9 GiB on a 12 GiB
`standard-4`. It is derived from the container's cgroup limit on every sample, so
an instance-type rollout changes it without a Worker deploy. Instances below
3 GiB (`basic` is 1 GiB) are **not** restarted — they cannot hold a healthy
gateway, so the watchdog would only restart-loop; `standard-1` is the practical
floor for a tenant that runs agents.

If you see that event, the watchdog fired **before** the kernel OOM killer could
— that is the intended behaviour, not an outage. It is deliberately gated to dev:
a replacement interrupts live agent sessions. Production keeps today's behaviour
(no automatic replacement).

**Fix the cause, not the symptom.** RSS growth is driven by oversized persisted
transcripts and accumulated `worker_thread` V8 heaps that an in-process restart
never reclaims; see
[`../../docs/plans/sandbox-memory-usage.md`](../../docs/plans/sandbox-memory-usage.md)
for the mitigation set and what is still open.

### 5.4e A recovery point that is billed but cannot be restored (`BACKUP_EXPIRED`)

**Symptom:** `backup-handle.json` and the Backups page list an archive, the R2
object is present and billed, but a restore logs
`Backup <id> has expired (created: … TTL: 604800s)` → `BackupExpiredError` →
`[persistence] Backup <id> expired/gone, removing from store`, and if it was the
recovery point the boot fails closed with `TenantStateUnavailableError`.

**Why:** the SDK treats `meta.json.ttl` as a hard restorability gate
(`createdAt + ttl`), not a storage-lifecycle hint. Retention prunes by **count**,
so an idle, frozen, or repeatedly failing tenant could keep `maxVersions`
archives that no boot could ever apply — while the dashboard still showed them as
the recovery point.

**Fix (current bundle):** new archives are written with
`SNAPSHOT_ARCHIVE_TTL_SECONDS` (10 years), and the durable `repair` step heals
existing tenants: `extendArchiveTtl` rewrites every `meta.json` below the minimum
(preserving `dir`, `createdAt`, `encrypted`, `encryptionVersion`) and reports
`ttlExtended` / `ttlFailed` from `POST /api/admin/storage/reconcile`. The restore
path also heals its own archive (`ensureArchiveRestorable`, right before
`sandbox.restoreBackup`), because the `repair` step only runs inside a snapshot
job — a tenant that cannot boot can never reach the repair that would make it
bootable.

Verify with:

```bash
npx wrangler r2 object get fouria-backup-<slug>/backups/<id>/meta.json --remote --file -
# ttl should be 315360000 (10 years), not 604800
```

**Unblock a tenant that is already wedged** (no deploy needed — this is R2-only
and works with the gateway down). Note the **`-X POST`**: `GET` on that path does
not match the route and falls through to the gateway proxy, which answers
`503 {"error":"Gateway not ready","details":"Tenant boot pending …"}`:

```bash
curl -X POST https://<tenant-host>/api/admin/storage/reconcile \
  -H "cookie: CF_Authorization=<access-jwt>"
# → { "success": true, "ttlExtended": ["<id>", …], … }
```

Then reload the tenant page: the next boot heals the archive and applies it. The
bytes are unaffected — only the SDK's restore guard reads `meta.json.ttl`.

**Symptom you will see instead if you skip this:** the loading page sits on
`Restore attempt failed during "Restoring the backup" — retrying — no progress
for N min`. That is a **fast failure**, not a slow restore: the SDK rejects the
expired archive in ~250 ms (one `meta.json` read), the durable boot job retries
three times 30 s apart (~70 s total) and then goes terminal `failed`, and the page
keeps rendering the last attempt's record. Read the bucket to confirm —
`restore-target.json` with `"outcome":"failed"`, `restore-needed` still present,
and a `backup-lock` tombstone (`"acquired":0`) with nothing written afterwards.
Current bundles replace that stale label with `Restore failed — <reason>` and
`Boot failed after N attempts: <reason>. Refresh the page to retry.`

### 5.4f OTLP billing and promotion show no effect (bridge / R2 transports)

Neither feature has an internet route any more, so "reachability" is not the
first question. There is no `/api/otel/*` and no `/api/admin/releases/import`:
diagnose them from the inside.

| Symptom                                         | Cause                                                                                                                                | Fix                                                                                                                                                               |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| no traces at all                                | `OTEL_TRACES_ENABLED` is not `true`                                                                                                  | set it on the tenant Worker; `diagnostics.otel.traces` is off by default                                                                                          |
| exporter gets `401`                             | `OTEL_SHARED_SECRET` is unbound (or differs from the one you replay with)                                                            | bind it on the tenant Worker (lerma platform secret)                                                                                                              |
| `403 TENANT_MISMATCH`                           | the Worker is not bound to the tenant slug                                                                                           | verify `CUSTOMER_SLUG` / `WORKER_NAME` on the tenant Worker                                                                                                       |
| no `billable-action` rows                       | the batch reached the bridge but had no billable span (check the response's `emitted`/`nonBillable`)                                 | confirm the span name/channel and the plan's `action_type`                                                                                                        |
| promotion says "staged" and nothing happens     | production has not booted/applied yet, or is parked / inside a maintenance window                                                    | cold-start it, or `POST /api/admin/releases/apply`; check `GET /api/admin/releases/status` (`available`, `release`)                                               |
| production never sees the release               | `PROMOTION_SOURCE_BUCKET` missing (older tenant) or targeting the wrong tenant/environment                                           | re-provision/upgrade the production instance so the sibling import bucket binding is added                                                                        |
| release applied but old agents remain           | production booted from an older snapshot with an older container pointer                                                             | the next boot re-reads `pending.json`; force `POST /apply`                                                                                                        |
| promotion "succeeded" but instances never start | the promoted image had no runtime artifact: instances sit in `starting`, then `failed` with `ImagePullError` (10-minute pull budget) | re-run the promotion (lerma prepares the image before rolling and gates on live health); check `wrangler containers images list` and the app's `health.instances` |

Verify from inside the container (the bridge is a virtual host, not a public
URL):

```bash
# 401 = secret missing/wrong, 200 = bridge reachable and ingest accepted
curl -sS -o /dev/null -w '%{http_code}\n' -X POST http://otel.internal/v1/traces \
  -H "authorization: Bearer $OTEL_SHARED_SECRET" -H 'content-type: application/json' -d '{}'
```

#### Where promotion runs (Worker API, not the container)

Promotion is an **operator API action on the tenant Worker**. Nothing runs inside
the container or through the OpenClaw CLI: the container is only the source (dev)
and the target (production) of the file copy.

| Step                                                                    | Instance       | Runs in                                                                                                                                                                  | Who                                     |
| ----------------------------------------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------- |
| Stage — `POST /api/admin/releases/promote`                              | **dev**        | the dev Worker: exports the dev container's `/home/openclaw` (config + workspace) and writes `promotion/staged/<id>.tar.gz` + `promotion/pending.json` to the dev bucket | operator session                        |
| History — `GET /api/admin/releases/history`                             | **both**       | the Worker: dev reads its own staged entries; production merges the sibling dev bucket with its own applied/failed entries                                               | operator session                        |
| Inspect — `GET /api/admin/releases/status`                              | **production** | the production Worker (container pointer + the sibling bucket)                                                                                                           | operator session                        |
| Apply — `POST /api/admin/releases/apply`, **or** a production cold boot | **production** | the production Worker: pulls the bundle, writes it into the production container, re-applies the `gateway` section and soft-reloads                                      | operator, or the boot job automatically |

- `promote` is deliberately dev-only: on a production instance it answers
  `409 NOT_DEV_INSTANCE`. `apply` on the dev instance finds nothing
  (`wrong-environment`), because the staged pointer targets `production`.
- An agent inside the dev container (e.g. the Fouria Builder) **cannot** perform
  the promotion: `/api/admin/*` sits behind the host Access app and requires a
  resolved ADMIN/ROOT role, which a container process does not have. It prepares
  the artifact and hands the operator the commands.
- The only container-adjacent step is the production boot-time apply, and that is
  still driven by the Worker (`ensureGateway`) _before_ the gateway launches —
  never by OpenClaw itself.

```bash
# dev instance: stage with a commit message (202 { staged: true })
curl -sS -X POST https://<dev-host>/api/admin/releases/promote \
  -H 'content-type: application/json' \
  -d '{"message":"Ship the new intake flow"}'
# both instances: push history (short hash, message, author, status)
curl -sS https://<dev-host>/api/admin/releases/history
curl -sS https://<prod-host>/api/admin/releases/history
# production instance: inspect, then apply (or just cold-start it)
curl -sS https://<prod-host>/api/admin/releases/status
curl -sS -X POST https://<prod-host>/api/admin/releases/apply
```

Every push is content-hashed (SHA-256 of the bundle) and carries the deployer's
commit message plus the verified Access identity as the author. The history on
the dev instance lists the pushes it staged; the production instance sees those
staged entries through its read-only sibling binding and records its own
`applied`/`failed` outcome, so both sides can follow the same sequence without
either bucket being writable by the other.

#### Promotion from the dashboard (`/_admin/releases`)

The same operations are available in the admin SPA — **Governance → Releases**
(ADMIN-only) — once the operator has the same Access session the `curl` calls
need. Open the tenant's dev host first and press **Stage release for
production**; then open the production host and press **Apply release now**.
The page shows the release waiting in the sibling bucket, the current release
pointer (`status: applied` is the only proof), the promoted/excluded agents, and
the post-apply verification checklist.

Two states the page surfaces explicitly, both worth checking before an apply:

- **"No `PROMOTION_SOURCE_BUCKET` binding"** — production cannot see the staged
  release. Re-provision/upgrade production (lerma creates the dev bucket if
  missing); staging again will not help.
- **A skipped apply with a reason** — `parked` (rollout in progress: resume it),
  `busy:<phase>` (backup/restore/maintenance window: apply later),
  `nothing-pending` / `already-applied` / `bundle-missing`. The API is
  idempotent, so re-pressing **Apply** after the blocker clears is safe.

The page is Worker-local (no gateway dependency), so it works while the
container is asleep.

#### Promotion runbook — the real prerequisites

Every operator route on both tenant hosts sits behind the host-level Access
application. `GET`/`POST` without a credential answers `302` to
`<team>.cloudflareaccess.com` with `www-authenticate: Cloudflare-Access`, and
there is **no path bypass to provision** (the `/import` machine path no longer
exists). Check only these:

| Prerequisite              | Detail                                                                                                                                                                                                                                                                                                                            |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Access credential         | A **browser/session identity** whose JWT resolves a role: ADMIN on production, and on staging a **Paso4 Root** session (Dispatch's staging gate is ROOT-only). An Access **service token** (`CF-Access-Client-Id`/`Secret`) clears the edge but resolves **no role** — the Worker answers `403`, so it cannot drive these routes. |
| Confirmed target          | Stage on the tenant's dev host (`https://<slug>-dev…`) and apply on its production host. For the staging tenant that is `https://staging-client.fouria.io`; never `https://client.fouria.io`.                                                                                                                                     |
| `PROMOTION_SOURCE_BUCKET` | The **production** instance must carry the read-only binding to its sibling dev bucket. A tenant provisioned before the binding existed stages forever without applying — re-provision/upgrade production to add it (lerma creates the dev bucket if missing).                                                                    |
| Applied pointer           | `GET /api/admin/releases/status` must show `available` before `POST /apply`; after applying it shows the release with `status: "applied"`. Nothing to do on the dev side after `202 { staged: true }`.                                                                                                                            |

`PROMOTION_PUSH_TOKEN` is **not** part of this model: there is no Worker→Worker
credential to provision, and no `/api/admin/releases/import` to protect.

> **Analytics Engine is never touched from the container.** The sandbox exports
> OTLP to the bridge; the Worker writes the `billable-action` row with its
> `ANALYTICS` binding — no credential involved — and the platform reconciles
> credits from its own AE credentials. No Analytics Engine credential is
> forwarded into the sandbox, and no agent, skill or software-manifest entry
> should expect one. If a build step asks for an AE token inside the container,
> it is asking for the wrong thing.
>
> `ANALYTICS_ENGINE_API_TOKEN` is **not needed for emission** and is no longer
> provisioned by Lerma: it only powers the optional dashboard read-back, and
> without it the widget reports `available: false` (usage is platform-aggregated).

### 5.4g The launch loops on a corrupt state database ("Starting OpenClaw Gateway" forever)

**Symptom:** every boot attempt ends within seconds with
`[openclaw] Reason: database disk image is malformed` (stack:
`readSqliteUserVersion` → `inspectStateDatabaseSchema` in
`openclaw-state-schema-preflight`), and the loading page keeps saying
`Still working on "Starting OpenClaw Gateway" (NNNs)` — the elapsed counter keeps
growing because the container's `startup-progress.json` still names the launch
step from the last attempt.

**Why:** a restored recovery point can carry a damaged
`state/openclaw.sqlite` (or an agent `openclaw-agent.sqlite`). OpenClaw's schema
preflight reads `PRAGMA user_version` before anything else, so the Gateway exits
immediately; relaunching cannot help, and before this change each attempt also
burned the full ~5-minute cold-start budget waiting for a port that would never
open.

**Automatic recovery:** the launch now detects the crash (a non-zero exit of the
supervised `start-openclaw.sh start` process before the port opens) and matches
the diagnostics against the corruption signatures. It queues the durable
`doctor-repair` job (`reason: gateway-launch-state-database`), which:

- runs `openclaw doctor --fix --non-interactive` with the Gateway stopped (the
  repair preserves the original databases before changing anything) **without**
  waiting for a gateway the corruption prevents from launching, and
- re-queues the durable boot when it succeeds, so the Gateway relaunches on the
  repaired state.

While that runs, `/api/startup/ws` reports `repair` and the loading page shows
"Repairing the OpenClaw state database (openclaw doctor --fix, attempt N)…".
Repeated failed launches fold into the same job (`409`-equivalent
"maintenance job busy"), so nothing piles up.

**Investigate:** filter the `cloudflare-workers` dataset for
`event = "gateway.launch.state-database-repair"` (outcome `queued` / `busy` /
`no-scheduler` / `failed`, with the matched `signature` and the job id) and
`event = "sandbox.process.failed"` with `failureReason =
"state-database-corrupt"` (plus `exitCode`). The job itself is visible at
`GET /api/admin/config/fix/jobs/<id>` (or its `/ws` stream), and its result
carries the `doctor` summary. If the repair fails, `POST /debug/sqlite-integrity`
tells you which database is still unhealthy, and restoring an older verified
backup (`POST /api/admin/storage/restore/<id>`) is the safe fallback.

### 5.5 Platform & setup troubleshooting

| Symptom                                                                                                                                                                         | Likely cause                                                                                                                                                                                                               | Action                                                                                                                                                                                                                                                                                                                                                                                     |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `npm run dev` / deploy: `Unauthorized`                                                                                                                                          | Cloudflare Containers not enabled for the account                                                                                                                                                                          | Enable Containers in the [Containers dashboard](https://dash.cloudflare.com/?to=/:account/workers/containers)                                                                                                                                                                                                                                                                              |
| `[sandbox-do] backup.restore … Presigned URL download failed (exit code 22) … 404`                                                                                              | restore hit the remote presigned-URL path for an archive that only exists in the local R2 binding (a `BACKUP_LOCAL_BUCKET`/`localBucket` backup), or `BACKUP_BUCKET_NAME` doesn't match the `BACKUP_BUCKET` binding bucket | redeploy so the backup catalog persists `localBucket` (§4.4) — local backups restore via the local binding; for remote backups verify `R2_ACCESS_KEY_ID`/`R2_SECRET_ACCESS_KEY`/`CLOUDFLARE_ACCOUNT_ID`/`BACKUP_BUCKET_NAME` all point at the same bucket as the `BACKUP_BUCKET` binding (§4.5)                                                                                            |
| `[sandbox-do] backup.restore error <id> — The operation was aborted (~30 s)` repeated on every boot, with `sandbox.exec … The operation was aborted` for health probes/`umount` | the archive restore ran under a **request** execution context, which Cloudflare cancels shortly after the response, so `sandbox.restoreBackup` was aborted mid-flight (large archives, e.g. 350 MB, never finish)          | fixed: request paths defer the restore to the durable `ActivityScheduler` boot job (`RestoreDeferredToBootJobError` → `GatewayBootPendingError`). Verify with `grep -c "await restoreIfNeeded" apps/fouria/src/index.ts` (must be 0) and look for `[Gateway] Restore deferred to the durable boot job`; the restore then runs in the DO alarm and shows `Durable boot job began/succeeded` |
| Gateway won't start                                                                                                                                                             | missing/incorrect secrets                                                                                                                                                                                                  | `npx wrangler secret list` + `npx wrangler tail`                                                                                                                                                                                                                                                                                                                                           |
| Config changes not taking effect                                                                                                                                                | stale Docker build cache                                                                                                                                                                                                   | bump the `# Build cache bust:` comment in `Dockerfile`, then redeploy                                                                                                                                                                                                                                                                                                                      |
| Slow first request                                                                                                                                                              | cold start (1–2 min)                                                                                                                                                                                                       | expected; later requests are faster                                                                                                                                                                                                                                                                                                                                                        |
| R2 not mounting                                                                                                                                                                 | an R2 secret missing, or running the local Vite dev server                                                                                                                                                                 | set all four R2 secrets (§4.5); R2 mount is production-only                                                                                                                                                                                                                                                                                                                                |
| Access denied on `/_admin` or `/api`                                                                                                                                            | `CF_ACCESS_TEAM_DOMAIN` / `CF_ACCESS_AUD` not set                                                                                                                                                                          | configure Cloudflare Access (§8.1)                                                                                                                                                                                                                                                                                                                                                         |
| Devices not appearing in admin UI                                                                                                                                               | device-list CLI takes 10–15 s (WS overhead)                                                                                                                                                                                | wait and refresh                                                                                                                                                                                                                                                                                                                                                                           |
| WebSocket issues in local dev                                                                                                                                                   | local `vite dev` WS-proxy limitation                                                                                                                                                                                       | deploy for full WS functionality                                                                                                                                                                                                                                                                                                                                                           |
| Windows: script exits `126`                                                                                                                                                     | CRLF line endings in shell scripts                                                                                                                                                                                         | add `.gitattributes` with `* text=auto eol=lf`                                                                                                                                                                                                                                                                                                                                             |

### 5.6 Deploy-window "Could not list processes" / platform interrupted

**Symptom:** logs show `[Gateway] Could not list processes:` followed by a stack trace ending in
`createPlatformInterruptedError` / `translatePlatformInterruption`, and/or
`/debug/processes` returns a `503` with `{ "status": "platform_updating" }`.

**Why:** the Sandbox SDK throws `OperationInterruptedError` ("Sandbox operation listProcesses was
interrupted while the platform was updating the sandbox runtime") when the Durable Object isolate is
replaced — exactly what happens during a **deploy / code update**. It is a transient platform
lifecycle event, not an application failure, and it self-heals on the next request once the new
isolate is live.

**Fix:** nothing to fix. In `findExistingGatewayProcess` this is logged as a single concise
"platform was updating the sandbox runtime" line instead of a raw stack trace; `/debug/processes`
returns `503 platform_updating` so debug tooling doesn't misreport a crash. If it persists beyond
the deploy window, re-check `wrangler tail` and the container instance health in the Containers
dashboard.

### 5.7 Container terminated/destroyed → every request 500s or burns a cold-start budget

**Symptom A — platform-terminated:** the platform SIGTERMed the container (idle recycle,
maintenance, rescue) and after that **all** requests fail with `[Gateway] Could not list processes:`
plus `sandbox.exec error … HTTP error! status: 500` — the container is dead and nothing ever
restarts it. The dashboard shows "Could not list processes" and the UI is fully bricked.

**Symptom B — out-of-band destroyed (orphaned):** the instance was destroyed from the Cloudflare
Containers dashboard (or a failed operator force-stop). The Sandbox DO is still alive but points
at a placement that no longer exists, so every request logs around `no container instance
available` / `maximum number of running container instances exceeded` and **each request burns the
SDK's full provisioning retry budget (≈20s instance + 90s port) before giving up** — CPU is spent
repeatedly trying to reach a container that can never be admitted, and the UI never recovers.

**Symptom C — control-protocol mismatch (deploy cutover):** during a deploy the new Worker bundle
carries a new `@cloudflare/sandbox` package while the still-running container predates the new
control handshake, so every request logs
`sandbox.exec error … — 'utils.getRuntimeMetadata' is not a function.` (or
`RuntimeControlProtocolError` / `unsupported-protocol-version`). This is the transient
stable↔1.0-preview skew described in `AGENTS.md`; it clears once the container rollout replaces
the instance.

**Why:** the Worker holds a Sandbox stub for a DO whose container instance is gone. Retrying any
RPC against that instance fails forever, and the old code swallowed the error and returned `null`
instead of reprovisioning — so no fresh container was ever created
(`gateway/sandbox-recovery.ts`).

**Fix (automatic):** `maybeRecoverSandbox` classifies all three failure classes and calls
`recoverStaleSandbox` (destroy the dead instance + reset the startup/persistence caches +
reset the orphan streak). The next RPC auto-provisions a fresh container and the normal cold-start
flow runs (R2 restore → `start-openclaw.sh`). Wired into `findExistingGatewayProcess`,
`checkGatewayHealth`, and the `ensureGateway` `startShell` retry (`gateway/process.ts`).

- **Stale (`HTTP error! status: 500`)** recovers immediately (`isStaleContainerError`).
- **Control-protocol mismatch** (`'utils.getRuntimeMetadata' is not a function` / `RuntimeControlProtocolError`)
  also recovers immediately: the SDK logs the `sandbox.exec error …` line but re-throws the bare
  `TypeError`, so `isStaleContainerError` matches the message and reprovisions from the (now
  matching) image config instead of erroring until the platform rollout catches up.
- **Orphaned (`no container instance available`)** is retryable on its own, so it is only treated
  as an orphan once the failures span **`ORPHANED_INSTANCE_GRACE_MS` (120s)** — deliberately longer
  than the SDK's ≈110s provisioning budget so a legitimately slow cold start is never cancelled
  mid-provision. The streak is cleared whenever a sandbox RPC succeeds (the container is
  demonstrably alive), so a healthy instance never trips the threshold later.

Recoveries are cooldown-guarded (`STALE_RECOVERY_COOLDOWN_MS`, 60s) to prevent destroy-storms
during request floods. Look for `[SandboxRecovery]` and `repeated provisioning failures` lines in
`wrangler tail`. Manual fallback: `POST /debug/reprovision` (works even when the gateway is gone),
or restart the container from the Containers dashboard.

**NOT covered:** permanent image errors (`no such image`) still require the rescue tool (§5.1),
and deploy-window `OperationInterruptedError` is expected and self-heals (§5.6).

### 5.8 Trace a Worker failure into its Sandbox process

The Tenant Worker writes structured `worker.request.failed` records for
unhandled request errors and `sandbox.process.failed` records when a Sandbox
command fails, times out, or cannot be launched. Worker logs also include
`worker.request.started` / `worker.request.completed` records with the method,
path, status, duration, and `cfRay` when provided by Cloudflare. Cloudflare adds
`$metadata.requestId` and `$metadata.traceId` to Worker log events automatically;
error records also pass an `Error` to `console.error` so `$metadata.error` and
exception details are populated.

In Workers Observability, filter the `cloudflare-workers` dataset by
`event = "sandbox.process.failed"` (or `worker.request.failed`). For a process
failure, copy its `processId`, then select the `containers` dataset and filter
on the same `processId` plus `$container.applicationId`. On `process.complete`,
use `exitCode` and `timedOut` as the outcome: the container's generic message
may read `success` even when the process exited non-zero. The structured Worker
record intentionally excludes the command and process output to avoid leaking
credentials; use the existing ROOT-only debug/process diagnostics when those
details are needed. Persistence events (`persistence.config-version.failed`,
`persistence.restore.failed`, and `storage.restore.failed`) include `backupId`
when an archive exists; use it to correlate Worker errors with the container's
`backup.create` / `backup.restore` events. `persistence.backup.failed` records
failures before the archive is necessarily created.

For CPU-limit errors during a backup or restore, filter Worker logs to
`event = "persistence.phase"`. The paired records share an `operationId`; use
`cfRayId` to join request-triggered work to `worker.request.*`, or `jobId` for
ActivityScheduler work. The last `started` phase without a terminal record is
the stage to investigate first. See the [Worker development guide](/llms/platform/apps/fouria/worker-development/index.md#cloudflare-error-and-trace-correlation)
for the phase fields and common stage names.

---

## 6. Subsystem Deep-Dives

### 6.1 Working with Cloudflare Containers

- **Image reference** is built as
  `registry.cloudflare.com/<accountId>/<name>:<tag>` (`buildImageReference`,
  `diagnostics.ts:213`). `name` defaults to `fouria` (`DEFAULT_CONTAINER_IMAGE_NAME`).
- **Rescue needs the Containers API** (`/accounts/{account}/containers/applications/...`,
  `createContainersApiClient`, `diagnostics.ts:235`). It uses `Authorization: Bearer
  <CLOUDFLARE_API_TOKEN>` and a `full_auto` rollout.
- **Timeouts** that keep a degraded/off container from hanging: `CONTAINER_INSTANCE_TIMEOUT_MS`
  (20s), `CONTAINER_PORT_TIMEOUT_MS` (90s — kept generous for cold starts), `CONTAINER_POLL_INTERVAL_MS`
  (300ms) (`config.ts:33`). Override via `SANDBOX_INSTANCE_TIMEOUT_MS` / `SANDBOX_PORT_TIMEOUT_MS`
  / `SANDBOX_POLL_INTERVAL_MS` (`.dev.vars.example:78`).

#### Container cost & lifecycle

The tenant container is a **`standard-1`** instance (½ vCPU, 4 GiB memory, 8 GB disk).
Approximate 24/7 monthly cost (per [Cloudflare Containers pricing](https://developers.cloudflare.com/containers/pricing/)):

| Resource                | Provisioned | Approx. cost   |
| ----------------------- | ----------- | -------------- |
| Memory (4 GiB)          | billed 24/7 | ~$26/mo        |
| CPU (½ vCPU, ~10% used) | usage-based | ~$2/mo         |
| Disk (8 GB)             | billed 24/7 | ~$1.50/mo      |
| Workers Paid plan       | flat        | $5/mo          |
| **Total**               |             | **~$34.50/mo** |

Notes:

- CPU is billed on **active usage**, not provisioned capacity. Memory/disk are billed on
  **provisioned capacity** for the full uptime.
- **Cold starts take 1–2 minutes**; the first request after a sleep/wake is slow.
- **Cost floor from the derived idle timeout** — non-billable containers sleep when idle
  instead of running 24/7. The timeout is derived from the instance hostname (staging 15m,
  dev 2h). Before a non-billable container idle-sleeps, the Worker snapshots it (a durable
  backup-then-stop job), so a sleep never discards uncommitted state. The **billable
  production environment does not auto-sleep**; it is stopped only by an explicit lerma
  freeze (which also snapshots first) and wakes on traffic. There is no `SANDBOX_SLEEP_AFTER`
  secret to set. Waking a sleeping container is the `ActivityScheduler` DO alarm's job.

### 6.2 R2 issues

| Symptom                           | Likely cause                                                                                                                                      | Action                                                                                                                                                                                                                                                                                                                                         |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Backup stored UNENCRYPTED warning | `BACKUP_ENCRYPTION_KEY` unset                                                                                                                     | Set the key (secret_text binding)                                                                                                                                                                                                                                                                                                              |
| Backup fails closed               | `BACKUP_ENCRYPTION_REQUIRED=true`, key missing                                                                                                    | Provide the key                                                                                                                                                                                                                                                                                                                                |
| Restore of encrypted backup fails | key mismatch / missing                                                                                                                            | Ensure `BACKUP_ENCRYPTION_KEY` matches the one used at write time                                                                                                                                                                                                                                                                              |
| Scheduled backup never runs       | missing `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` / `BACKUP_BUCKET_NAME` / `CLOUDFLARE_ACCOUNT_ID`, or the tenant is never probed while healthy | Sandbox SDK presigned-URL path needs all four; `hasBackupCredentials` (`persistence.ts`) gates the scheduler, `requirePresignedURLSupport` throws `InvalidBackupConfigError` at snapshot time. Scheduled snapshots now fire on **any healthy** `/api/health` (passive included), so verify the lerma `HealthMonitor` actually polls the tenant |
| Reprovision returns `409`         | no restorable backup and the checkpoint failed                                                                                                    | Expected fail-closed behaviour. Restore a checkpoint via `POST /api/admin/storage/sync` first; only pass `{"force":true}` when the container is already gone and its state is unrecoverable                                                                                                                                                    |
| Old config versions pile up       | retention is 30 days; normal                                                                                                                      | `cleanupOldConfigVersions` prunes automatically                                                                                                                                                                                                                                                                                                |
| More backups than `maxVersions`   | a run of snapshots failed before their retention prune, so reconciled orphans accumulated                                                         | self-heals via the explicit reconcile: the durable `repair` step (and `POST /api/admin/storage/reconcile`) enforces retention through `reconcileBackupCatalog`, while `GET /storage/backups` itself never mutates                                                                                                                              |

R2 persistence uses the Sandbox SDK backup API; there is no s3fs mount to inspect.
Verify backup state with `GET /api/admin/storage/backups` and never delete tenant
backup objects outside retention or decommission procedures.

### 6.3 AI Gateway issues

The **only** provisioned AI provider path is Cloudflare AI Gateway, configured when all
three are set (`start-openclaw.sh:161`, `config.ts:97`):

```bash
npx wrangler secret put CLOUDFLARE_AI_GATEWAY_API_KEY   # your gateway API key
npx wrangler secret put CF_AI_GATEWAY_ACCOUNT_ID        # Cloudflare account id
npx wrangler secret put CF_AI_GATEWAY_GATEWAY_ID        # gateway id within the account
```

Cloudflare AI Gateway gives caching, rate limiting, analytics, and cost tracking, and the
`cloudflare-unified-billing` plugin extends the catalog to 150+ models (Google Gemini,
Anthropic Claude, OpenAI, xAI Grok, DeepSeek, Alibaba Qwen, MiniMax, Moonshot Kimi, and
more) via the AI Gateway REST API. The plugin — plus the `cloudflare-ai-gateway`
provider and `diagnostics-otel` — is **pre-installed into the image at build time** from
ClawHub (unpinned = latest release; published at
https://clawhub.ai/paso4/plugins/cloudflare-unified-billing) so no network install runs
on the latency-critical boot path. `start-openclaw.sh` only installs from ClawHub as a
fallback when the extension state is missing. Lerma provisions the AI-scoped
credential to tenant workers as both `CLOUDFLARE_AI_GATEWAY_API_KEY` and the REST alias
`CLOUDFLARE_API_TOKEN`.

**Premade agents:**

Production deployments ship **no** premade agents — only the `main` agent written by
`openclaw onboard`. The dev-only **Fouria Builder** and **Fouria Q&A agent** are injected
by moltlazy only on dev instances (the Worker derives `OPENCLAW_DEV_MODE` from the
instance's own `-dev` hostname, or `DEV_MODE` locally; they can self-install plugins and
read per-tenant client documentation materialized from R2). Neither agent is ever part of
a production instance: their purpose is to model the production agent gateway and
subagent squad (agents, skillsets, plugins, tools, automations) that satisfies the
client's business logic. Because OpenClaw has no per-agent run timeout, dev instances
raise the global `agents.defaults.timeoutSeconds` to 1800 s (30 min) so the Builder is not
aborted mid-build; production keeps the 300 s baseline.

| Agent            | Environment | Primary model     | Purpose                                                |
| ---------------- | ----------- | ----------------- | ------------------------------------------------------ |
| `fouria-builder` | dev only    | Claude Sonnet 4.6 | Read client proposal/phases, build a working POC       |
| `fouria-qa`      | dev only    | Claude Sonnet 4.6 | Run `cctr` corpus validation against the Builder's POC |

| Symptom                                     | Cause                                                | Action                                                                                                                                                                    |
| ------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Onboard skips AI Gateway auth               | one of the three vars missing                        | set all three; mark `NEED_ONBOARD=true`                                                                                                                                   |
| `cloudflare-ai-gateway` plugin missing      | build-time pre-install missing/removed               | it is pre-installed in the image; `start-openclaw.sh` reinstalls from ClawHub as a fallback (bounded, non-fatal)                                                          |
| `cloudflare-unified-billing` plugin missing | build-time pre-install missing/removed               | it is pre-installed in the image; `start-openclaw.sh` reinstalls the latest `@paso4/cloudflare-unified-billing` from ClawHub as a fallback (unpinned, bounded, non-fatal) |
| `diagnostics-otel` install fails            | pre-install missing/removed                          | it is pre-installed in the image; the fallback installs the latest ClawHub build (unpinned). There are no version pins to bump                                            |
| Model requests fail with auth errors        | wrong/expired gateway key                            | rotate `CLOUDFLARE_AI_GATEWAY_API_KEY`                                                                                                                                    |
| No embedding / memory search errors         | no embedding key (`CF_AI_GATEWAY_OPENAI_COMPAT_KEY`) | memory search auto-disabled (`start-openclaw.sh:422`) — expected                                                                                                          |

The `cloudflare-unified-billing` plugin (40+ models via the REST API) is pre-installed in
the image and enabled alongside the gateway provider; enablement is written by a single
`config patch` (which also enables `admin-http-rpc` and disables `diagnostics-otel` unless
its endpoint is set), **not** a `plugins enable` CLI call, to avoid a hanging ClawHub
registry refresh.

When migrating existing tenants from the old provisioner, re-provision the tenant
and perform a cold restart after the AI credential is configured. Older R2 snapshots
may contain a stale native-provider auth profile and the old random AI binding; a
warm restore alone does not replace those persisted values.

### 6.4 Startup process issues

The startup sequence is `start-openclaw.sh` (`STEPS` array, `start-openclaw.sh:69`):
`init → onboard → plugins → config-patch → config-validate → tunnel → gateway`. Watch it
live via:

```bash
curl "https://<worker>/api/health"        # status: starting while script runs
curl "https://<worker>/debug/gateway-logs" | tail -n 50
```

| Symptom                                                | Cause                               | Action                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------------------------------ | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Stuck at `config-validate` → startup aborts            | invalid generated config            | `moltlazy validate` locally; fix feature flags                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Stuck at `onboard` > 60s                               | AI Gateway plugin not present       | the `cloudflare-ai-gateway` provider is pre-installed in the image (so its plugin phase is a no-op); if it still times out, check ClawHub/npm reachability — onboard is capped at 60s and continues, gateway starts `--allow-unconfigured`                                                                                                                                                                                                                                |
| Gateway never becomes ready (readiness gate times out) | enabled-but-missing provider plugin | the provider plugins are pre-installed in the image, so this should be rare; `start-openclaw.sh` falls back to installing the latest ClawHub spec when the extension state is missing (look for `Installing … from ClawHub` in `/tmp/startup.log`). The enablement `config patch` must not carry `plugins.bundledDiscovery` (removed in 2026.9.x — including it fails validation and drops the whole patch)                                                               |
| Gateway dies before port opens                         | gateway binary crash                | `GET /debug/gateway-logs` for the crash; check `EADDRINUSE`. On `Reason: database disk image is malformed` the Worker queues the `doctor --fix` repair automatically and relaunches after it (see §5.4f); if the log shows `SQLite integrity_check failed for …/state/openclaw.sqlite`, run `POST /debug/sqlite-integrity` (then poll its job id) to confirm which database is corrupt and consider restoring an older verified backup instead of repairing the only copy |
| `STARTUP_TIMEOUT_MS` (300s) exceeded                   | fallback plugin installs slow       | plugins are pre-installed in the image so the normal boot skips installs; if it stalls, the fallback is doing network installs — check ClawHub reachability                                                                                                                                                                                                                                                                                                               |
| Loading page never advances                            | `startup-progress.json` not written | check `/tmp/startup.log` via `debug/gateway-logs`                                                                                                                                                                                                                                                                                                                                                                                                                         |

The loading page renders from `/tmp/startup-progress.json` + `/tmp/startup.log`
(`routes/public.ts:131`, read bounded by `STARTUP_PROGRESS_TIMEOUT_MS`).

---

## 8. Cloudflare Platform Setup

One-time operator setup tasks for the current FourIA Worker deployment.

### 8.1 Cloudflare Access (protects `/_admin`, `/api`, `/debug`)

1. **Enable Access on the worker** — Workers & Pages dashboard → your Worker → **Settings →
   Domains & Routes → `workers.dev` → Enable Cloudflare Access**. Copy the **Application
   Audience (AUD)** tag.
2. **Configure who can access** — Zero Trust → **Access → Applications** → your worker app →
   add allowed identities (email, Google, GitHub, …).
3. **Set the secrets**:

   ```bash
   npx wrangler secret put CF_ACCESS_TEAM_DOMAIN   # e.g. myteam.cloudflareaccess.com
   npx wrangler secret put CF_ACCESS_PLATFORM_AUD  # platform AUD (always required)
   npx wrangler secret put CF_ACCESS_CLIENT_AUD    # optional per-tenant AUD
   ```

   Alternatively create a **Self-hosted** Access application and protect paths
   `/_admin/*`, `/api/*`, `/debug/*`.

> `/debug/*` must be Access-protected. Never expose it publicly — it can stop, destroy, and
> rescue containers.

### 8.2 Browser Automation (CDP)

A Chrome DevTools Protocol shim backed by the Worker's Cloudflare **Browser
Rendering** binding (`BROWSER`). Set:

```bash
npx wrangler secret put CDP_SECRET   # shared secret for CDP auth
npx wrangler secret put WORKER_URL   # worker public URL
npm run deploy
```

| Endpoint                | Description                               |
| ----------------------- | ----------------------------------------- |
| `WS /cdp?secret=…`      | CDP WebSocket channel (commands + events) |
| `GET /cdp/json/version` | CDP discovery (`webSocketDebuggerUrl`)    |
| `GET /cdp/json/list`    | List available browser targets            |
| `GET /cdp/json`         | Alias for `/cdp/json/list`                |

All endpoints require `?secret=<CDP_SECRET>`. The endpoint is a **remote CDP**
surface: point an OpenClaw profile at it with
`browser.profiles.<name>.cdpUrl = "https://<worker>/cdp?secret=…"` (default
`openclaw` driver) — see https://docs.openclaw.ai/tools/browser/remote. The
E2E coverage is `test/e2e/r_browser_cdp.txt`.

> **Tenant agents do not use this public route.** On a tenant hostname the host
> Cloudflare Access application intercepts `/cdp` with a `302` login redirect, and
> no bypass is added. Moltlazy instead points the browser tool at the
> container-local proxy (`http://127.0.0.1:9223`), which forwards CDP commands to
> the Worker's private `cdp.internal` bridge; the browser session lives in the
> tenant's Sandbox Durable Object and the Browser Rendering credential never
> enters the container. See
> [WORKER-DEVELOPMENT.md § Browser Rendering](/llms/platform/apps/fouria/worker-development/index.md#browser-rendering-private-httpcdpinternal-bridge)
> and `test/e2e/r_browser_remote_profile.txt`.

### 8.3 Obsidian Sync

The `obsidian-sync` skill runs an Obsidian LiveSync-compatible service on port `18790`
inside the container. Tenant Cloudflare Tunnel support was **removed**: the container no
longer starts `cloudflared`, and Lerma provisions no `CLOUDFLARE_TUNNEL_TOKEN` binding.
The service is therefore reachable only from inside the container unless an operator
builds their own ingress path.

### 8.4 Built-in skills

The container ships pre-installed skills in `/root/clawd/skills/`, including
**`cloudflare-browser`** (documents the CDP connection pattern for the Worker's
Browser Rendering shim; needs `CDP_SECRET` + `WORKER_URL`).

---

## 9. Requirements & References

**Requirements**

- [Workers Paid plan](https://www.cloudflare.com/plans/developer-platform/) ($5/mo) — required
  for Cloudflare Sandbox containers.
- An AI provider key via Cloudflare AI Gateway (API key, account id, gateway id) — the only
  supported AI path. Unified Billing is available as an alternative
  ([docs](https://developers.cloudflare.com/ai-gateway/features/unified-billing/)).

Free tiers exist for Cloudflare Access, Browser Rendering, AI Gateway, and R2 Storage.

**See also**

- [OpenClaw](https://github.com/openclaw/openclaw) · [OpenClaw Docs](https://docs.openclaw.ai/)
- [Cloudflare Sandbox](https://developers.cloudflare.com/sandbox/) ·
  [Cloudflare Access](https://developers.cloudflare.com/cloudflare-one/policies/access/)
- [Worker deep-dive](/llms/platform/apps/fouria/worker-development/index.md) · [Root DEVELOPMENT guide](../../docs/DEVELOPMENT.md)
- [Env vars reference](../../docs/ENV-VARIABLES.md)

---

## 7. Quick Reference

### Environment variables (operator-relevant)

| Variable                                                                                  | Purpose                                                                                         | Notes                                                                                                |
| ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `DEBUG_ROUTES`                                                                            | Enables `/debug/*`                                                                              | Set `true` only on operator/ROOT deploys                                                             |
| `CLOUDFLARE_API_TOKEN`                                                                    | AI Gateway REST auth; Containers:Edit for rescue                                                | AI-scoped tenant alias; rescue token on ROOT/ops                                                     |
| `CONTAINER_APPLICATION_ID`                                                                | Target app for image rollout                                                                    | Operator-only                                                                                        |
| `CONTAINER_IMAGE_TAG` / `CONTAINER_IMAGE_NAME`                                            | Expected image `unstable-<sha>` / `fouria`                                                      | Operator-only                                                                                        |
| `CLOUDFLARE_AI_GATEWAY_API_KEY` / `CF_AI_GATEWAY_ACCOUNT_ID` / `CF_AI_GATEWAY_GATEWAY_ID` | AI Gateway auth                                                                                 | Required for AI provider                                                                             |
| `BACKUP_ENCRYPTION_KEY` / `BACKUP_ENCRYPTION_REQUIRED`                                    | R2 encryption at rest                                                                           | Recommended `true` in prod                                                                           |
| `BACKUP_BUCKET_NAME` / `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY`                        | R2 persistence                                                                                  | All four (incl. `CLOUDFLARE_ACCOUNT_ID`) required for scheduled backups; SDK signs R2 presigned URLs |
| `BACKUP_INTERVAL_MINUTES` / `BACKUP_MAX_VERSIONS`                                         | Backup cadence floor (auto-stretched to 2× the last snapshot duration) / retention              | Defaults 5 / 3                                                                                       |
| `BACKUP_CHANGE_DETECTION`                                                                 | Skips scheduled snapshots of unchanged tenant state (**on by default**)                         | Set `false` only to force a full snapshot every tick; the scan fails closed to "changed"             |
| `BACKUP_EXPIRED` on a catalogued archive                                                  | The SDK refuses to restore past `createdAt + ttl`; the archive is billed but unrestorable       | Healed automatically: the durable `repair` step widens the TTL to 10 years and reports `ttlExtended` |
| "not encrypted" badge on a backup                                                         | The archive is not encrypted at rest (a legacy archive, or one whose encryption step never ran) | The next backup repairs it in place; it is never promoted to the recovery point                      |
| `SANDBOX_INSTANCE_TIMEOUT_MS` / `SANDBOX_PORT_TIMEOUT_MS` / `SANDBOX_POLL_INTERVAL_MS`    | Container fail-fast bounds                                                                      | See §6.1                                                                                             |
| `CLOUDFLARE_ACCOUNT_ID`                                                                   | Cloudflare account id (plain var)                                                               | Read by the Sandbox SDK for R2 presigned URL signing; legacy `CF_ACCOUNT_ID` is deprecated           |
| `CF_ACCESS_TEAM_DOMAIN` / `CF_ACCESS_PLATFORM_AUD` / `CF_ACCESS_CLIENT_AUD`               | Cloudflare Access auth                                                                          | Protects `/_admin`, `/api`, `/debug`                                                                 |
| `CDP_SECRET` / `WORKER_URL`                                                               | Browser automation (CDP)                                                                        | See §8.2                                                                                             |
| `STARTUP_LOG_VERBOSITY`                                                                   | `steps`/`detailed`/`verbose` boot log                                                           | Secret-free                                                                                          |

### One-line recovery cheatsheet

```bash
# 1. Is it up? (no warm-up)
curl "https://<worker>/api/health?passive=1" | jq '{status,degraded,containerError}'

# 2. What's the lock state?
curl "https://<worker>/debug/gateway-lock" | jq .state

# 3. Degraded (no such image)? Rescue (dry first):
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":true}' | jq .
curl -X POST "https://<worker>/debug/rescue-container" -d '{"dryRun":false}' | jq .

# 4. Gateway wedged but config OK? Stop, wait 8s, restart (via Admin API).
# 5. Lost state? Inspect backups, then restore by id or let next boot auto-restore.
```

---

_This guide is generated from the source in `apps/fouria/src/diagnostics.ts` and its
dependencies (`gateway/process.ts`, `gateway/health-probes.ts`, `gateway/lock.ts`,
`persistence.ts`). When the behavior of
those modules changes, update this document in the same change._
