# FourIA — Worker Dashboard Architecture

Deep-dive into the `apps/fouria` component, the core Tenant Worker of FourIA.

> See [../docs/DEVELOPMENT.md](../../docs/DEVELOPMENT.md) for project-level overview and [../docs/ENV-VARIABLES.md](../../docs/ENV-VARIABLES.md) for environment variable reference.

---

## Architecture Overview

```
Browser
   │
   ▼
┌─────────────────────────────────────────┐
│       Cloudflare Worker (index.ts)       │
│                                          │
│  Middleware: Logging → Auth → RBAC       │
│                                          │
│  Routes:                                 │
│    /api/*     → Hono API routes          │
│    /_admin/*  → React SPA (Vite build)   │
│    /debug/*   → Debug endpoints (ROOT)   │
│    /*         → Proxy to OpenClaw GW     │
│                                          │
│  Activation: Worker restore → onboard    │
│              → patch → launch gateway    │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│   Cloudflare Sandbox Container           │
│                                          │
│  start-openclaw.sh:                      │
│    1. openclaw onboard (auth setup)      │
│    2. Enable plugins (unified billing)   │
│    3. moltlazy patch (config injection)  │
│    4. Start Cloudflare Tunnel (optional) │
│    5. background gateway + health poll   │
│                                          │
│  ┌────────────────────────────────────┐  │
│  │     OpenClaw Gateway               │  │
│  │  - Control UI on port 18789        │  │
│  │  - WebSocket RPC protocol          │  │
│  │  - Agent runtime                   │  │
│  └────────────────────────────────────┘  │
└─────────────────────────────────────────┘
```

---

## Frontend Architecture (Vite + React SPA)

The dashboard is a React 19 SPA built with Vite 6 and served from `/_admin/`.

### Build & Dev Flow

```
bun run dev           → Vite dev server on :8787 (vite.ui.config.ts)
    │                     Hot-reload for frontend changes
    │                     Proxies /api/*, /debug/* to :8788 (worker)
    │
bun run dev:worker    → Vite dev server on :8788 (vite.config.ts)
    │                     Runs the actual Worker in workerd via the
    │                     Cloudflare Vite plugin; Worker changes hot-reload
    │
bun start             → bun run build:docker && vite dev
                          Full local stack (build + Worker dev server)
```

> **Key insight:** both dev servers are Vite. `bun run dev` only handles the
> admin SPA; the Worker (routes, auth, gateway, DOs, containers) runs under the
> Cloudflare Vite plugin via `bun run dev:worker`, the same plugin used by
> `vite build`. The plugin has no `wrangler dev` flags: local E2E vars are
> injected through its `config` hook and `.dev.vars` is loaded from this
> directory.

### Vite Configuration

```typescript
// vite.config.ts — Worker dev server (bun run dev:worker)
base: '/_admin/'              // All assets served under /_admin/
server: {
  port: 8788,
  allowedHosts,               // DEV_ALLOWED_HOSTS (tunnel hostnames)
}

// vite.ui.config.ts — admin SPA dev server (bun run dev, :8787)
server: {
  port: 8787,
  proxy: {
    '/api':   'http://localhost:8788',
    '/debug': 'http://localhost:8788',
    '/cdp':   'http://localhost:8788',
  },
}
```

> Exposing the Worker through a tunnel (e.g. Telegram webhooks): Vite blocks
> unknown `Host` headers, so pass the public hostname explicitly —
> `DEV_ALLOWED_HOSTS=foo.trycloudflare.com bun run dev:worker` (comma-separated;
> a leading dot allows a whole domain). Point the tunnel at `:8788`, not the SPA
> at `:8787` (it does not proxy `/telegram-webhook`).

### Client Module Structure

```
src/client/
├── App.tsx              # Root component with React Router
├── main.tsx             # Entry point — mounts React app
├── api.ts               # Typed API client → /api/admin/*
├── gateway-gate.ts      # Health circuit breaker (park/resolve gateway queries)
├── GatewayHealthContext.tsx  # App-wide passive health subscription + gate wiring
├── pages/               # Route-level pages
│   ├── HomePage.tsx       # Dashboard overview
│   ├── AgentsPage.tsx     # Agent management
│   ├── ChannelsPage.tsx   # Connections/integrations
│   ├── KnowledgePage.tsx  # Knowledge base/wiki
│   ├── AccountPage.tsx    # User/team management
│   ├── IntegrationsPage.tsx   # Integration config
│   ├── SecretsPage.tsx    # Vault management
│   ├── BackupsPage.tsx    # Backup management
│   ├── DiagnosticDashboard.tsx  # Costs + diagnostics
│   ├── AuditLogsPage.tsx  # Gateway ledgers: audit activity + background tasks
│   └── AutomationEditorPage.tsx # Automation workflows
├── components/          # Shared UI components
└── hooks/               # Custom React hooks
```

### Frontend ↔ Backend Communication

The frontend (`src/client/api.ts`) calls `/api/admin/*` endpoints. These are served by the Worker's Hono routes (`src/routes/api.ts`), which:

1. Verify RBAC (`requireWriteRole()` for mutations)
2. Execute OpenClaw CLI commands inside the Sandbox container
3. Parse CLI output and return JSON

#### Client circuit breakers (health-gated gateway queries)

Two client-side circuit breakers keep the dashboard responsive when the OpenClaw gateway is not ready (issue #371):

- **#1 — gateway queries are asynchronous.** Pages paint immediately and never block first paint on a gateway response; mutations are fire-and-notify through `NotificationProvider`. `api.ts` routes every gateway-touching call through `gatewayApiRequest`.
- **#2 — gateway queries require the passive health stream to be UP.** The single app-wide `GatewayHealthProvider` (`src/client/GatewayHealthContext.tsx`) subscribes once to `/api/health-ws?passive=1` and drives a module-singleton `GatewayGate` (`src/client/gateway-gate.ts`). **UP** is defined as `snapshot.readiness.ok === true`; `starting`/`stopped`/`sleeping` are **not** UP (`sleeping` is healthy-idle but the gateway is not running). Before dispatching, `apiRequest` awaits `whenGatewayUp(signal)`; while down the query is parked and the `GatewayGateNotice` indicator shows, then the queue flushes automatically on the next UP snapshot. A transient (`503`/transport) failure is retried once after the next UP.

The gateway-touching boundary is the `GATEWAY_TOUCHING_PREFIXES` manifest in `api.ts` (`/agents`, `/models`, `/channels`, `/config`, `/devices`, `/integrations`, `/secrets`, `/knowledge`, `/audit-logs`, `/dreaming`, `/gateway/restart`). Diagnostics (`/debug/*`, `/logs`, `/gateway-api`, `/gateway/status/polling`, `/gateway-token`, `/version`) and worker-local surfaces (`/storage`, `/costs`, `/billing`, `/users`, `/vault`, `/releases`, `/fouria-docs`, `/whoami`) are deliberately **not** gated, so an operator can still inspect a down gateway and non-gateway features keep working.

**Diagnostics (`DiagnosticDashboard`).** The ROOT dashboard is explicitly _skip-not-park_: `useGatewayRunning()` (`GatewayHealthContext.tsx`) reports `snapshot.readiness.ok` **ignoring the ROOT bypass**, and the page only dispatches its container reads (`/config/raw`, `/config`) while it is `true`; they re-run automatically when the gateway transitions back to running. The debug-routes flag is Worker-local and still loads. Every call is dispatched after paint (`scheduleAfterPaint` → `requestIdleCallback`, `setTimeout(0)` fallback) and the widgets render their waiting state first, so nothing is blocked on a container response: the Configuration Editor renders its tabs/editor with a non-blocking "Loading configuration…" / "Waiting for the gateway to start…" notice (`gatewayReady` prop on `ConfigSection`) and the device list shows a waiting state until Refresh. Gateway-only actions (device list/approve, config save, channel probe) are disabled/guarded while down; **`doctor --fix` repair is NOT one of them**: it is the recovery action for a broken/down gateway (the backend stops the gateway itself), so a dedicated **Repair Environment** control sits outside the Configuration Editor (always rendered for ADMIN/ROOT, disabled only while a repair is in flight) and the in-editor repair button stays enabled while the gateway is down. The Debug Tools that call the container (logs, processes, version, sqlite integrity) and the recovery actions (restart/rescue/destroy) stay available on purpose — they are how an operator inspects and revives a down gateway — while the ones that require a running gateway (`gateway-api`, `stop-gateway`) are skipped with an inline explanation.

**ROOT bypass.** ROOT operators are never gated: `App.tsx` passes `bypass={whoami?.role === 'ROOT'}` to the provider, which flips the gate's bypass (`GatewayGate.setBypass`). This preserves the "ROOT can always reach a tenant" rule and keeps the operator's **non-passive** device-pair request (`GET /api/admin/devices/ws` → `listDevices` → `ensureGateway({ waitForReady: false })`) able to wake a sleeping container. The notification bell is **passive for every non-ROOT role** (`subscribeDeviceNotifications(..., { passive: true })` → `?passive=1`), so a regular dashboard load never wakes a sleeping tenant; only ROOT's stream is non-passive. Server-side, a passive `/api/admin/devices/ws` first does the non-waking `readRuntimeActive()` read and, when the runtime is not live, emits an empty snapshot and skips the gateway event stream. Once the ROOT wake brings the gateway UP, the shared health stream flips to `up` and every parked query (for all roles) flushes. The `GatewayGateNotice` is suppressed while bypassed, since ROOT actions are never queued.

The gate is fail-open when no provider is mounted (non-SPA callers, unit tests): the module default is `up`. The provider installs the shared gate during its first render, before any child page effect runs. It is a module singleton rather than a `useState` initializer because React StrictMode double-invokes initializers in dev — two gates would leave `api.ts` parked on a different instance than the provider drives.

---

## Backend Architecture (Hono Worker)

### Request Pipeline

```
Incoming Request
    │
    ├─► Logging middleware (all routes)
    ├─► Sandbox init (getSandbox() into context)
    │
    ├─► Public routes (no auth):
    │     /sandbox-health
    │
    ├─► CF Access Auth middleware:
    │     Extract JWT → validate → set accessUser
    │
    ├─► RBAC resolution:
    │     resolveCustomerRole → set userRole, customerSlug
    │
    ├─► Route dispatch:
    │     /api/*     → API routes (auth + RBAC)
    │     /_admin/*  → Admin UI (SPA assets)
    │     /debug/*   → Debug routes (ROOT only)
    │     /*         → Catch-all proxy to gateway
    │
    └─► Gateway proxy:
          HTTP proxy → containerFetch(port 18789)
          WS proxy   → WebSocket relay with token injection
```

### API Route Structure

```typescript
// src/routes/api.ts — route aggregation
api.route('/admin', adminApi)
  ├── /whoami, /version, /logs, /gateway-api, /debug-status, /gateway-token  (misc)
  ├── /devices/*      → devicesRoutes
  ├── /storage/*      → storageRoutes
  ├── /releases/*     → releasesRoutes
  ├── /gateway/*      → gatewayRoutes
  ├── /config/*       → configRoutes
  ├── /cron/*         → cronRoutes
  ├── /dreaming/*     → dreamingRoutes
  ├── /users/*        → usersRoutes
  ├── /integrations/* → integrationsRoutes
  ├── /knowledge/*    → knowledgeRoutes
  ├── /secrets/*      → secretsRoutes
  ├── /agents/*       → agentsRoutes
  ├── /models/*       → modelsRoutes
  ├── /channels/*     → channelsRoutes
  ├── /costs/*        → costsRoutes
  ├── /vault/*        → vaultRoutes
  ├── /billing/*      → billingRoutes (plan, checkout, portal, invoices, freeze-status, freeze, unfreeze)
  └── /audit-logs/*   → auditLogsRoutes (audit activity ledger + task ledger)

// Write gate: POST/PUT/DELETE/PATCH require ADMIN or ROOT
// Read gate: GET/HEAD available to all authenticated roles
```

> **Build identity.** `GET /api/admin/version` and `GET /debug/version` report the
> FourIA Worker's own `fouria_version` (component CalVer from
> `apps/fouria/package.json`) and `fouria_commit` (short git SHA) alongside the
> container's OpenClaw/Node versions. Both are injected into the bundle by
> `vite.config.ts` (`__FOURIA_VERSION__` / `__FOURIA_COMMIT__`, consumed in
> `src/build-info.ts`; `GITHUB_SHA` wins in CI, `git rev-parse --short HEAD`
> locally). The DiagnosticDashboard **Version Info** tool surfaces them for
> operators.

### Cloudflare error and trace correlation

The Worker emits structured `worker.request.started` and
`worker.request.completed` records with the method, path, status, duration, and
Cloudflare Ray ID (`cfRay`). Unhandled exceptions are logged as
`worker.request.failed` with the exception name/message/stack and the same
request fields. Cloudflare also attaches its invocation `$metadata.requestId`
and `$metadata.traceId` to each Worker log, so the invocation's request, logs,
and trace can be opened together in Workers Observability. Error records pass an
`Error` object to `console.error` as well, so Cloudflare populates its exception
metadata (including `$metadata.error`) alongside the indexed event fields.

Sandbox command failures are emitted as `sandbox.process.failed`. These records
include a stable operation label and phase, the Sandbox `processId` when a
process was launched, and available exit-code/timeout/duration fields. The
`processId` is the join key to the `processId` field on the Containers dataset's
`process.complete` event; inspect `exitCode` and `timedOut` there rather than
trusting the generic event message, which can say `success` when the process
itself exited non-zero. Raw shell commands, stdout, and stderr are deliberately
omitted because they can contain tenant config or credentials.

Persistence failures use stable events such as `persistence.backup.failed`,
`persistence.config-version.failed`, `persistence.restore.failed`, and
`storage.restore.failed`. Config-version and restore errors include `backupId`
when an archive exists; use it to join to the `backup.create` / `backup.restore`
container events. A config-version capture failure is best-effort: it identifies
the archive whose supplemental config snapshot was not captured, but does not
mean the archive itself failed to be created.

Backup and restore work also emits paired `persistence.phase` records. Each
record has an `operationId`, `operation` (`backup` or `restore`), `phase`, and
`status` (`started`, `completed`, or `failed`); terminal records include
`durationMs`, and phases after an archive handle exists include `backupId`.
`source` names the route or job, `cfRayId` correlates request-triggered work with
`worker.request.*`, and `jobId` correlates work running later in the
ActivityScheduler. Use `operationId` to group the phases; if a started phase has
no terminal record, the Worker may have been interrupted during that phase.
Useful phases include `sandbox.createBackup`, `archive.encrypt`,
`catalog.commit`, `sandbox.restoreBackup`, and `legacy-state.materialize`.

For a failure investigation, filter the `cloudflare-workers` dataset to
`event = "sandbox.process.failed"`, record `processId` and the Worker
`$metadata.requestId`/`$metadata.traceId`, then search the `containers` dataset
for the same `processId` (and the tenant's `$container.applicationId`). For a
request-level failure, filter to `event = "worker.request.failed"` or use
`cfRay` to pivot to the matching edge request. For backup/restore CPU-limit
investigations, filter to `event = "persistence.phase"`, find the last
`started` event for the relevant `cfRayId` or `jobId`, then group by its
`operationId` to see which phases completed before it.

### Worker observability export destinations

Account-level Cloudflare **Observability → Destinations** export this Worker's
logs and traces to Lerma over OTLP/HTTP JSON (issue #337). The destination names
are referenced in the `observability` block:

```jsonc
// staging
"observability": {
  "logs":   { "destinations": ["lerma-worker-logs"],   "head_sampling_rate": 0.1, "persist": false },
  "traces": { "destinations": ["lerma-worker-traces"], "head_sampling_rate": 0.1, "persist": false }
}
// production
"observability": {
  "logs":   { "destinations": ["lerma-worker-logs-prod"],   "head_sampling_rate": 0.1, "persist": false },
  "traces": { "destinations": ["lerma-worker-traces-prod"], "head_sampling_rate": 0.1, "persist": false }
}
```

The `-prod` destinations point at `https://lerma.paso4.io/api/ingest/otel/v1/*`;
the un-suffixed pair points at `https://lerma-dev.paso4.io/...`.

- `apps/fouria/wrangler.jsonc` carries the matching names in each env block
  (`staging` and `production`, direct deploys).
- WFP-deployed tenant workers get the names from Lerma's
  `Lerma.Cloudflare.Observability.worker_metadata/0`
  (`WORKER_OBSERVABILITY_LOGS_DESTINATIONS` /
  `WORKER_OBSERVABILITY_TRACES_DESTINATIONS`). The CD sets the staging names for
  staging and the `-prod` names for production, so a redeploy inherits them.
- Lerma is the destination of record: `persist: false` stops the duplicate
  Cloudflare-side copy, and the head sampling rate (10% at rollout) bounds export
  volume. Lerma drops message text, stacks, URLs/queries, user agents,
  request/response bodies and DB query text at ingestion; only allowlisted
  metadata is stored.
- Cloudflare gzips each OTLP/HTTP JSON batch (`Content-Encoding: gzip` with
  `Content-Type: application/json`). `LermaWeb.Plugs.OtlpBodyReader` runs before
  `Plug.Parsers` on the ingest routes and decompresses it; without that step the
  normal JSON parser returns `400` and the destination retries forever.
- The export only feeds diagnostics + the Lerma instance **Worker telemetry**
  widget. It never replaces `HealthMonitor` (still the source of
  `instances.status`) or billing, and it is not a probing path — no container is
  woken to produce it.

Ops: create the two destinations under **Observability → Destinations** with the
custom header `Authorization: Bearer <OTEL_INGEST_TOKEN>`, set the token and the
destination names on Lerma (staging first), then redeploy the tenants. Delivery
status shows **Last: n minutes ago**; the uptime numbers come from
`GET /api/analytics/v1/uptime/<script_name>?window=24h`.

### Gateway start lifecycle + stuck-start watchdog

A cold start is one shared promise (`ensureGateway`) spanning the restore
barrier → the supervised `start-openclaw.sh start` launch → the port wait. A
wrapper that never `exec`s the gateway, or a container control-channel drop, can
leave that promise "in progress" for many minutes while every request logs
`Gateway start already in progress … waiting...` and waits on it.

The start is now tracked per sandbox key with a phase (`probe` → `restore` →
`launch` → `port-wait`) and a per-phase timer. `post-start` runs **detached**
after readiness (see below), so it is deliberately not part of the awaited start
promise:

- `gateway.start.begin` / `gateway.start.phase` structured events record the
  attempt, phase, elapsed time, and (after launch) the process id; the
  "already in progress" log carries `phase`, `age`, and `attempt`.
- `isGatewayStartStuck` flags a phase that overruns its budget +
  `GATEWAY_START_STUCK_GRACE_MS` (the `restore` phase gets
  `GATEWAY_RESTORE_PHASE_BUDGET_MS` because the legacy stage may take 10 min; all
  other phases use `STARTUP_TIMEOUT_MS`).
- When the watchdog trips, `recoverStuckGatewayStart` emits a
  `gateway.start.stuck` structured event, kills the stuck supervised process
  (`killGateway`), clears the startup/readiness caches, and returns so the next
  request starts clean. After `GATEWAY_START_STUCK_MAX_ATTEMPTS` consecutive
  resets it reprovisions the container (`recoverStaleSandbox`).
- Separately, `findExistingGatewayProcess` uses `classifyGatewayStart` to detect a
  `start-openclaw.sh start` wrapper older than the cold-start budget with the
  port closed and kills it immediately (source `startup-wrapper-age`) instead of
  waiting out another full readiness budget.
- A **non-owner** isolate that merely _observes_ a running wrapper waits to the
  same deadline the owner's watchdog uses — `STARTUP_WRAPPER_STUCK_BUDGET_MS`
  (`STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS`) measured from the
  wrapper's start — and re-probes `/health` before any kill. A fixed
  `STARTUP_TIMEOUT_MS` measured from observation expired while the wrapper had
  already `exec`ed the gateway and was still binding, killing it mid-boot,
  releasing the startup lock, and making the waiting second wrapper re-run the
  entire setup (observed on `staging-client-dev`: a ~14-minute cold boot). The
  kill now only fires once the wrapper has genuinely overrun the owner's own
  budget with the port still closed.

**The `port-wait` also completes on a positive `/health` probe.** The SDK's
`waitForPort` drives a container-side port-watch + log-readiness subscription
that can hang indefinitely on some boots **even after the gateway is serving**.
Waiting on it alone let the shared start promise overrun its budget, so the
watchdog force-killed a **healthy** gateway and reprovisioned the container
(observed on `tenant-client-dev`: `gateway.start.stuck phase=port-wait` after
368 s while HTTP proxying returned 200). The wait races `waitForPortResilient`
against the same positive `/health` probe the public proxy uses. If that waiter
reaches its deadline and parks, the watchdog makes one fresh bounded health
check before resetting: a late-positive response emits
`gateway.start.late-ready` and is adopted without killing the gateway. A failed
health RPC is treated as **unknown**, not closed, and defers the reset until a
later request can probe again; only a confirmed closed port follows the normal
stuck-start reset path.

**Process count is not a health signal.** `listProcesses()` returns every
tracked process for the container's lifetime — including `exited`/`error`
entries the Worker never garbage-collects — so the raw length only grows. The
log line now reads `listProcesses returned <total> processes (<running> running)`
and every health decision (`classifyGatewayStart`, the gateway matcher) considers
**only `running` entries**. The SDK exposes no completed-process cleanup, so the
list is append-only; never gate health on the total.

#### Gateway protocol ledgers (`/api/admin/audit-logs`)

The `AuditLogsPage` reads the two append-only ledgers from the Gateway protocol
([docs.openclaw.ai/gateway/protocol/ledgers](https://docs.openclaw.ai/gateway/protocol/ledgers)):

- `GET /api/admin/audit-logs` — audit **activity** ledger (`audit.activity.list`,
  `operator.read`), newest first. Supports `agentId`, `sessionKey`, `runId`,
  `kind`, `status`, `direction`, `channel`, `after`, `before`, `limit`, and
  `cursor`; returns `{ events, nextCursor? }`. The bundled `admin-http-rpc`
  plugin allowlists `tasks.*` but **not** `audit.*`, so this read goes over the
  Gateway **WebSocket** RPC using OpenClaw's reference client
  (`@openclaw/gateway-client/browser` + `@openclaw/gateway-protocol`) bridged
  through `sandbox.wsConnect`, authenticating with the shared gateway token as
  an `operator` (see `src/gateway/gateway-rpc.ts`). It falls back to
  `openclaw audit --json` when the WS path is unavailable (older Gateway); the
  CLI fallback gets the resolved `OPENCLAW_GATEWAY_TOKEN` injected because a
  fresh `sandbox.exec` shell does not inherit the gateway process env.
- `GET /api/admin/audit-logs/tasks` — **task** ledger (`tasks.list`,
  `operator.read`), newest first. Supports `status` (repeatable), `agentId`,
  `sessionKey`, `limit`, `cursor`, and `sortBy`; returns
  `{ tasks, nextCursor? }`. RPC-first over the admin-http-rpc plugin, with an
  `openclaw tasks list --json` CLI fallback (same token injection).

Both surfaces are ADMIN-only (absent from `DIAGNOSTICS_PATHS`) and return an
empty page with an `error` string when the ledger is unavailable, so the
dashboard degrades gracefully instead of failing the request.

> **Why a WebSocket client.** `admin-http-rpc` only exposes an allowlisted
> method subset. Methods outside it (notably `audit.*`, `plugins.*`,
> `secrets.*`, `agents.files.*`) require the wire-v4 Gateway WebSocket
> handshake (`connect.challenge` → `connect` → `hello-ok`).
> `src/gateway/gateway-rpc.ts` adapts `sandbox.wsConnect` to the reference
> client's `GatewayProtocolSocket`; the client version sent in the connect frame
> is injected at build time from `catalog.openclaw` (`__OPENCLAW_VERSION__`).

#### RPC-first agent board + integrations

Eligible dashboard routes try Admin HTTP RPC first, the official Gateway
WebSocket RPC second, and keep the CLI as the final fallback for older
Gateways (helpers return `null`/`false` on transport or method failure):

- **Agent board (`/api/admin/agents`)** — list reads `agents.list` and channel
  bindings in `config.get` over Admin HTTP RPC. If either result is unavailable,
  the Gateway WebSocket client fetches the missing read(s) over one connection;
  the CLI remains the final fallback. Create → `agents.create`; delete →
  `agents.delete`; identity (name/emoji/avatar) → `agents.update`; model (+
  fallbacks) → `config.patch` on `agents.entries.<id>.model`. Explicit
  bind/unbind operations have no matching RPC method and stay on the CLI;
  persona read/write are filesystem operations.
- **Devices (`/api/admin/devices`)** — list, approve, reject, and approve-all
  retain Admin HTTP RPC as the first path, use Gateway WebSocket methods
  (`device.pair.list`, `.approve`, `.reject`) if HTTP RPC is unavailable, and
  retain the CLI as the compatibility fallback.
- **Models (`/api/admin/models`)** and **channel probe
  (`/api/admin/channels/probe`)** — `models.list` and `channels.status` use
  Admin HTTP RPC first, then Gateway WebSocket RPC, then the existing CLI path.
- **Integrations (`/api/admin/integrations`)** — the whole endpoint is
  **WebSocket-free**. `GET /` reads the allowlisted `channels.status` +
  `agents.list` over the **admin HTTP RPC**, and resolves plugin enabled-state
  from `config.get` (`plugins.entries.<id>.enabled`) merged with the static
  `PLUGIN_FIELDS` catalog — no `plugins.list` socket. When no channel accounts
  are configured the supported-channel catalog is synthesized locally from
  `CHANNEL_FIELDS` instead of shelling out to `openclaw channels list --all`.
  Plugin/channel toggles and non-secret configure fields → `config.patch`;
  secrets reload → the `openclaw secrets reload` CLI. `secrets.apply` and agent
  bindings have no 1:1 RPC method and stay on the CLI. CLI fallbacks remain for
  a degraded HTTP RPC transport. The admin RPC fetcher also cancels a late
  `containerFetch` body when its timeout wins, preventing the Sandbox DO's
  "ReadableStream received over RPC disconnected prematurely" warning and the
  churn that dropped later sockets. The same cleanup is shared by the public
  proxy and the `gateway-api`/diagnostics probes via `src/utils/container-rpc.ts`
  (`containerFetchWithTimeout` cancels a late body, `wsConnectWithTimeout` closes
  a late socket). Reading a **running** gateway's logs must not open its live SDK
  stream (it never settles and is later disconnected): `GatewayProcess.getLogs()`
  tails `/tmp/gateway-stdout.log`, and `/debug/processes?logs=true` /
  `/api/admin/logs?id=` refuse a live stream for a still-running process. The
  catch-all proxy logs only an allowlisted header subset (`safeHeaderLog`) —
  never the forwarded map, which carries CF Access credentials and cookies.
- **Config / secrets / agent resolution** — config set/unset and dreaming
  settings use `config.patch` over Admin HTTP RPC, then Gateway WebSocket RPC,
  then the CLI fallback. `POST /api/admin/secrets/reload` uses
  `secrets.reload` (WebSocket); `resolveAgentFlag` (shared by the
  `models`/`knowledge` CLI calls) resolves the roster with `agents.list` over
  Admin HTTP RPC, then WebSocket RPC, then CLI.

> **`secrets apply` has no `--yes`.** The current CLI accepts only `--from
> <plan.json>`, `--dry-run`, `--allow-exec`, and `--json`; the plan itself is the
> confirmation. Both `integrations` and `dreaming` apply without `--yes`.

#### Agent Studio (`/_admin/agents`)

The Agent Garage is an **all-in-one, RPC-only workbench** (roster + Overview,
Capabilities, MCP, Memory, Automations, Connections, Health). The old
character-builder page was replaced; Channels/Integrations/Automation remain the
platform-wide configuration surfaces.

The studio never shells out to `openclaw` and never runs `cat`/`find`/`stat`:

| Surface                                                              | Gateway method                                                                    | Transport                    |
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------- | ---------------------------- |
| Agent entries, bindings, channels, MCP, plugins, skills declarations | `config.get`                                                                      | Admin HTTP RPC (WS fallback) |
| Roster / identity / workspace                                        | `agents.list`                                                                     | Admin HTTP RPC (WS fallback) |
| Bootstrap (`AGENTS.md`, `SOUL.md`, …)                                | `agents.files.list` / `.get` / `.set`                                             | **WS**                       |
| Durable notes + dreaming files                                       | `agents.workspace.list`                                                           | **WS**                       |
| Tool catalog                                                         | `tools.catalog`                                                                   | **WS**                       |
| Effective surface for a session                                      | `tools.effective` (`sessionKey` required)                                         | **WS**                       |
| Skills, workshop, trust, required bins                               | `skills.status`, `skills.workshop.list`, `skills.securityVerdicts`, `skills.bins` | **WS**                       |
| Hooks                                                                | `hooks.status`                                                                    | **WS**                       |
| Plugins + runtime state                                              | `plugins.list`                                                                    | **WS**                       |
| Sessions + store/WAL sizes                                           | `sessions.list`, `sessions.storage.status`                                        | **WS**                       |
| Jobs + run history                                                   | `cron.list`, `cron.status`, `cron.runs`                                           | Admin HTTP RPC (WS fallback) |
| Memory provider                                                      | `memory.status`                                                                   | **WS**                       |
| Node/process/disk                                                    | `system.info`                                                                     | **WS**                       |
| Update / backups                                                     | `update.status`, `backup.status`                                                  | HTTP / **WS**                |
| Activity (future)                                                    | `audit.activity.list`                                                             | **WS**                       |

Endpoints (all ADMIN-only, mounted at `/api/admin/agent-studio`):
`GET /` (instance + roster), `GET /instance`, `GET /:agentId`
(`?sessionKey=` selects the effective session), `GET|PUT /:agentId/files/:name`,
`GET /:agentId/automations`, `POST /:agentId/automations/:jobId/:action`,
`PUT|DELETE /:agentId/bindings`, `POST /plugins/:pluginId/:action`.

The dedicated automations read uses the same job normalization as the agent
detail: `schedule` is always display text, never the Gateway's structured
`{ kind, everyMs, anchorMs }` object. This prevents React error #31 when opening
the Automations pane. The shared gateway status indicator queries the ROOT-only
`/api/admin/debug-status` endpoint only for ROOT users; ADMIN users retain the
health indicator without an unauthorized debug-status request.

Health lists plugins in pages of 10 with Previous/Next controls and a total
count. Pagination is client-side (no additional RPC reads), resets when switching
agents, and hides its controls when the list fits on one page.

Capabilities uses independent 10-item pages for Allowed, Denied, Subagents,
Skills, Available tools, Effective surface, and discovery Notices. Tool pages
count individual tools across groups while preserving their group headings.
Filtering applies before pagination and resets the skills/catalog/effective
pages; changing agents resets all capability pages. No extra RPC reads are made.

**Reads are non-waking.** The route checks `isRuntimeActive()` first and returns
`{ available: false, reason: "sleeping" }` **before** opening a socket, so a
sleeping tenant is never started just to paint the dashboard. Every endpoint
opens **one** `withGatewayRpcClient` connection for its WS methods and guards each
call independently, so one unsupported method only removes its pane (e.g.
`skills: { known: false }`) instead of failing the page. Core reads use the admin
HTTP RPC when available and fall back to `config.get`/`agents.list` over WS.

**Writes:** bootstrap files use `agents.files.set` with the hash from
`agents.files.get` (`409 agent_file_conflict` on a stale hash); bindings are a
`config.patch` on the top-level `bindings` array (no `openclaw agents bind`);
automation run/enable/disable use `cron.run`/`cron.update`; plugin
enable/disable/refresh use `plugins.setEnabled`/`plugins.refresh` with
`operator.admin`.

**RPC gaps (deliberately rendered as unknown, never faked):** there is no MCP
probe/reload method, so MCP health is the last-known `tools.effective` inventory
plus its discovery notices; there is no binary-presence method, so the Health
pane lists `skills.bins` required binaries and per-skill missing requirements
rather than claiming presence.

Tests: `src/agents/studio.test.ts` (pure normalizers/builders),
`src/routes/api/agent-studio.test.ts` (WS/HTTP method map, per-call degradation,
sleeping short-circuit, secret redaction, file conflict, bindings via
`config.patch`), `src/client/pages/__tests__/role-isolation.test.tsx`
(BASE_USER/ADMIN), and `test/ui/agent-studio.spec.ts` (roster, tabs, file editor,
sleeping state).

#### Realtime device-pairing notifications (`/api/admin/devices/ws`)

The dashboard `NotificationBell` subscribes to `GET /api/admin/devices/ws`:

- On connect it pushes a `{ type: "device.pair.snapshot", pending, paired }`
  frame (best-effort, non-blocking), then relays Gateway
  `device.pair.requested` / `device.pair.resolved` / `device.pair.changed` events
  as `{ type: "device.pair.event", event, payload }`.
- The Gateway subscription uses `openGatewayEventStream` (the official
  `@openclaw/gateway-client` browser client over `sandbox.wsConnect`), which
  reconnects with backoff until the browser socket closes.
- Worker-hosted notification sockets rotate after 10 minutes with close code
  `1012`; this is intentionally below the observed Cloudflare load-shed lifetime
  for long-lived Worker invocations. The dashboard reconnects and receives a
  fresh snapshot, releasing the old Worker invocation and Gateway subscription.
- Admins approve/reject directly from the bell:
  `POST /api/admin/devices/:requestId/approve` / `.../reject` (RPC-first
  `device.pair.approve` / `device.pair.reject`, CLI fallback). The client
  helper is `subscribeDeviceNotifications` in `src/client/api.ts`.

**Dashboard notification centre.** `NotificationBell` also renders application
notifications (backup created/failed, backup restored/failed) from the
`NotificationProvider` context (`src/client/NotificationContext.tsx`) that wraps
the admin layout. Long-running admin actions push their result there instead of
blocking on a modal, so the outcome still reaches the admin if they navigate away
before the request settles. The bell badge counts unread app notifications plus
pending device-pairing requests; opening the panel marks app notifications read
and each can be dismissed.

**Persistence-lock indicator.** `PersistenceLockIndicator` (rendered beside the
bell) shows a **closed red lock** while a dashboard-initiated backup or restore
is in flight, and an **open green lock** otherwise. The state comes from
`PersistenceLockContext` (`src/client/PersistenceLockContext.tsx`): the
`BackupHistoryPanel` raises it (`begin('backup' | 'restore')`) when it fires an
async snapshot/restore and drops it (`end()`) when its completion check settles
(success, failure, or timeout), so the indicator is **synchronous with the
operation lifecycle — there is no polling and no lease read**. A counter keeps
the lock held if two operations overlap. The widget is **ADMIN/ROOT only** (a
BASE_USER cannot trigger a backup).

### OTel Ingest (sandbox bridge `http://otel.internal`, shared-secret auth)

There is **no public `/api/otel` route**. The container's OpenTelemetry exporter
targets `http://otel.internal/v1/{traces,metrics}`, a virtual host resolved by the
Sandbox SDK inside this Worker (`TenantSandbox.outboundByHost`, alongside
`docs.r2.internal` / `releases.r2.internal`). The traffic never leaves the
sandbox ↔ Worker control plane, so Cloudflare Access never interposes a login
redirect and no path has to be published.

The per-tenant `OTEL_SHARED_SECRET` is still verified (constant time) by the
handler: the bridge is only resolvable for this tenant's sandbox, but a leaked
handler must not become an open sink. `buildEnvVars` always sets the two
endpoints and the bearer header; `WORKER_URL` is irrelevant to the exporter now.

- `POST /api/otel/v1/metrics` — OTLP/HTTP JSON metrics (cost/tokens). AI cost
  rows keep `openclaw.provider` + `openclaw.model` so lerma can attribute the
  spend (unified-billing gateway-routed vs. direct provider / OTel-only).
- `POST /api/otel/v1/traces` — OTLP/HTTP JSON traces (`{resourceSpans:[...]}`).
  Only `openclaw.message.processed`, `openclaw.harness.run`, `openclaw.tool.execution`
  spans are approved; others are counted as `ignored`. Each approved span is
  classified (`src/billing-classification.ts`) into a **semantic** billing
  action (`telegram_query`, `signal_analysis`, `email_draft`, …) — an explicit
  `openclaw.billing.action_type` marker wins, span attributes are the fallback.
  Approved-but-unmapped spans are accepted and counted as `nonBillable`.
  Stable `action_id = lowercase(traceId) + lowercase(spanId)` (no separator);
  `action_key` dedupes a business unit and `trace_id` drives precedence
  (`email_draft`/`review_analysis` suppress `telegram_query`).
  Tenant is required per span (`customer.slug` et al.; one batch = one tenant).
  `application/x-protobuf` is rejected with `415` (use `http/json`).
  See `docs/contracts/otel-trace-action-id.md`. Normalized traces are held
  in-memory for downstream reconcile; Analytics Engine stays source of truth.
- **One identity authority for both paths.** `/v1/metrics` and `/v1/traces`
  resolve the caller through the same `resolveCostIdentity(env)`
  (`src/sandbox-tenant.ts`): the explicit trusted `CUSTOMER_SLUG` (or the
  `WORKER_NAME` it mirrors) first, then the routing resolver so a legacy worker
  that only carries a routable `WORKER_URL` is still attributed to its own
  tenant instead of the `default` sentinel. A **billable** instance whose
  identity cannot be resolved fails closed with `503 TENANT_UNRESOLVED` on both
  paths, and a metric that names a foreign customer through any of
  `customer.slug` / `customer_id` / `customer` is rejected with `403`. The cost
  summary read side uses the same resolver, so the dashboard reads exactly the
  identity the write path emitted.
  **Emission guard:** `isBillableEventsEnabled(env)` (`src/dev-mode.ts`) decides
  whether an instance may mint billable actions. **Billable production always
  emits and cannot be opted out**, and **the staging platform always emits** in
  every environment so the billing pipeline runs with production fidelity before
  release (its rows land in the staging Analytics Engine dataset). A **dev
  environment** is non-emitting by default — it accepts spans (200) and the cron
  drain drops buffered traces — and emits only when `BILLABLE_EVENTS=true` opts
  it in (e.g. to exercise billing end-to-end from a dev instance). The override
  is emission-only — `getCostContext` still labels rows with the real
  `platform`/`environment` (`staging`/`dev` stay non-billable _cost_ axes), and
  `shouldAutoSleep` is unchanged, so an opted-in dev instance keeps its idle
  timeout and staging keeps its 15m one.

### Browser Rendering (private `http://cdp.internal` bridge)

The agent browser is served by Cloudflare Browser Rendering through the Worker's
`BROWSER` binding. It is deliberately **not** a public `/cdp` dependency for the
container: on a tenant hostname the host Cloudflare Access application answers
`/cdp` with a `302` login redirect, and adding a path-scoped bypass is not
acceptable. The public `/cdp` shim remains for direct/host-side CDP clients and
the cloud E2E worker (workers.dev), not for the tenant.

The private path has three parts:

1. **Container proxy** (`apps/fouria/cdp-proxy.mjs`, started by
   `start-openclaw.sh` in `launch_gateway_process`). It listens on
   `127.0.0.1:9223`, serves the small CDP HTTP surface (`/json/version`,
   `/json/list`, `/json/new`, `/json/close/:id`) and a WebSocket command channel,
   and forwards each command to `http://cdp.internal/command`. It is detached
   with `setsid` and closes fd 9 so it never holds the gateway startup lock.
2. **Worker bridge** (`src/sandbox-cdp-bridge.ts`, registered in
   `TenantSandbox.outboundByHost`). `POST /command` runs the CDP command through
   the tenant's **Sandbox Durable Object** (`cdpCommand`), `DELETE /session`
   closes the browser, and `GET /health` is a liveness probe. The Sandbox SDK
   resolves `cdp.internal` inside the Worker, so the hop never reaches the
   internet and Cloudflare Access never sees it.
3. **Session ownership** (`src/cdp-session.ts` + `TenantSandbox`). The browser
   is launched lazily from `this.env.BROWSER` and kept in the DO across the
   proxy's per-action WebSockets, serialized by a command queue. The Browser
   Rendering credential never enters the container. The CDP handlers are the
   same `handleCDPMethod` used by `/cdp`, so the two transports cannot drift.

Moltlazy writes the matching **remote-CDP profile** (fill-only, user-editable):
`browser.enabled=true`, `browser.defaultProfile="cloudflare"`,
`browser.profiles.cloudflare` with `cdpUrl="http://127.0.0.1:9223"` and
`attachOnly=true` (a loopback `cdpUrl` without `attachOnly` is treated as a
local managed Chrome), and
`tools.alsoAllow: ["browser"]` (the `coding` profile excludes the browser tool).
A restored boot re-applies `--only gateway,browser` so the profile self-heals
from an older snapshot.

Known limit: OpenClaw's Playwright-backed `snapshot`/`evaluate` paths expect
CDP session multiplexing (`Target.setDiscoverTargets`/`setAutoAttach`,
`sessionId` routing) that the raw shim does not implement yet; raw `open`,
`tabs`, navigation and screenshots work. The E2E covers registration, `open`,
and cross-call `tabs`.

#### Cost-tracking dimensions & development expense

Every Analytics Engine cost-tracking row (`ai-query`, `container-active`,
`r2-operation`, `cron-trigger`, `ws-session`, `action-completed`, `action-step`,
`health-check`) carries two trailing blobs — `platform` (`production` |
`staging`) and `environment` (`production` | `dev`) — resolved by
`getCostContext(env)` (`src/metrics.ts` → `resolveInstanceDimensions` in
`src/dev-mode.ts`). The canonical `customer` key is the **base client slug**
(`CUSTOMER_SLUG`, else `tenant-{slug}[-dev]` with the prefix/suffix stripped),
so worker-side rows join against the `client.slug` lerma reconciles. `health-check`
rows (emitted from non-passive `GET /api/health`) feed lerma's uptime/downtime
statistics. Dev/staging rows are non-billable **development expense** — infra +
AI cost of building for a client, tracked in AE and pulled asynchronously by
lerma, never charged to the client's credit pool.

---

## Software Provisioning (declared, re-applied every reprovision)

Software that agent skills install at runtime (a plugin, an `apt`/`npm`
package, a downloaded binary) lives **only** in the container filesystem, so it
is lost when the image is upgraded or the container is destroyed on idle-sleep.
The `softwareProvisioning` feature flag makes that software declarative:

| Step              | Where                                                                                                                                                                             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Declaration       | `customers/<slug>/software.json` in `BACKUP_BUCKET` (`PUT /api/admin/software/manifest`), `{ id, check, install, verify?, timeoutSec? }` per package                              |
| Materialization   | `materializeSoftwareManifest()` → `/home/openclaw/.openclaw/software.json` (inside the backup root, so a warm restore also carries it)                                            |
| Reconcile         | `moltlazy software provision` (`packages/moltlazy/src/software/`): `check` → skip, else `install` → optional `verify`; per-package timeout, **non-fatal**, exit 0                 |
| Boot trigger      | `ensureGatewayImpl` dispatches the CLI **detached** (`void spawnSoftwareProvision(...)`) just before `start-openclaw.sh start`, so it runs in parallel and never blocks readiness |
| On-demand trigger | `POST /api/admin/software/provision` (ADMIN/ROOT) runs the same CLI to completion, bounded by `SOFTWARE_PROVISION_RUN_TIMEOUT_MS` (5 min)                                         |

It applies on **every provisioning cycle** — cold start, restored boot, idle
wake, image upgrade, and `/debug/restart-openclaw` — because it lives in the
Worker's `ensureGatewayImpl` rather than in the (skippable) setup body of
`start-openclaw.sh`. A soft reload (`SIGUSR2`) does not re-run it; nothing is
lost there.

Outputs: `/tmp/software-report.json` (structured; read by
`GET /api/admin/software/report`) and `/tmp/software-progress.log` (ticks; read
by `/api/startup-progress` as `softwareLog`). The CLI truncates the progress log
at the start of every run because `/tmp` survives an in-place restart, so
otherwise ticks from earlier boots would be interleaved.

Operational notes:

- Both `GET` endpoints return `enabled` so the dashboard can distinguish
  "flag off", "instance asleep", "not run yet" and "nothing declared"; the
  homepage widget's **Run provisioning** button (ADMIN/ROOT) calls the on-demand
  endpoint and renders the fresh report.
- A failed package stays `failed` in the report and does not abort the rest of
  the manifest or the boot. A report is only rewritten when the CLI actually
  runs, so a stale report can exist inside one container after the flag is
  switched off — the `enabled: false` marker is what the UI keys on.
- The manifest is **arbitrary shell** executed in the tenant container. The
  upload and on-demand routes are ADMIN/ROOT-only; there is no command allowlist
  by design (the container is single-tenant).

---

## Container Startup Flow (`start-openclaw.sh`)

`start-openclaw.sh` takes a phase argument (`$1`):

| Mode             | Behaviour                                                                                                                                                                                                                |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `start`          | Production cold-start (single entry point): acquire the container-wide startup lock **first**, then run the full setup body, then become the single OpenClaw owner — `exec`ing the gateway while still holding the lock. |
| `setup`          | Standalone/smoke only: run everything up to (not including) the gateway launch, then exit.                                                                                                                               |
| `launch`         | Standalone/smoke only: acquire the container-wide launch lock and become the single OpenClaw owner (or wait for the current owner to become healthy), then `exec` the gateway.                                           |
| `post-start`     | After the gateway is ready — dev automation + boot-log reset, then exit. Dispatched **detached** (fire-and-forget) by the Worker so it never extends the shared start promise.                                           |
| `full` (default) | Manual/back-compat + boot smoke: run setup, then launch + poll the gateway in the background, then exit.                                                                                                                 |

The Worker's `ensureGateway` drives production cold-start as a **single**
`/usr/local/bin/start-openclaw.sh start` call followed by `post-start`. Unlike
the standalone phases, `start` acquires the container-wide startup lock **before
running setup and holds it through the Gateway `exec`**, so setup and launch are
serialized under the same owner — the Worker no longer invokes `setup` and
`launch` separately. The gateway is **not** a shell background job: the `start`
owner `exec`s `openclaw gateway run` while holding the lock. The lock is held for
the **Gateway process lifetime** via the inherited fd (released exactly when the
gateway exits), so the supervised process _is_ the gateway — tracked by
`listProcesses`/`getProcess` for the life of the container and killable via its
handle. A concurrent invocation that loses the lock polls `/health` and exits 0
without starting a duplicate owner. The serial
`test/e2e/ac_gateway_startup_concurrency.txt` suite checks that exactly one owner
exists and that neither the migration-input-change error nor the SQLite
offline-maintenance error appears. The standalone `setup`/`launch` modes (used
only by smoke/manual runs) keep the older split behaviour — a `setup` `flock`
that serializes the slow setup body and is released before a separate `launch` —
which is why ownership there is enforced solely by the dedicated launch lock.
`full` keeps the legacy background-launch behaviour for manual runs and
`test:gateway:boot`.

**Restored containers skip the setup body (zero-data-loss).** The R2 restore
happens in the Worker (`restoreIfNeeded` → `sandbox.restoreBackup()`), _before_
`start-openclaw.sh` runs — the script never performs the restore. When the
launch runs against a container that already holds tenant state, `ensureGateway`
passes `OPENCLAW_RESTORED=true` in the launch env (see
`hasRestoredTenantState`), and the script skips the entire setup body — onboard →
plugin installs → `moltlazy patch` → config validate — emitting those phases as
skipped and launching the gateway directly from the restored filesystem. The
restored state is therefore authoritative: the cold-start budget is spent
launching it, not re-mutating it (which could diverge from or clobber the
snapshot, and which wastes startup time). A brand-new tenant (no state, no
backup) still runs the full setup body and onboards normally; the image ships no
`openclaw.json`, so "state present" is exactly "already initialized/restored".
Each boot rewrites a durable receipt at `/tmp/openclaw-restored-setup-skipped`
(written on the skip path, removed on the setup path) so an operator can confirm
which path ran; the local backup E2E
(`test/e2e-backup/zzz_backup_restore_skip_setup.txt`, opt-in via
`E2E_BACKUP_RESTORE=1`) asserts the receipt and that the restored setup state is
left byte-for-byte intact.

`OPENCLAW_RESTORED` is decided by **`hasRestoredTenantState()`**, which combines
the in-memory restore status (`getRestoreStatus()` — `hadBackup && restored`, set
by `restoreIfNeeded`) with the container-state probe (`containerHasState` — the
restored `openclaw.json`, legacy `/root` or v1 `/home` layout). The in-memory
result alone is **not** sufficient, because the launching isolate is frequently
not the one that restored:

- the durable boot job is two steps (`restore`, then `launch` in a later alarm),
  so a DO eviction in between loses `_restoreState`;
- any request that reaches the proxy while the restored container has not been
  launched yet (the loading page polls, the browser keeps its tab open, an API
  call arrives) calls `ensureGateway` from its own isolate, where
  `restoreIfNeeded` only runs the cheap request-scoped check and never records a
  restore status;
- `clearPersistenceCache()` (freeze / resume / stale-container recovery) resets
  it too.

Without the durable fallback those launches passed no `OPENCLAW_RESTORED`, so
`start-openclaw.sh` re-ran the whole setup body — re-onboarding and re-installing
plugins over the just-restored snapshot (observed on restored staging tenants
after a rollout/reprovision). The restore decision is deliberately **not** keyed
on the durable R2 `restore-needed` latch, which is a separate cross-isolate
retry/fence (written before `sandbox.restoreBackup()`, cleared only after restore

- legacy-state materialization succeed): its absence is normal while idle and
  after a successful restore, and the restore itself is triggered by the
  container-state probe plus the committed recovery point (`stable-backup.json`).
  The loading page's "Restoring…" spinner observes the in-memory restore status the
  same way, so it appears the moment a cold restore is in flight regardless of
  whether the marker was present in R2.

**One targeted exception: the restored boot re-applies only the `gateway`
section.** The Worker-critical `gateway` config (`trustedProxies`, `auth`,
`controlUi.allowedOrigins`) must self-heal even on a restored boot: a
warm-restored `openclaw.json` can carry a stale `gateway.trustedProxies` (e.g.
the pre-2026-09 default `['127.0.0.1','::1']`) that omits the Cloudflare Sandbox
bridge range (`10.1.0.x`) the Worker proxy connects from, so OpenClaw rejects
every proxied request with `proxy_attribution_required` (symptom: the tenant
serves the gateway's JSON error `{"error":{"type":"proxy_attribution_required"}}`
on the root HTML/WS proxy). The skip path therefore runs
`moltlazy patch --only gateway --file "$CONFIG_FILE"` (bounded, non-fatal): it
heals `trustedProxies` (drop catch-alls, union the required ranges — see
`healTrustedProxies`) and re-asserts auth/origins, but never regenerates the
user-editable `agents`/`session`/`tools`/`logging` sections nor re-seeds
workspaces, so the snapshot's user data is untouched (the backup E2E fingerprints
`moltlazy-session.json` + a sentinel, both unaffected).

**The skip is guarded on the restored config actually existing.** A "restored"
boot can carry partial container state but no materialized `openclaw.json` — an
interrupted restore (a container-image rollout changes the runtime mid-restore),
or `restoreIfNeeded`'s "container already holds tenant state; skipping duplicate
restore" short-circuit, which skips the legacy-state materialization yet still
reports `restored=true` and therefore sets `OPENCLAW_RESTORED=true`. Skipping the
setup body then launches OpenClaw with **no config at all** (observed on
`tenant-tarjetero-dev` after the `v2026.10.13` rollout). `start-openclaw.sh`
therefore only honours `OPENCLAW_RESTORED=true` when `$CONFIG_FILE` exists;
otherwise it logs a warning and falls through to the setup body, which
onboards/repairs the config instead of launching a configless gateway.

**The cold boot is a durable `ActivityScheduler` DO job, not a request
`waitUntil()`.** Cloudflare cancels a request's background (`waitUntil`) work a
short time after the response, so a cold boot driven from a request was killed
mid-restore and re-attempted forever: `[Gateway] Gateway start already in
progress … (phase=restore, age=686365ms)` with `start-openclaw.sh` never run,
141 `waitUntil() tasks did not complete` warnings, 14× re-decrypt of the same
archive, and 64 probe processes hammering the Sandbox DO (observed on
`tenant-client-dev`). The boot is now a durable job (`src/activity/boot-job.ts`)
on the `ActivityScheduler` DO: `POST /api/warmup`, non-passive `GET /api/health`
and `/api/startup/ws` only **request** it (`requestBoot`, deduped) and return;
the alarm performs `restoreIfNeeded` → `ensureGateway` and retries on a bounded
lease independent of any request. When the `ACTIVITY` binding is absent (local
dev / unit tests) the routes fall back to the request-scoped `waitUntil` boot.

**A request never owns a destructive restore.** Even with the durable boot job,
every `ensureGateway` call on a cold path re-asserts the restore barrier. A
request that would actually need an archive restore now **defers** it:
`restoreIfNeeded(..., { restoreOwner: 'request' })` (the default for
`ensureGateway`) performs only the cheap non-destructive checks — the
`restore-needed` latch and `containerHasState()` — returns immediately when the
container already holds tenant state, and otherwise throws
`RestoreDeferredToBootJobError`. `ensureGateway` catches it, calls
`requestDurableBoot()`, and throws `GatewayBootPendingError`, so the proxy/route
answers `503` (or the loading page) while the DO alarm performs the restore +
launch. Durable callers (the `ActivityScheduler` boot/wake jobs and the queue
consumer) pass `restoreOwner: 'durable'` and run it inline. The proxy no longer
calls `restoreIfNeeded` itself. Motivation: on `tenant-tarjetero-dev` a 350 MB
encrypted archive was re-decrypted and `sandbox.restoreBackup` aborted at
~30.1 s (`backup.restore error … The operation was aborted`) on every cold
request, because the restore ran under a request execution context that
Cloudflare cancels shortly after the response — a reboot/restore loop while the
tenant was actively used.

Three supporting bounds keep a stalled boot from hiding behind the long restore
phase: the SDK `sandbox.restoreBackup` RPC is bounded by
`RESTORE_BACKUP_TIMEOUT_MS` (fail fast on a hung control channel); the restore
reports sub-phase progress (`onProgress`) so `isGatewayStartStuck` judges the
`restore` phase on `GATEWAY_RESTORE_NO_PROGRESS_MS` of **no progress** rather
than the full 11-min budget; and the persistence `withLease` releases the shared
R2 lease when its enclosing request aborts (`PersistenceDiagnosticContext.signal`),
so a cancelled operation cannot leave `restore-needed` + a held `backup-lock`
re-triggering the loop.

**Snapshot lease renewal and admin snapshot mode.** The shared R2
`backup-lock` TTL is `STEP_DEADLINE_MS` (12 minutes), with a heartbeat every
quarter-TTL (3 minutes). Every acquire/CAS write is bounded by
`LOCK_WRITE_TIMEOUT_MS` (10 s) and a timeout is reported as
`persistence.lock.write-timeout` + a lost acquire: a conditional `put` that never
settles would otherwise hold the whole durable step (6 min of silence, then
`job-step.interrupted`) instead of failing the acquire. This was observed in
local dev, where a takeover of an expired lease wrote the object server-side
while the promise never resolved — the operator saw only
`Restore — Waiting for the backup lock` for the entire step. Snapshot work
explicitly renews before/after legacy
staging, `sandbox.createBackup`, and encryption, and before final config capture;
restore renews around decrypt and archive application. Renewals are serialized on
one promise chain, awaited before the release tombstone, and best-effort (a failed
renewal is logged but does not abort the snapshot). Structured events report
renewed, late (at least half a TTL), and failed renewals. A 12-minute TTL trades a
longer stale-holder recovery window for avoiding false lease expiry during a
legitimate long step; lock waiters can still hit their bounded acquisition timeout.

`POST /api/admin/storage/sync` uses the durable backup job by default when the
`ACTIVITY` binding is configured (`202` with `jobId`, `409` if another backup/restore
owns the slot). `?inline=1` explicitly requests synchronous completion; `?async=1`
keeps the accepted-then-background behavior. Environments without `ACTIVITY` retain
the inline fallback. E2E scenarios that need an immediate backup id must use
`?inline=1`.

**Never call `output()`/`runShell` on the live gateway handle.** The gateway never
exits, so the SDK's buffered `output()` only opens an RPC log stream, waits for
the local timeout, and disconnects it, surfacing workerd's `ReadableStream
received over RPC disconnected prematurely`. Read diagnostics from the
`/tmp/gateway-stdout.log` / `/tmp/startup.log` tails (`readLogTail`) instead.

**The `setup`/`post-start` phases must stay child-free.** The Worker waits on
them with `waitForExit`, which — despite its name — only settles once the
supervised _terminal_ exits, which never happens while the script leaves a live
background child. `setup` is awaited on the critical path; `post-start` is
dispatched detached but still waited with a bounded `waitForExit` (a lingering
child would keep the post-start process alive until the
`GATEWAY_POST_START_TIMEOUT_MS` kill). So neither phase may start a `tail -f` log
mirror: the mirror is started by the native launch (`buildGatewayRunScript`). A
lingering child in `setup` stalls cold start for the full `STARTUP_TIMEOUT_MS`
and makes the Worker re-run setup on retry (observed: ~11 min to first gateway
launch).

**`post-start` never blocks readiness.** Once the port wait succeeds and
`markGatewayReady` runs, the gateway is serving; `ensureGatewayImpl` dispatches
`start-openclaw.sh post-start` with `void runGatewayPostStart(...)` instead of
awaiting it. Dev instances run `openclaw automations list/create` there (30–60s),
and awaiting it previously held the shared start promise so every request logged
`Gateway start already in progress … waiting...` while `/health` already returned 200. `startShell` dispatches the `sandbox.exec` synchronously before its first
`await`, so the work still runs server-side even if the isolate is evicted;
`runGatewayPostStart` swallows its own errors (best-effort).

The entrypoint runs sequentially:

```
1. Fast-path check: Is gateway already running? → exit 0 (skipped for `post-start`)

2. Serialize concurrent startup/setup/full invocations (exclusive `flock` on
   /tmp/openclaw-setup.lock, ~900s wait). The Worker's in-process
   `gatewayStartPromises` guard is per-isolate, so bursty cold-start polling can
   land several runs in different isolates at once; without the lock they
   all run the slow setup body concurrently, contend on OpenClaw's native
   config/state locks, and stall cold start for many minutes (observed: ~18 min
   to gateway-ready in run 36692775633). Only one body runs; a waiter re-checks
   the gateway fast-path after acquiring and exits 0 once the winner is up.
   In production, `start` takes this lock **before** the setup body and holds it
   through the Gateway `exec`; `post-start` is outside the lock (it runs after
   the gateway is reachable). The standalone `launch` mode has its own dedicated
   launch lock and must not double-lock. `full` releases fd 9 before
   backgrounding the gateway so the child does not inherit the setup lock.

3. Worker-side lifecycle coordination
   - Restore, snapshot, restart, and destroy are coordinated by the Worker.
   - OpenClaw's native lock protects gateway ownership; the startup `flock`
     above serializes onboarding/config/plugin filesystem mutations **and** the
     launch across concurrent Worker isolates — `start` holds it from before
     setup until the gateway is `exec`'d, so a single owner covers both. The
     standalone `setup`/`launch` smoke modes still split the lock (setup lock
     released before the separate launch lock), so within that path single-owner
     launch is enforced by the dedicated launch lock alone.

4. Pre-backup optimization:
   - Move logs to /tmp (excluded from squashfs snapshots)
   - Clean stale backup files

5. Onboard (if needed):
    - Detect Cloudflare AI Gateway auth from env vars
    - Ensure the `cloudflare-ai-gateway` provider plugin is present before
      onboard. It is **pre-installed into the image at build time** (see
      Dockerfile), so this is normally a no-op detection; the install is a
      resilience fallback. Non-interactive onboarding cannot approve plugin
      capabilities, so without the plugin its plugin phase fails and burns the
      60s onboard cap on every cold start. With it present, the phase is a no-op
      and onboarding completes cleanly.
    - Run: openclaw onboard --non-interactive --accept-risk

6. Enable plugins (single working `config patch`):
   - All three plugins are **pre-installed into the image at build time**
     (Dockerfile, unpinned → latest ClawHub release). The installs below run
     only as a fallback when the extension directory is missing (e.g. a warm R2
     restore replaced state). Detection (`plugin_extension_present`) checks that
     directory — the artifact the gateway actually loads — and deliberately does
     **not** read `openclaw plugins list --json`: the registry refresh hangs for
     its full `CLI_TIMEOUT` when ClawHub/npm are slow, and a hung/partial list
     reads as "not installed" and triggers a needless network install (~57s
     ai-gateway + ~74s otel on a full cold boot on `staging-client-dev`). See
     `docs/plans/cold-start-latency.md`.
   - **Build/boot guard:** every ClawHub install is best-effort. A ClawHub
     outage, a transient network error, or a plugin that requires a newer plugin
     API than the pinned OpenClaw runtime (observed: `@openclaw/cloudflare-ai-gateway-provider`
     resolving to 2026.9.9 against the 2026.9.8 runtime) logs a `WARNING` and
     continues — the Docker build no longer fails, and the boot proceeds. The
     enablement patch below only enables a plugin whose extension directory is
     present, so a failed install cannot invalidate the whole patch.
   - cloudflare-unified-billing (when installed) — the Paso4 plugin, served from
     https://clawhub.ai/paso4/plugins/cloudflare-unified-billing. Entry opts
     into `hooks.allowConversationAccess=true` so the
     `agent_end`/`message_received` hooks run (OpenClaw 2026.8.2 blocks
     non-bundled conversation hooks without it)
   - cloudflare-ai-gateway (when installed)
   - admin-http-rpc (always) — the Worker's RPC-first admin read path
   - diagnostics-otel (enabled only when `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`
     is set and not in E2E mode; otherwise explicitly disabled). Exporter
     protocol is always `http/json`. When `WORKER_URL` is set (and
     not in E2E mode) `buildEnvVars()` also derives
     `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT={WORKER_URL}/api/otel/v1/traces`.
     `diagnostics.otel.traces` is opt-in via `OTEL_TRACES_ENABLED=true`
     (default: disabled); the config shape is unit-tested in
     `src/gateway/diagnostics-otel.ts` (`buildDiagnosticsOtelConfig`).
   - The patch must NOT carry `plugins.bundledDiscovery` (removed in OpenClaw
     2026.9.x — including it fails schema validation and drops the whole patch)
     nor `plugins.allow` (an allowlist disables every bundled provider not
     listed). Memory search is disabled via `memory.search.enabled=false`.
   - **memory-core dreaming cadence (dev):** the same patch slows the dreaming
     sweep to weekly (`plugins.entries.memory-core.config.dreaming.frequency =
     "0 3 * * 1"`, default `0 3 * * *`) when `OPENCLAW_DEV_MODE=true`. The full
     sweep is a heavy isolated agent turn (~6 min, `toolsAllow: ["*"]`) that the
     cron replays on every wake (`cron: running missed jobs after restart`) — a
     major RSS spike source on the 1-vCPU dev sandbox (observed 1.6 → 2.08 GiB
     during a replayed sweep). Production keeps the OpenClaw default cadence.
   - **Version drift:** the plugin-API mismatch above means a build on the
     pinned runtime ships **without** the Cloudflare provider plugins until
     OpenClaw is bumped (see the openclaw-migration skill) or the specs are
     pinned. The guard keeps the image and boot healthy in the meantime; it does
     not restore the missing provider by itself.

7. Patch config via moltlazy:
   - Run: node /app/moltlazy/dist/cli.js patch
   - Generates per-section includes (moltlazy-session/tools/logging.json) injected
     via per-section $include; gateway/agents written inline (no root $include)
   - References the gateway token as an env SecretRef (kept in the container environment)
   - Run: node /app/moltlazy/dist/cli.js validate
   - **Validation is FATAL**: an invalid config would prevent the gateway from
     booting, so the script aborts (exit 1) instead of proceeding.

8. Harden file permissions:
   - chmod 600 moltlazy-*.json, chmod 700 .openclaw

9. Start gateway:
    - `start` mode (production): after setup, `exec` the gateway under the same
      startup lock the phase acquired at step 2 — the Worker no longer calls
      `launch` separately (its stdout log mirror / `post-start` run as usual)
    - `setup` mode: smoke/manual-only standalone path — `mark_step gateway`,
      then exit 0 without launching the Gateway. The production Worker uses the
      atomic `start-openclaw.sh start`, which holds the startup lock and `exec`s
      the Gateway, followed by `post-start`
    - `full` mode: background `openclaw gateway run --port 18789 --verbose
      --allow-unconfigured --bind lan`, poll `/health`, then exit while the
      gateway child remains detached
```

### Startup Progress (`/api/startup-progress`)

The loading page shows a transparent boot stepper + live log while the gateway
starts. `start-openclaw.sh` declares its boot phases in a single `STEPS` array
(single source of truth) and emits progress to two `/tmp` files on each phase
(kept in `/tmp` so they stay out of squashfs backups and R2):

| File                         | Content                                                                       |
| ---------------------------- | ----------------------------------------------------------------------------- |
| `/tmp/startup-progress.json` | `{ steps:[{id,label}], currentStep, currentStepId, totalSteps, skipped, ts }` |
| `/tmp/startup.log`           | Clean, **secret-free**, chronological boot detail (verbosity-gated)           |

`mark_step <id> [skipped]` resolves the label/index from `STEPS`, writes the
progress JSON atomically (`mktemp` + `mv`), and echoes the transition to the
container stdout. `log_boot <level> <msg>` writes clean boot detail to
`/tmp/startup.log`. Step transitions are intentionally **not** written to the
log — the loading page's stepper shows step progress, so the log stays clean
(no duplicated step/checked info). Adding a phase = one `STEPS` row + one
`mark_step` call — the loading page renders the list dynamically, so no other
code changes.

`GET /api/startup-progress` (public, no warm-up) reads both files and returns
`{ available, steps, currentStep, currentStepId, totalSteps, skipped, ts, log }`.
It is bounded by `STARTUP_PROGRESS_TIMEOUT_MS` and never calls
`restoreIfNeeded`/`ensureGateway`, so it adds no load to the health path.

The loading page consumes this data over the **`GET /api/startup/ws`** WebSocket
(not by polling). Each push is a combined `{ ...HealthSnapshot, progress, boot,
jobLog, restoreProgress?, recovery?, repair? }` payload — the health snapshot
(which drives readiness/redirect) plus the boot steps + log, the durable boot
job, and the formatted job-status log. On socket open the endpoint registers
**one** `ensureGateway` warm-up with `waitUntil`; every subsequent health
snapshot is **passive/read-only**. It does not re-enter `restoreIfNeeded` on
every push. This distinction matters when restore is blocked on an R2
`backup-lock`: the former per-push warm-up queued a new 120-second restore-lock
waiter every 5–30 seconds, which could starve gateway startup and the
pre-replacement checkpoint while the UI kept its socket open.

The loading page shows the boot log panel **while the gateway is starting up** —
both while `start-openclaw.sh` is running its setup body (`processType ===
'script'`) and while the gateway binary is binding its port (`processType ===
'gateway'`). The `processType === 'gateway'` state no longer hides the log:
`start-openclaw.sh start` `exec`s the gateway and a `start_boot_log_stream`
mirror tails the gateway's own stdout (`/tmp/gateway-stdout.log`) into the
curated boot log, so the log keeps filling with real launch progress (SQLite
migration, plugin load, port bind) instead of going blank for the minutes the
gateway takes to bind — the exact "Launching the gateway — no progress" window
that previously looked stuck. The mirror is killed (`stop_boot_log_stream`) and
the buffer cleared once the gateway is ready.

**The terminal never depends on `/tmp/startup.log` alone.** That file does not
exist during the durable boot job's restore phase (no `start-openclaw.sh`
process yet) and can stay quiet while the gateway binds its port, so the panel
used to sit blank next to a slow-step notice like `Starting OpenClaw Gateway
(188s)`. The Worker now formats the durable state it already reads into a
`jobLog` string (`activity/job-status-log.ts`, pure and unit-tested):

| Line                                                             | Source                                                                          |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| `[HH:MM:SS] Boot queued / running / retrying / failed — …`       | `ActivityScheduler.getBootJob()` (step, attempt, elapsed, next attempt, reason) |
| `[HH:MM:SS] Restore — <phase label> (<percent>%)`                | `getRestoreProgress()` (idle annotation and terminal failure included)          |
| `[HH:MM:SS] Repair running — openclaw doctor --fix, attempt N/3` | `getActiveMaintenanceJob()`                                                     |

`/api/startup/ws` pushes `jobLog` on every frame, and the non-passive
not-yet-ready `/api/health` / `/api/health-snapshot` carries the same fields for
the page's HTTP polling fallback. `loading.html` renders the container's own log
when it has content and otherwise the formatted job status, labelling the
terminal source (`fouria@gateway: /tmp/startup.log` vs `boot job status`). The
panel is therefore visible for the whole restore phase instead of being replaced
by the brand logo, and the brand logo only appears when there is nothing at all
to show (no process and no durable job state).

While a cold restore is in flight, the page also shows a **restoring spinner**
beside the status line (title "Restoring FourIA" / status "Restoring data from
backup…"). It is driven by the health snapshot's in-memory `restoreStatus`
(`hadBackup && !restored`) — the same signal the `startup.ok` probe uses to
report "Backup found but not restored" — or the durable `recovery.restoring`
marker (below), and is cleared the moment the restore commits (or the start
fails/redirects). It is intentionally **not** keyed on the R2 `restore-needed`
latch, so the spinner shows for a restore that would have run anyway (a fresh
container with a committed recovery point).

The page also **names the backup + date being loaded**, so a cold restore is
legible instead of a generic spinner. `/api/health` (non-passive, not-yet-ready)
and `/api/startup/ws` attach a durable `recovery` object read from R2
(`readRecoveryState`): `{ restoring: { id, created } | null, committed: { id,
created } | null }`. `restoring` is the in-flight `restore-target.json` written
by `restoreIfNeeded`/`restoreBackupById` just before `sandbox.restoreBackup` and
cleared on success; `committed` is the `stable-backup.json` recovery point a cold
boot restores. It is read from R2 (not the per-isolate `restoreStatus`) because
the durable boot/restore runs in the `ActivityScheduler` DO, whose in-memory
state is invisible to the request isolate serving the page. Lerma's frequent
**passive** poll never pays the two R2 reads; they are bounded by
`RECOVERY_STATE_TIMEOUT_MS` and fail soft to `null`.

During a restore the loading UI is a **progress bar synchronized with the DO's
own phase events** (not a client-side timer). `restoreIfNeeded` /
`restoreBackupById` call their `onProgress(phase)` hook on every
`persistence.phase`; the DO wires that to `recordRestoreProgress`, which maps the
phase to a label + a monotonic fraction (`describeRestoreProgress`, in
`activity/restore-progress.ts`) and exposes it via `getRestoreProgress()`.
`/api/startup/ws` pushes it as `restoreProgress: { phase, label, fraction }`;
`loading.html` renders the bar and hides the stepper while a restore is in
flight, then switches back to the container's `startup-progress.json` stepper
once the gateway launch begins. A request-scoped restore with no DO feed shows an
indeterminate bar. `/api/admin/boot-state` also returns `restoreProgress` for
operators.

The label stays honest when a restore is interrupted. A failed attempt is marked
before the durable job rethrows (`recordRestoreFailure`), so the bar reads
`Restore attempt failed during "<phase>" — retrying` instead of the last phase.
A kill that skips that catch (e.g. a Worker CPU-limit kill) leaves the last
durable phase in place, so `getRestoreProgress()` appends
`— no progress for N min` once the record is idle for 2 minutes. The fraction
never moves, so a stale bar is labelled rather than presented as live.

**A terminal failure must not be presented as a retry.** `— retrying` is only
true while the durable job still has attempts left. When the boot job (or a
manual restore job) exhausts them, the DO records
`describeRestoreTerminalFailure` (`recordRestoreTerminalFailure`) with
`terminal: true`: the label becomes `Restore failed — <reason>` (or
`Gateway launch failed — …` for the `launch` step), `describeRestoreProgressIdle`
stops appending an idle note (a finished failure is not stalled), and the next
`boot.start` clears the flag. `readRecoveryState` likewise stops reporting a
`restore-target.json` whose `outcome` is `failed` as `restoring`, and
`loading.html` gives a `failed` boot precedence over the leftover progress bar
and status line (`Boot failed after N attempts: <reason>. Refresh the page to
retry.`). Without this, a tenant whose recovery point could not be applied sat on
`Restoring the backup — no progress for 10 min` forever: the three failed
attempts took ~70 s, and the page then rendered the stale record (with the
restoring spinner, because any `restore-target.json` counted as `restoring`)
instead of the terminal failure the job had already recorded.

Archive decryption (`decryptStream` / `encryptStream` in
`crypto/backup-encryption.ts`) assembles each 4 MiB chunk from R2 reads with a
byte queue that copies each byte once. The earlier `concat`-per-read buffer
recopied the pending chunk on every ~64 KB read, which exhausted the Worker CPU
budget on large recovery archives (`Worker exceeded CPU time limit` during
`Decrypting backup …`). Measured in isolation on a 96 MiB archive: decrypt
1507 ms → 140 ms at 64 KB reads, and 5614 ms → 137 ms at 16 KB reads.

The R2 binding delivers the archive in **4 KiB** reads (local workerd: 64 MiB
arrives as 16,385 reads), so the old per-read `concat` was the dominant cost.
Each restore logs `[crypto] Decrypt read profile` with the read count, bytes,
largest and average read, and chunk count, so the read pattern can be checked
on a live tenant.

The loading page uses a **single push channel** — `GET /api/startup/ws` — and
does not poll while the WebSocket is open. Flow:

1. **Handshake** — the first WS push reports whether the gateway is already
   running; if so the page redirects immediately, otherwise the stream keeps
   warming and pushing step updates.
2. **Boot phase** — while boot steps are in progress, the sandbox pushes the
   health snapshot + `progress` (steps + log) every
   `STARTUP_WS_POLL_INTERVAL_MS` (~5s). No client polling. If a transient
   DO↔container transport drop makes the probe fail, the stream re-sends the last
   known state and backs off exponentially up to `STARTUP_WS_MAX_BACKOFF_MS`
   (30s) until the control session reconnects, so the page keeps waiting and the
   same error is not surfaced/re-logged every 5s.
3. **Fallback** — if the WebSocket drops (e.g. local dev server, a network
   blip), the page falls back to slow HTTP polling (`/api/health` every 5s)
   **indefinitely**. The page has no wall-clock budget or retry cap of its own: the
   cold boot is driven out-of-band by the `ActivityScheduler` DO, which owns
   retries and terminal failure. A run of failed polls does **not** hard-fail the
   page ("Unable to reach server"): a container-image / Worker deploy briefly
   resets the DO↔container control session, so the page keeps retrying and shows
   "Reconnecting — the instance may be updating…".

The page only declares **failure** on a _permanent_ container error
(`degraded: true` in the `/api/health` snapshot — deleted image, missing app,
where a rescue is required), surfaced immediately via the container error, or on
the durable **boot job's** terminal `boot.status === 'failed'` state. Every other
state — still booting, gateway reachable but warming up, restore in progress —
keeps waiting, because a cold start legitimately takes 3–5+ minutes (onboard +
gateway warmup; the provider plugins are pre-installed in the image, so plugin
installs only run on the fallback path). Mixed probe states (e.g. `startup.ok`
lagging behind `readiness.ok` while a backup restore is pending) are **not**
treated as failures, and there is no client-side timeout that interrupts a
long-but-progressing boot.

The loading page treats the canonical gateway status (`status === 'running'`,
driven by the readiness probe) as "ready" rather than trusting `startup.ok`
alone, because `startup.ok` is gated on backup-restore state that can lag a
healthy, already-serving gateway.

**A slow boot step is surfaced as an alert.** Once the current boot step has run
longer than `SLOW_STEP_MS` (120s), the loading page shows a non-fatal notice
naming the step and its elapsed time (e.g. "Still working on "Installing
plugins" (185s)…"), driven by the script's own `progress.ts`. It is **not** a
failure — cold starts legitimately take minutes — and it is hidden on ready/error.
It exists so an operator can tell a slow-but-progressing boot from a stuck one
without opening logs or the Workers Observability console. (See
`docs/plans/cold-start-latency.md`.)

The Worker's root HTML route (`app.all('*')` in `src/index.ts`) uses the **same**
ready predicate (`snapshot.readiness.ok`, i.e. `status === 'running'`) — it must
**not** additionally require `startup.ok`. Requiring both made the two disagree
whenever `readiness.ok` was true but the startup probe lagged (a restore still
settling, a stale cached startup result, or a `start-openclaw.sh start` wrapper
not yet replaced by the gateway binary): the loading page redirected to `/`
while the root route re-served the loading page — an **infinite reload loop**
where OpenClaw never appeared to boot (observed on `tenant-client-dev`). Keep the
two predicates in lockstep.

**The gateway WebSocket proxy gates on the same readiness predicate.** The
catch-all WS path in `src/index.ts` must call `isGatewayReadyForProxy()` before
`sandbox.wsConnect()`. `wsConnect` (and `containerFetch`) wait for their target
port **inside the Sandbox DO** for `containerTimeouts.portReadyTimeoutMS`
(`CONTAINER_PORT_TIMEOUT_MS`, 90s). During a container-image rollout the
container is healthy — Cloudflare reports it active and the container logs are
clean — but the gateway has not bound 18789 yet, so that one RPC parks for the
full 90s and every other RPC (health, components, other proxies) queues behind
it, surfacing as "Unable to reach server" on the loading page. When the gateway
is not ready the WS path returns a fast, retriable `503` + `Retry-After: 5`
(and fires `ensureGateway({ waitForReady: true })` via `waitUntil` so the cold
start still progresses) instead of calling `wsConnect`. The HTML and non-HTML
HTTP proxy paths already gate the same way.

A `"Timed out waiting for runtime port"` error — the DO's port wait expiring
because the container is up but the service has not begun listening — is
classified by `isRuntimePortTimeoutError()` (`src/gateway/sandbox-recovery.ts`)
as **transient**: the WS path returns a retriable `503`, never a `502`, and it
must **not** trigger `killGateway`/stale-container recovery, which would discard
a healthy container mid-rollout. It is deliberately excluded from
`isStaleContainerError`.

Likewise, `Network connection lost`, `WebSocket connection failed`, and the
Worker's bounded `Operation timed out after …` from `sandbox.wsConnect()` are
transient Sandbox DO WebSocket-transport failures, not proof that the gateway
process died. The proxy returns `503` with `Retry-After: 5` and leaves the
container and R2 state untouched. This matters when the admin HTTP RPC
health/status path still works: it uses `containerFetch`, a separate transport
from the browser's WebSocket connection.

**Security invariant:** the boot log contains curated `log_boot` lines — the
script's raw stdout (which can contain the AI Gateway API key as an `openclaw
onboard --cloudflare-ai-gateway-api-key …` argument) must never be written to
`/tmp/startup.log`. In addition, once the gateway launches, a
`start_boot_log_stream` mirror tails the **gateway's** own `--verbose` stdout
(`/tmp/gateway-stdout.log`) into the boot log so the loading page can follow
launch progress; this is the same output already surfaced to ADMIN/ROOT via
`/debug/gateway-logs` and `/api/admin/logs`, and it is stopped (`stop_boot_log_stream`)
and the buffer truncated at readiness. The gateway's startup output is a distinct
stream from the script's onboard stdout (which is where the API key appears as a
CLI argument and is still never captured).

Verbosity is controlled by `STARTUP_LOG_VERBOSITY` (`steps` | `detailed` (default) | `verbose`),
passed through from the Worker via `buildEnvVars()`.

### `moltlazy patch` — Config Injection

`moltlazy patch` is the **mandatory config system**. It runs every startup and generates:

| Config Section     | Type          | Content                                                             |
| ------------------ | ------------- | ------------------------------------------------------------------- |
| `gateway.*`        | Mandatory     | Port, mode, bind, auth                                              |
| `session.*`        | Mandatory     | Session scope, DM settings                                          |
| `tools.toolSearch` | Mandatory     | Tool search configuration                                           |
| `logging.*`        | Mandatory     | Log levels, format, destinations                                    |
| `agents.defaults`  | Always-on     | Default model, system prompt                                        |
| `agents.entries`   | Always-on     | Premade agents keyed by id (dev-only `fouria-builder`, `fouria-qa`) |
| `skills.entries`   | Always-on     | Built-in skills                                                     |
| `memory.*`         | Feature-gated | Memory backends (knowledge graph)                                   |
| `channels.*`       | Env-gated     | Channel tokens (Telegram, Discord, etc.)                            |
| `plugins.*`        | Feature-gated | Plugin management config                                            |

The moltlazy-owned sections (`session`/`tools`/`logging`) are generated into per-section include files and injected into `openclaw.json` via per-section `$include` directives — **not a root `$include`** (root includes fail closed for Control UI saves / `config.apply`). They are regenerated from scratch every startup — never edit manually. `gateway.*` and `agents.*` are written inline into `openclaw.json`; the gateway token is an **env SecretRef** (resolved from the container environment, which `start-openclaw.sh` keeps set).

**Memory-pressure limits.** Two moltlazy-owned defaults keep the session store
from growing into the sandbox's memory/disk: `agents.defaults.compaction`
(`maxActiveTranscriptBytes: '8mb'`, `keepRecentTokens: 12000`,
`midTurnPrecheck.enabled: true`) and `session.maintenance`
(`maxDiskBytes: '512mb'`, `highWaterBytes: '384mb'`, `maxEntries: 500`,
`pruneAfter: '14d'`, `resetArchiveRetention: '14d'`,
`coldStorage: { enabled: true, afterDays: 7 }`). OpenClaw's own defaults
(20 MB transcript, 10 GB per-agent disk budget, 5000 entries, 30-day retention)
are sized for a developer machine and are too large for a 4–12 GB container.
These limits bound growth but cannot reclaim worker-thread heaps that already
grew; the hard restart is still the only RSS reset (see
[docs/plans/sandbox-memory-usage.md](../../docs/plans/sandbox-memory-usage.md)).

### Admin RPC (`admin-http-rpc`) — RPC-first read path

The Worker reaches the gateway's admin control plane over the gateway's loopback
HTTP listener using OpenClaw's bundled **`admin-http-rpc`** plugin
(`POST /api/v1/admin/rpc`). The typed client lives in `packages/moltlazy/sdk`
(`createClient`, HTTP `RpcClient`); `apps/fouria/src/gateway/admin-rpc.ts`
supplies the transport — a `fetch`-shaped adapter over `sandbox.containerFetch`
(global `fetch` cannot resolve the container's localhost).

This replaced the CLI-first read path for the hot dashboard routes. CLI
invocations (`startShell(sandbox, 'openclaw ...')`) reap a fresh process and
re-establish a WebSocket on every call (~10–15s); the HTTP RPC path is a single
container request. Routes attempt RPC first and fall back to the CLI
(`runCliFallback` / the existing parsers) whenever the transport is unavailable
or returns an unexpected shape:

- `GET /api/admin/devices` → `device.pair.list`
- `POST /api/admin/devices/:id/approve` → `device.pair.approve`
- `GET /api/admin/models` → `models.list`
- `GET /api/admin/channels/probe` → `channels.status { probe: true }`

**Enablement:** `start-openclaw.sh` enables the plugin with a dedicated
`plugins.entries["admin-http-rpc"].enabled = true` config patch. It is kept
separate from the batched plugin patch because OpenClaw 2026.9.x rejects that
batch (`plugins.bundledDiscovery` was removed upstream), and setting
`plugins.allow` would disable every bundled provider that is not listed. Enablement
applies to a running gateway automatically; no restart.

**Security:** the plugin exposes a full operator control-plane surface on the
gateway listener. It is only reached in-process via `sandbox.containerFetch`;
`src/index.ts` shadows `/api/v1/admin/rpc` with a `404` before the public
catch-all gateway proxy, so it is never reachable through a tenant hostname.
Never remove that guard.

**Cold-start guard (health probe).** `sandbox.containerFetch` auto-starts the
container and **blocks until the requested port is ready**. The RPC-first health
probe (`getRpcHealthSnapshot`, `src/gateway/health-probes.ts`) therefore only
issues the RPC once a gateway process is actually detected; while the container
is cold, or the production `start-openclaw.sh start` wrapper is still running
setup, it returns `null` — RPC health is skipped until the local Gateway
health/port is open — so `/api/health` falls back to the exec-based
`getHealthSnapshot` (which never waits on the gateway port). Callers that already hold a process record pass it as
`{ proc }` to skip the extra scan — `buildHealthSnapshot` does this with the
`findExistingGatewayProcess` result it already fetched.

**Cold-start boot lifetime.** #318 split the boot so the **Worker** launched the
gateway _after_ `start-openclaw.sh setup` exited. `/api/health` is bounded to a
5s snapshot, so the request returns long before the (multi-second) setup
finishes; workerd then cancels the unregistered `ensureGateway` continuation and
the container-side setup completes with **no gateway ever launched** (the
observed cold-start stall: 68 processes, setup exit 0, every `curl /health` exit
7). The fix was to keep the boot alive past the response, and the current runtime
contract collapses the production path into a single
`/usr/local/bin/start-openclaw.sh start` call (startup lock acquired before
setup and held through the Gateway `exec`) followed by `post-start` — the Worker
no longer invokes `setup` and `launch` separately. Any route that triggers a cold
boot must still keep it alive past the response with
`c.executionCtx.waitUntil(ensureGateway(sandbox, env, { waitForReady: true }))`
— `waitForReady: true` returns the shared start promise unwrapped (no read
budget) and dedupes onto the promise the bounded snapshot already started.
`/api/health` and `/api/warmup` do this; do not replace it with a bare
`await ensureGateway(..., { waitForReady: false })`.

The SDK's WebSocket transport (`createWsClient`) is **not** used: the current
OpenClaw wire protocol (v4) requires a `connect` handshake and a paired
`operator.admin` device identity, which the shared-token loopback path does not
provide. HTTP RPC over the shared token is the least-effort RPC-first transport
and is what `admin-rpc.ts` implements.

---

## RBAC & Access Control

### Roles

| Role          | Permissions                                   | Access Scope                     |
| ------------- | --------------------------------------------- | -------------------------------- |
| **ROOT**      | `admin:platform`, `admin:customer`, `api:all` | All tenants, all debug endpoints |
| **ADMIN**     | `admin:customer`, `api:all`                   | Own tenant only                  |
| **BASE_USER** | `api:own`                                     | Own tenant, read-only            |

### Dashboard Pages by Role

| Page               | ROOT | ADMIN | BASE_USER |
| ------------------ | ---- | ----- | --------- |
| Dashboard Home     | Yes  | Yes   | Yes       |
| Agents             | Yes  | Yes   | Read-only |
| Connections        | Yes  | Yes   | Read-only |
| Knowledge          | Yes  | Yes   | Read-only |
| Account/Team       | Yes  | Yes   | Read-only |
| Integrations       | Yes  | Yes   | No        |
| Secrets Vault      | Yes  | Yes   | No        |
| Backups            | Yes  | Yes   | No        |
| Releases           | No   | Yes   | No        |
| Diagnostics        | Yes  | No    | No        |
| Debug (`/debug/*`) | Yes  | No    | No        |
| Audit Logs         | Yes  | No    | No        |

> **ROOT** is the maintainers' intended role. It has full access to debugging tools, container inspection, and platform-level diagnostics at `/_admin/diagnostic` and `/debug/*`. ROOT is restricted to diagnostics-only API paths (`DIAGNOSTICS_PATHS` in `src/routes/api.ts`) — management pages (Agents, Connections, Backups, etc.) are ADMIN-gated. As a diagnostic exception, ROOT can still verify backup connectivity (`Test Backup Connectivity` on the Diagnostic Dashboard → `POST /api/admin/storage/test-connectivity`) without the Backups page.

> **Workflow commit tree.** The tenant's workflow git history (commit tree, branches, tags, releases) is **client data**. It is served by `GET /api/admin/releases/commits`, which is deliberately **not** in `DIAGNOSTICS_PATHS`: ROOT is blocked at the diagnostics gate and BASE_USER by `requireAdminOrRoot`, leaving ADMIN-only access. The tree is produced from the sandbox git repository by the release tooling; until that substrate ships the endpoint returns an empty tree. It is no longer rendered on the Backups page — the ADMIN-gated Releases page (`/_admin/releases`) is the promotion surface, and no dashboard view renders the commit tree today.

> In a **harness/dev mode** (`DEV_MODE` or `E2E_TEST_MODE`), `/debug/*` additionally permits `ADMIN` so the automated E2E suite can drive CLI/operator routes from a bare `workers.dev` worker that has no Cloudflare Access or Dispatch Worker in front of it. Real production (`DEV_MODE=false`, `E2E_TEST_MODE=false`) stays ROOT-only per the table above.

### Agent Push (dev → production promotion)

A tenant owns two instances: its `dev` environment builds and validates agents
(the dev-only `fouria-builder` / `fouria-qa` personas), and its `production`
environment serves them. **Agent push** promotes the built content from dev to
production.

The promoted artifact is the **full promotable manifest**, owned by
`@moltlazy` (`packages/moltlazy/src/promotion/index.ts`): `openclaw.json`, the
moltlazy per-section `$include` files, and the agent `workspace/`. The dev-only
agents and all runtime state — `state/`, `credentials/`, `agents/`, `sessions/`,
`secrets/` — are **never** promoted. Provider/channel credentials carried _inside_
`openclaw.json` do travel; only the runtime secret stores are excluded.

Promotion is driven entirely from the Worker API (operator calls on the dev and
production instances); no promotion command runs inside the container or through
OpenClaw. The transport is **private R2, never HTTP**. The dev instance stages the
sanitized bundle in its **own** bucket (`promotion/staged/<id>.tar.gz` +
`promotion/pending.json`); the production instance pulls it through a read-only
`PROMOTION_SOURCE_BUCKET` binding to that sibling bucket. Both buckets stay
separate — production only reads the prefix — and no tenant route is exposed for
the transfer.

Production applies the release as a **custom backup**: it keeps its own runtime
state (sessions, transcripts, the control-plane SQLite) while overwriting config
/ agents / tools and re-applying the `gateway` section for its own `WORKER_URL`,
then soft-reloads. The apply happens **on boot, after the restore barrier and
before the launch** (so a restored tenant still loads the deployed release), or
when an operator calls `POST /api/admin/releases/apply`. It runs under the shared
persistence lease (`withPersistenceLease`), so a scheduled snapshot can never
capture a half-applied tree; it is skipped while the tenant is parked or while a
maintenance window / restore is in flight; and it is idempotent — the container
release pointer (which is part of the snapshot) is the applied marker, so a
restore cannot silently roll it back and an interrupted apply simply reruns on
the next boot.

Routes (`src/routes/api/releases.ts`, `src/services/promotion/r2-source.ts`):

| Route                              | Instance   | Behavior                                                                                                                                                                                                                                                                                                                             |
| ---------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GET /api/admin/releases/commits`  | both       | Workflow commit tree (client data; ADMIN-only, not a ROOT diagnostic).                                                                                                                                                                                                                                                               |
| `GET /api/admin/releases/status`   | both       | The release this instance runs (container pointer, falling back to the applied pointer in R2), plus `available` when the sibling bucket holds a newer release. Also describes this instance: `environment`, `canPromote` (dev), `canApply` (production) and `sourceBinding` (whether `PROMOTION_SOURCE_BUCKET` is provisioned).      |
| `GET /api/admin/releases/history`  | both       | The push history this instance can see, newest first. Dev reads its own `promotion/history/<id>.json` entries; production merges the sibling dev bucket's staged entries with its own applied/failed entries.                                                                                                                        |
| `POST /api/admin/releases/promote` | dev        | Export a sanitized bundle and stage it in this instance's own bucket, targeted at its production sibling. Body `{ message?: string }` records the deployer's commit message; the verified Access identity is stored as the author. Returns `202 { staged: true }`. Refuses to run on a production instance (`409 NOT_DEV_INSTANCE`). |
| `POST /api/admin/releases/apply`   | production | Pull the pending release (R2), apply it under the persistence lease, checkpoint it as a recovery point, and reload the gateway. Idempotent.                                                                                                                                                                                          |

Every push is **content-hashed** (SHA-256 of the bundle, computed with
`crypto.DigestStream` while the bundle is uploaded) and carries the deployer's
optional commit message. The hash identifies the push across both instances,
and the history makes the sequence of pushes auditable without exposing either
bucket to writes from the other side:

- the **dev** instance writes a `staged` entry (`promotion/history/<id>.json`)
  into its own bucket;
- the **production** instance reads those entries through the read-only
  `PROMOTION_SOURCE_BUCKET` binding and writes its own `applied`/`failed`
  entry into its own bucket;
- the history endpoint merges the two views by release id, with the local entry
  authoritative (production's `applied` wins over the dev `staged` record).

There is **no `POST /api/admin/releases/import`** and no `PROMOTION_PUSH_TOKEN`:
nothing is pushed across the wire, so there is nothing to authenticate on the
production side. The R2 binding is the authority and is scoped to the tenant's
own sibling bucket.

All promotion routes (plus the commit-tree endpoint) are ordinary operator routes
behind Cloudflare Access and `requireAdminOrRoot()`. Two practical consequences:

- On the production platform any ADMIN user whose JWT resolves to the tenant
  works; on **staging** the Dispatch gate is ROOT-only, so an operator **session**
  with the Paso4 Root group is required.
- An Access **service token** (`CF-Access-Client-Id`/`Secret`) gets past the edge
  but resolves **no role** (it has no email and no `custom.role`), so the Worker
  refuses it. Use a browser session or a forwarded root identity.

Export builds a gzip bundle: the config files + workspace are staged in the
container (dev-agent workspaces skipped), `openclaw.json` is sanitized (dev-only
agent entries removed, ambient-owner pointers repaired), then archived and
streamed into R2 with `sandbox.readFile(…, { encoding: 'none' })`. That call
returns the raw stream **and** its authoritative byte size from the same RPC, so
the `FixedLengthStream` used for the R2 upload can never disagree with the bytes
it writes (measuring the file separately with `wc -c` did, and workerd rejected
the upload with "Attempt to write too many bytes through a FixedLengthStream").
The production side reads the object, writes it with `sandbox.writeFile`,
extracts over `/home/openclaw` (state untouched — it is not in the archive),
re-sanitizes, runs `moltlazy patch --only gateway` (restored-boot self-heal), and
soft-reloads.
`src/services/promotion/index.ts` keeps its own copy of the exclusion list so the
Worker bundle never depends on the moltlazy config module; a unit test pins both
lists together.

E2E: the former `test/e2e-push/` suite drove the HTTP push and is retired with the
transport (two local Workers cannot share a miniflare R2 bucket). The R2 pull is
covered by `src/services/promotion/r2-source.test.ts` (stage → pending → apply →
applied pointer, plus the park/busy/idempotency skips) and the route tests in
`src/routes/api/releases.test.ts`.

#### Releases admin page (`/_admin/releases`)

`ReleasesPage` (`src/client/pages/ReleasesPage.tsx`, ADMIN-only, in the
Governance group) is the dashboard face of the routes above. It reads
`GET /api/admin/releases/status` and `GET /api/admin/releases/history` in
parallel and renders the steps that apply to the instance it is running on, so
the operator no longer needs `curl`:

- **Dev instance** (`canPromote`): a **Commit message** field and a **Stage
  release for production** button (`POST /promote`). The message and the
  verified operator identity are stored on the hashed push, and the page notes
  that production applies the result on its next cold boot or on demand.
- **Production instance** (`canApply`): an **Apply release now** button
  (`POST /apply`), the release waiting in the sibling bucket (hash, message,
  author), and — when `sourceBinding` is false — an explicit warning that the
  instance cannot see the staged release and must be upgraded/re-provisioned.
- Both: the **Push history** (newest first, with short hash, commit message,
  author, status and dates), the current release pointer (status badge, source,
  dates, bundle size, promoted agents and the excluded dev-only agents), what is
  and is not promoted, and the post-apply verification checklist. A skipped
  apply is explained from the route's `reason` (`parked`, `busy:<phase>`,
  `already-applied`, `nothing-pending`, `bundle-missing`, …).

The page never gates on the gateway: `/releases/*` are Worker-local routes
(absent from `GATEWAY_TOUCHING_PREFIXES`) and stay usable while the container is
asleep. Results — including failures — also surface in the notification bell.
UI coverage: `src/client/pages/__tests__/ReleasesPage.test.tsx` and
`test/ui/releases.spec.ts` (mocked API, no Worker).

### Auth Flow

1. Cloudflare Access intercepts request → user authenticates (email OTP, Google, GitHub)
2. Access injects `Cf-Access-Jwt-Assertion` header
3. Worker verifies JWT against Cloudflare Access JWKS endpoint
4. `resolveCustomerRole()` resolves the RBAC role — Paso4 ROOT users (platform Access AUD) are always `ROOT`; client users get their lerma-assigned role (`ADMIN`/`BASE_USER`) forwarded by the Dispatch Worker via `X-FourIA-Role`
5. `userRole` and `customerSlug` are set in Hono context
6. Route handlers check role before serving content

### Dev-instance role preview (ROOT → ADMIN)

On a **dev instance** — a `-dev` hostname, or an explicit `OPENCLAW_DEV_MODE`
override (see `isViewAsEnabled` in `src/dev-mode.ts`) — a Paso4 ROOT user may
**view and perform as** a lower role from the dashboard's "View as" switcher:
the `X-Mock-Role` cookie (`POST /api/admin/dev/set-mock`) downgrades a resolved
ROOT role to `ADMIN` (or `BASE_USER`) for that session. This lets support/ops
inspect and drive a tenant's dev dashboard as the client's ADMIN without the
production diagnostics-only restriction.

The preview is a **strict downgrade and dev-only**:

- `resolveCustomerRole()` only honours the cookie when the role it already
  resolved is `ROOT` (`src/auth/middleware.ts`). A non-ROOT user setting the
  cookie cannot escalate — e.g. `ADMIN` → `ROOT` is ignored.
- `POST /api/admin/dev/set-mock` returns `404` outside `DEV_MODE` /
  `E2E_TEST_MODE` / dev instances, and `403` for non-ROOT callers on a dev
  instance.
- The tenant slug is **pinned** to the instance's own tenant; `X-Mock-Customer`
  is ignored in preview mode, so the preview can never cross tenants.
- Production instances are unaffected — `canViewAs` stays false and ROOT keeps
  its diagnostics-only scope.

`GET /api/admin/whoami` reports `canViewAs: true` for a ROOT user on a dev
instance so the SPA renders the switcher (`DevUserSelectorCompact`); local
`DEV_MODE` keeps the existing full mock-role switcher (`devMode`).

### Billing Freeze Gate (F4)

Durable per-customer `billing_frozen` flag (`src/billing-freeze.ts`). Frozen blocks
**new** billable actions only — in-flight actions complete, basic non-billable service
continues. This is **distinct from tenant suspension** (Dispatch/Lerma-level), which
denies everything.

- Durability: persisted per-customer as `billing-freeze/<slug>.json` in `BACKUP_BUCKET`
  (R2 — survives cold starts; never in-memory only). No CustomerRegistry DO exists.
- Lerma → Fouria: Dispatch Worker propagates `billing_frozen` inside `X-FourIA-Metadata`
  (shared Postgres is the source of truth; Fouria never calls Lerma directly).
  Effective freeze = stored flag **OR** header flag (fail closed).
- Fouria → Lerma: `GET /api/admin/billing/freeze-status` (ROOT-readable diagnostics path)
  plus the R2 object itself.
- Writes: `POST /api/admin/billing/freeze` | `/unfreeze` (ADMIN-only; ROOT is blocked by
  the diagnostics-only gate).
- freezeGate(env, customer) is implemented in src/billable-actions.ts; it reads the durable per-customer R2 freeze record (no request headers available, fails open on read error); enforcement happens at emit time inside emitBillableActions, which filters billing-frozen customers' events before writing Analytics Engine and enqueueing to BILLING_QUEUE, so in-flight work completes and basic service continues.

---

## Development Tips

### Debugging the Container

```bash
# Get container ID from Worker logs
docker ps | grep sandbox

# Enter container shell
docker exec -it <container_id> bash

# Inside container: check openclaw status
openclaw status
openclaw devices list --json
openclaw config get gateway.port

# Check logs
tail -f /tmp/openclaw-logs/*.log
```

### Understanding Boot Logs

During cold start, these log patterns are normal:

```
[Gateway] listProcesses returned 0 processes
[Gateway] Starting new OpenClaw gateway...
[ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP)
```

The gateway takes **up to 3 minutes** to become ready. The Worker retries automatically. If errors persist beyond 3 minutes:

1. Check `docker logs <container_id>` for OpenClaw errors
2. Verify CF AI Gateway config (`CLOUDFLARE_AI_GATEWAY_API_KEY` + `CLOUDFLARE_API_TOKEN` + `CF_AI_GATEWAY_ACCOUNT_ID` + `CF_AI_GATEWAY_GATEWAY_ID`)
3. Run `bun run dev:worker` to see full Worker output

### Service Integrations (Cloudflare Tunnel)

Cloudflare Tunnel support for tenant containers was removed. The container does
not start `cloudflared`, and no `CLOUDFLARE_TUNNEL_TOKEN` binding is provisioned.
The `obsidian-sync` skill still runs its sync server on port 18790 locally.

### CRON and Scheduled Tasks

The Worker runs one `scheduled` handler against two crons (see
`wrangler.jsonc` → `triggers.crons`). The handler branches on `controller.cron`
(`CRON_BILLABLE_DRAIN` / `CRON_OPENCODE_CREDITS` in `src/config.ts`):

| Cron            | Job                                                                                          |
| --------------- | -------------------------------------------------------------------------------------------- |
| `*/2 * * * *`   | Drain buffered billable OTel traces to `BILLING_QUEUE` (retry safety net for the sync emit). |
| `0 */8 * * 1-5` | Enqueue the OpenCode credits check onto `TASKS` (the queue consumer runs it in the sandbox). |

The credits enqueue is deliberately **not** on the frequent tick — otherwise the
OpenCode credits notification would fire every couple of minutes. Staging and
production keep only the weekday 8-hourly cron; the frequent drain is a
local/E2E aid because the synchronous `/api/otel/v1/traces` emit already covers
the normal path.

Containers sleep per a **derived** idle timeout (staging 15m, dev 2h — the Sandbox SDK 1.0
`setInactivityTimeout()` platform cap). **The billable production environment does not
auto-sleep**; it is stopped only by an explicit lerma freeze (`POST /control` →
`sandbox.destroy()`), so a warmed gateway never incurs a cold-start/downtime window after
an idle period. There is no `SANDBOX_SLEEP_AFTER` binding; `resolveSleepAfter()` derives
the value from the instance's own `WORKER_URL` (`src/dev-mode.ts`), the same source the
dev/billable axes use. Waking is driven by the per-tenant `ActivityScheduler` Durable Object
alarm, which fires ahead of the next automation and then runs an interval-gated
R2 checkpoint.

The container's idle-sleep decision is owned by the Worker's Sandbox Durable
Object class, **`TenantSandbox`** (`src/sandbox-do.ts`), exported under the DO
binding name `Sandbox` (`wrangler.jsonc` → `class_name: "Sandbox"`). It extends
the SDK's legacy `Sandbox` and overrides `onActivityExpired()`: on a **non-billable**
instance the hook ignores the **persistent** gateway (its lock wrapper, the gateway
binary, and the `tail -f` stdout mirror) and stops the container once only that
persistent work remains — matching Sandbox SDK 1.0, where code inside the instance
does not keep it alive. Any _other_ running process (an in-flight CLI/backup
command) still counts as activity and defers to the SDK gate. On the **billable
production** environment the hook defers to the SDK's `keepAlive` gate instead,
so the container never stops itself.

When a non-billable instance is due to idle-sleep, the hook first proves that an
existing archive already covers the current filesystem: either a committed
snapshot **inside the idle window** (the container has had no activity for the
whole window), or — with change detection on, the default — an `unchanged` result
from the same change scan the scheduled path uses (a quiet tenant takes no
scheduled snapshot, so its newest archive can age past the window). Either way the
hook stops inline through `stopIfIdle()` without re-archiving. That shortcut
exists because the previous unconditional snapshot was itself renewing the DO
activity window, so the stop-time re-check refused and the next idle expiry
snapshotted again — a loop of 422 MB archives that kept `tenant-tarjetero-dev`
awake. Otherwise (state changed, no archive, no `BACKUP_BUCKET`, or a probe
failure) it enqueues a durable **deprovision** job (mode `sleep`) on the
`ActivityScheduler` DO, which snapshots the tenant and only then calls
`sandbox.stopIfIdle()` — fail closed, so a container is never slept without a
committed archive. Without the `ACTIVITY` binding (local/dev) it falls back to
stopping inline. See **Backup before deprovisioning** below.

Lerma's `HealthMonitor` probes the **passive** `GET /api/health?passive=1` every
few minutes (authenticated at the Cloudflare edge with a service token — see the
path-scoped public-health Access applications in `apps/fouria/iac/access.tf`).
Passive probes never run the exec RPCs that would cold-start a stopped
container: `getHealthSnapshot` first does the non-waking `isRuntimeActive()`
read and reports `status: "sleeping"` (HTTP 200, healthy-idle) for an idle
container. Health probes are observability-only and never schedule a backup.

The container runs scheduled tasks via OpenClaw's cron system.

### Task Buffer (Cloudflare Queues)

The Worker is a durable buffer for OpenClaw-side jobs (`src/tasks/`): it never
implements the job logic itself.

- **Producer** — `scheduled` enqueues the recurring OpenCode credits check onto
  `TASKS`; `POST /api/admin/tasks/opencode-credits-check/enqueue` is the manual
  trigger (ADMIN/ROOT).
- **Consumer** — `default.queue` wakes/heals the sandbox via `ensureGateway`,
  runs `openclaw automations run opencode-credits-check` inside it, then acks
  (Cloudflare Queues own retries/backoff; `max_retries: 5`).
- `POST /api/admin/tasks/opencode-credits-check` is the manual consumer trigger
  (runs the automation directly, bypassing the queue). It clamps its total
  wall-clock to `TASK_TOTAL_TIMEOUT_MS` (90s) — the gateway wake, exec launch,
  CLI wait, and log recovery are each clamped to the remaining budget — so a
  cold/contended boot always answers within the endpoint's ceiling instead of
  outliving the caller (the e2e suite's `curl --max-time` hit 120s exactly on
  cold runs). The shared gateway start promise keeps booting in the background.
- **Automation definition** — `start-openclaw.sh`'s `ensure_opencode_credits_automation`
  (dev-only) **creates** the recurring `opencode-credits-check` cron job; the
  Worker only **runs** it. The guard queries `openclaw cron list --all --json`
  in addition to the legacy `automations list --json`, which missed a prior
  boot's job and produced two duplicate `opencode-credits-check` jobs on a dev
  instance.
- **Heartbeat cadence** — `start-openclaw.sh`'s `ensure_dev_heartbeat_cadence`
  (dev-only) reconciles the `heartbeat-fouria-builder` job on every post-start.
  OpenClaw 2026.10.5 retired `agents.entries.*.heartbeat`: a heartbeat is now an
  ordinary automation job, and config reloads/restarts never rewrite an existing
  job, so the instance created by the migration kept the **default 30-minute**
  cadence even though moltlazy declares `heartbeat.every: 12h` (observed on
  `tenant-tarjetero-dev`: 30-minute heartbeats for 9 days, ~317k tokens each on
  the main session, until the OpenCode subscription hit 0%). The reconciler
  reads `openclaw automations get <id> --json`, compares
  `schedule.everyMs`/`sessionTarget`/`payload.lightContext`, and only then runs
  `automations edit --every 12h --session isolated --light-context` (split so a
  rejected `--light-context` cannot lose the cadence fix). It never resets the
  schedule anchor when the job already matches.

### Telegram Webhook Ingress (#364)

Managed tenants switch Telegram to **webhook mode** so an inbound update wakes a
sleeping container (long polling runs _inside_ OpenClaw and cannot wake it). The
Worker is the always-on ingress (`src/telegram-ingress/`):

- **Route** — `POST /telegram-webhook/:account` is mounted **before** Cloudflare
  Access (secret-auth, not JWT). It resolves the account's config from a
  Worker-readable F4E1 registry, constant-time validates
  `X-Telegram-Bot-Api-Secret-Token`, and rate-limits — all _before_ any container
  wake, so a rejected request never starts or renews the container.
- **Registry** — `telegram-ingress/<slug>/<account>` in `BACKUP_BUCKET`
  (F4E1-encrypted, `BACKUP_ENCRYPTION_KEY`). Holds the per-account webhook secret
  - bot token, readable by the Worker _without_ the container running. Populated
    by `POST /api/admin/telegram-ingress/:account` (multi-bot).
- **Hot path** — when the gateway is running and ready, the update is forwarded
  straight to OpenClaw's webhook route (port 18789) and its
  `x-openclaw-delivery-accepted` response is passed back to Telegram.
- **Cold path ("instance is not in hot mode")** — the Worker notifies the sender
  via `sendMessage` ("waking up…"), enqueues the update on the
  `TELEGRAM_INGRESS_QUEUE` (Cloudflare Queue), acks Telegram (the message is now
  durable), and kicks the cold start. The queue consumer wakes/heals the gateway,
  waits for readiness, and forwards — acking **only after OpenClaw confirms
  receipt**, so a message is never "popped" until it has arrived (at-least-once,
  mirroring Telegram's own delivery). `max_retries: 10, retry_delay: 60s` covers a
  multi-minute cold boot.
- **Freeze policy** — enforced upstream by the Dispatch Worker: a
  frozen/operator-stopped instance is never woken by a Telegram update (no user
  role → always non-ROOT); it gets an explicit `503` and the message stays in
  Telegram's retry queue.
- **Dispatch ingress** — a dedicated host (`hooks.fouria.io` /
  `hooks.dev.fouria.io`, `TELEGRAM_INGRESS_HOST`) excluded from the wildcard
  `*.fouria.io` Access app. It resolves `/v1/<slug>/<account>`, applies the freeze
  gate, and forwards (rewritten to `POST /telegram-webhook/:account`) with no
  Access JWT.
- **moltlazy** — `start-openclaw.sh` passes the Worker-injected
  `TELEGRAM_WEBHOOK_MANIFEST` to moltlazy, which writes
  `channels.telegram.accounts.<account>.webhookUrl/webhookSecret/webhookPath/legacyWebhook`
  (fill-only, preserving the user's `botToken`/`dmPolicy`).
- **Queue observability** — the per-tenant `TelegramIngress` DO records every
  ingress event (bounded counters + the last 20): `queued` (a cold update was
  enqueued and woke the container), `forwarded` (hot path), `delivered`
  (queue consumer confirmed receipt), `failed` (a delivery attempt failed and
  will retry). `GET /api/admin/telegram-ingress/status` returns the snapshot, and
  `GET /api/admin/telegram-ingress/ws` streams it (a frame per change) for the
  dashboard. The admin shell's `TelegramIngressProvider` opens that socket only
  while the feature is in use, raises a **notification-bell** entry ("Telegram
  message queued — waking your instance", or a delivery-failure alert), and feeds
  the **Telegram Queue** widget on the home page (state + queued/delivered/failed
  - last event), which is itself hidden until the ingress is in use.
- **Coalesced status writes** — the status object is not rewritten per update
  (durable-agents audit finding F8). `telegram-ingress/coalesce.ts` batches events
  in memory and flushes them on **one** alarm, `TELEGRAM_INGRESS_STATUS_FLUSH_MS`
  (1 s) after the first event of a burst; `TELEGRAM_INGRESS_STATUS_MAX_PENDING_EVENTS`
  (25) forces an earlier flush so memory and staleness stay bounded. `getStatus`
  — and therefore the status route and its WebSocket — merges the unflushed
  batch, so an event is visible as soon as it is recorded. A failed flush puts
  the batch back and the alarm retry persists it; only an eviction inside the
  window can lose counters, and those are UI state, not billing records.

### Backup and Hard-Restart Policy

Backups are snapshots of `/home/openclaw`, not a live filesystem journal. Agent
workspaces are therefore kept under `/home/openclaw/workspace`; files outside
that tree are not protected by the Sandbox snapshot.

The Sandbox SDK's `createBackup` returns a handle marked with `localBucket: true`
when local-bucket mode is enabled (`persistence.ts`). That mode is an explicit
`BACKUP_LOCAL_BUCKET=true` opt-in (falling back to `DEV_MODE` only when unset, for
local development) — it must **not** be keyed on `DEV_MODE` alone, because the
cloud E2E deploy sets `DEV_MODE=true` merely to bypass Access and its backups are
real remote-R2 snapshots (see `ENV-VARIABLES.md`). The flag is persisted in
`backup-handle.json` and forwarded to `restoreBackup()`, so a local backup is
always restored through the local binding — never through the remote
presigned-URL path, which 404s for archives that exist only in the local bucket.
Legacy catalog entries without the flag default to the current local-bucket mode,
and a presigned-URL 404 during restore is treated as a gone backup and pruned from
the catalog.

Health probes are **observability-only**: `GET /api/health` must never write the
R2 backup catalog — no `scheduleBackupIfNeeded`, passive or active. A snapshot is
only meaningful for a **live** instance (a sleeping container has no workload, so
its filesystem has no changes to store), and waking one purely to snapshot would
restart billing. Snapshots therefore run only against a running runtime:

- the **Dispatch Worker cron** sends a `backup` control action; the tenant
  asserts `sandbox.isRuntimeActive()` (a non-waking bridge read) and returns
  `skipped: 'not-running'` for a sleeping container instead of waking it. A
  running instance is acknowledged immediately (`202`) and the snapshot is
  enqueued as a **durable job** on the per-tenant `ActivityScheduler` DO — a
  `waitUntil` task is cancelled ~30s after the response (platform cap) and was
  observed dropping the snapshot mid-cold-boot, so the cron never blocks on it
  and the alarm runs it to completion;
- `POST /debug/restart-openclaw` quiesces the gateway and checkpoints inside a
  `restart` maintenance window (see the hard-restart paragraph below);
- `POST /api/admin/storage/sync` is the explicit admin checkpoint. `?async=1`
  acknowledges with `202` and snapshots in `waitUntil`; the default (no flag)
  blocks and returns the result, which the E2E checkpoint relies on;
- `POST /debug/backup/sync` and `POST /debug/backup/async` (ROOT) trigger the
  blocking and non-blocking checkpoint modes on demand;
- `POST /api/admin/storage/reconcile` (ADMIN/ROOT) runs the catalog heal
  (`reconcileBackupCatalog`) on demand: adopt orphaned `backups/*/meta.json`
  archives, sweep metadata-less uploads, re-encrypt plaintext archives, and
  enforce `BACKUP_MAX_VERSIONS`. The durable `repair` step runs the same work
  before every scheduled snapshot, and `GET /api/admin/storage/backups` is a
  **pure read** that never heals the catalog.

Scheduled snapshots additionally refuse to run while a `restore-needed` marker
is present, rechecked under the shared lease, so a wake that lands mid-restore
cannot snapshot and promote a partial filesystem over the committed recovery
point. The marker is the durable, R2-backed restore latch — the scheduled gate
does **not** use a container-state probe, because the legacy `exists` probe can
fail in a deployed container and failing the gate closed would silently disable
every scheduled backup.

Soft reload (`POST /api/admin/gateway/restart`) does not snapshot. When no live
gateway is found it starts the gateway through `ensureGateway` (never arming
`restore-needed`, so live changes survive) and returns `409` during a maintenance window.

#### Restore trigger policy (when may a backup be applied?)

A restore is destructive: `sandbox.restoreBackup` (legacy) and
`DirectoryBackup.restore` (v1) replace the writable layer, so every write made
after the archive was taken is lost. A **gateway reboot is not a restore
trigger**: a gateway that exited, crashed, OOM'd, or is being soft-reloaded keeps
its container filesystem, and the boot must launch on that live state. The
policy is enforced in `restoreIfNeeded` / `probeContainerState`
(`src/persistence.ts`) for the legacy lane and
`SandboxV1.restoreTenantStateIfNeeded()` (`src/sandbox-v1.ts`) for v1:

| Trigger                                                                    | Legacy                                                                                        | v1                                                             |
| -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| Admin `POST /api/admin/storage/restore/:id` (ADMIN/ROOT)                   | `restoreBackupById` — stops the gateway, opens a maintenance window, force-promotes the point | `SandboxV1.restoreRecord` inside the DO                        |
| Durable restore job (enqueued only by that route)                          | `restoreBackupById`                                                                           | n/a                                                            |
| `POST /debug/restart-openclaw` (legacy)                                    | quiesced checkpoint → `signalRestoreNeeded` latch                                             | kill + relaunch on the **same** state, no latch                |
| Container positively reports **no tenant state** (replaced/killed)         | `probeContainerState() === 'absent'` → apply the committed point                              | `tenantStatePresent()` false → apply the committed record      |
| Gateway down / restart                                                     | **no restore** (probe returns `present`)                                                      | **no restore** (state file present)                            |
| Probe inconclusive (transport error / malformed answer / `success: false`) | `ContainerStateUnknownError` — boot fails closed                                              | the state-probe exec throws or exits non-1 — boot fails closed |
| Request-owned boot (`restoreOwner: 'request'`)                             | deferred to the durable boot job; never runs a destructive restore under a request            | the DO owns the boot                                           |

Two consequences worth knowing when operating a tenant:

- A **legacy** container is only read as empty when every state path answers a
  definite "no". The probe checks `/root/openclaw/.openclaw/openclaw.json`,
  `/home/openclaw/.openclaw/openclaw.json`, and `/home/openclaw/.fouria-restored`.
  A `success: false` answer (the legacy SDK returns the flag alongside `exists`)
  is a failed probe, not a "no". A runtime that exposes **no** `exists()` method
  is probed through the shell over the same paths instead of being assumed empty;
  an unavailable shell or unrecognized output is `unknown`, which fails closed.
- A **v1** container is stopped by its inactivity policy (`keepAlive` for
  billable production; finite for dev/staging). Sandbox 1.0 loses an instance's
  disk on stop, so a stopped v1 container legitimately boots by restoring the
  committed `DirectoryBackup`. The data-loss bound is therefore the
  `SANDBOX_V1_SNAPSHOT_INTERVAL_MS` cadence, not the gateway process lifetime.

Adversarial coverage of the full trigger matrix (legacy + v1) lives in
`src/restore-trigger-adversarial.test.ts` and the "a v1 boot restores only an
empty container" section of `src/sandbox-v1-adversarial.test.ts`.

> **Pre-reprovision checkpoint (legacy).** When the stuck-start watchdog has
> exhausted its resets (`GATEWAY_START_STUCK_MAX_ATTEMPTS`), it no longer destroys
> the container unconditionally. It first takes a bounded
> (`GATEWAY_STUCK_RECOVERY_CHECKPOINT_TIMEOUT_MS`) snapshot — the gateway is
> already stopped by then, so the archive is quiesced — and proceeds **only if the
> committed recovery point advanced to that archive**. A failed checkpoint, a
> pending restore over the point, or a `recovery-hold` blocks the reprovision and
> logs `gateway.start.stuck.reprovision-blocked`, leaving the container and its
> state intact for an operator (`POST /debug/reprovision` checkpoints and fails
> closed the same way). Without this guard, two stuck boots on a gateway with a
> persistent startup failure silently reverted the tenant to the previous
> snapshot. v1 has no equivalent automatic destroy.

Snapshot promotion is further gated. A snapshot taken while `restore-needed` is set
over a committed recovery point is archived but does not move the pointer, and a
pre-replacement `checkpoint` is refused with `409` (`RestorePendingError`) before the
gateway is touched. A tenant with no recovery point yet may promote its first snapshot.
A restore-only restore (`setRecoveryPoint: false`) writes `recovery-hold.json`: scheduled
snapshots archive without promoting until an explicit snapshot clears the hold.

All three snapshot paths — scheduled, manual sync, and the hard-restart
checkpoint — share **one tenant-scoped R2 `backup-lock` lease**, so only one
isolate ever stages the fixed legacy stage directory or calls
`sandbox.createBackup` at a time. The lease has a 60s TTL, is renewed every 20s
while a snapshot is in flight, and is released by compare-and-swap to a
tombstone (never by deleting the object, which would race a concurrent
takeover). The two entry paths differ only in how they acquire it:

- **Scheduled** acquisition is non-blocking — `scheduleBackupIfNeeded` takes the
  lease with `tryAcquireLock` and, if another isolate holds it, skips this tick
  (returns without waiting).
- **Manual** (`POST /api/admin/storage/sync`) and the **hard-restart checkpoint**
  wait for the lease with bounded exponential backoff (up to **120s**), then
  **fail with an error** instead of silently reporting success.

The lease is held across the legacy stage + `sandbox.createBackup` consumption
and released via the CAS tombstone when the snapshot finishes. The dashboard's
`PersistenceLockIndicator` mirrors the lock in the UI from the operation
lifecycle (`PersistenceLockContext`) rather than reading the R2 lease: the
`BackupHistoryPanel` raises it when it fires an async backup/restore and drops it
when the completion check settles, so there is no polling.

**Async admin operations + non-blocking Backups page.** A manual snapshot or a
restore can wait up to the 120s lease and then transfer a large archive, so
holding the admin HTTP request open is a poor UX and a needless failure mode (a
restore that lost the lease race surfaced as a bare `500` after ~132s, with the
underlying error dropped). Both admin routes therefore accept `?async=1`:
`POST /api/admin/storage/sync?async=1` and `POST /api/admin/storage/restore/:id?async=1`
return `202 { accepted: true }` and run the work in `executionCtx.waitUntil`. The
`BackupsPage` reads the fast raw config (`GET /api/admin/config/raw`, a single
`cat`) instead of the slow structured `/api/admin/config` (which awaits
`ensureGateway` + up to two 30s shell waits) and paints every section
independently, so no single request blocks the page. `BackupHistoryPanel` fires
the async endpoints, immediately pushes a "started" notification to the bell,
then polls the durable job (`GET /api/admin/storage/jobs/:id`, `getBackupJobStatus`)
while a snapshot is in flight (falling back to the history diff when the server
returns no job id) or `/api/health` (restore) and pushes success/failure when the
operation completes — the outcome reaches the admin via `NotificationProvider`
even after navigating away. Restore failures are now `console.error`'d for
Observability.

**Crash-consistency (issue #344).** If a Worker isolate is killed by its CPU
budget between `sandbox.createBackup()` uploading the archive and the catalog
write, the archive would otherwise be orphaned (present in R2 but absent from
`backup-handle.json`, hence invisible to `listBackups`/restore). Two guards
prevent/repair this: the catalog entry is committed **immediately after the
archive is created** (after encryption, before the config-version capture and
retention prune), and the catalog is **reconciled explicitly** from
`backups/*/meta.json` by `reconcileBackupCatalog` (the durable `repair` step,
`POST /api/admin/storage/reconcile`, `restoreIfNeeded`,
`scheduleBackupIfNeeded`, and the reprovision guard) whenever an archive is
missing from the handle. `getBackupHistory` is a **pure read**: a dashboard poll
never adopts, prunes, sweeps, or writes. The scheduled `backup` control action
also runs under a dedicated CPU budget (`BACKUP_CONTROL_CPU_MS` in
`workers/fouria-dispatch`) so the orchestration can finish; a killed isolate's
lease is reclaimed via the 60s TTL.

**Retention is enforced by the reconcile, not by a catalog read.** The two are
coupled: an isolate killed after `createBackup()` leaves an orphan that
reconciliation adopts, and if later snapshots then fail before their own prune
the catalog and R2 grow past `BACKUP_MAX_VERSIONS` indefinitely (observed on
`tarjetero-dev`: 7 archives under `maxVersions: 3`, the catalog written entirely
by reconcile, with the scheduled snapshots failing at the legacy stage).
`reconcileCatalogState` (behind `reconcileBackupCatalog` and
`reconcileOrphanedBackups`) runs `cleanupOldBackups` after merging, and the
durable `repair` step runs it before the `gate` step of every scheduled snapshot,
so the cap holds however the snapshot path ends. It still performs no
write/delete when the catalog is already within the cap, and no read path can
trigger it.

**The catalog is a compare-and-swap, not a last-writer-wins file.** Every
`backup-handle.json` write goes through `mutatePersistenceStore(bucket, mutate)`
(`persistence.ts`): it reads the handle and its etag, computes the caller's
mutation, and writes with `onlyIf: { etagMatches }` (`etagDoesNotMatch: '*'` for a
create-only handle). On a conflict it re-reads and **re-applies the mutation**
against the fresh catalog — never a blind merge, because retention and restore
mutations legitimately remove entries — up to five times before raising
`PersistenceStoreConflictError`. A mutation returns `null` to skip the write.
Reconciliation, snapshot finalization (`catalog.record-outcome`), the retention
prune and the schedule-timestamp commit all run inside this CAS, so a dashboard
poll overlapping a snapshot can no longer drop the newly committed archive (or
resurrect a pruned one).

**R2 enumeration follows `truncated`/`cursor`.** `listAllObjects(bucket, prefix)`
backs the backup reconcile sweep, the orphan archive sweep, `cleanupOldConfigVersions`
and `listConfigVersions`. Previously each site read a single page (1,000 backup
objects, 500 config versions) and silently treated it as the whole result. A
truncated page without a forward cursor logs and stops rather than looping; page
sizes are otherwise unchanged.

**Restore robustness.** `restoreIfNeeded`/`restoreBackupById` persist the
`restore-needed` marker before changing the mounted filesystem or applying an
archive. This includes an ordinary cold restore where no marker existed before
startup: if the restore or legacy materialization fails after writing partial
state, the next invocation retries instead of treating a leftover config file as
proof of success. The marker is cleared after `restoreBackup` and legacy
materialization both succeed, or when candidate scanning finds no restorable
archive and there is no stable recovery point (the explicit clean-start case). The
pre-restore `umount` is best-effort: `|| true`
guards the shell command, but the **exec RPC itself** can still throw (a
cold-start/platform interruption); that is caught and logged instead of aborting
the restore before the archive is applied. An empty catalog read also never
deletes a pending marker — it can be transient (a reconcile `list` failure, or
an archive not yet visible), and dropping it would silently lose a requested
retry.

`scheduleBackupIfNeeded` refuses to run while the durable `restore-needed`
marker is present (or its status cannot be read), rechecking under the shared
lease, so a scheduled wake that races a restore cannot snapshot a partial
filesystem and replace the stable recovery point. It deliberately does not gate
on a container-state probe: the legacy `exists` probe is unreliable in a
deployed container, and a false "no state" would silently disable scheduled
backups. The marker is normally absent while idle or after a successful restore;
cold restore selection uses the committed recovery point, not marker presence
alone.

When candidate scanning finds no restorable archive and there is no stable
recovery point, restore remains a permitted clean start; the temporary retry
marker is removed so onboarding and the next scheduled snapshot can proceed. If
a stable recovery point exists but cannot be restored, startup fails closed and
the marker remains.

**Fail-closed restore (no silent empty boot).** A committed `stable-backup.json`
is the tenant's recorded recovery point, so its presence means the tenant was
initialized. If `restoreIfNeeded` finds a recorded recovery point but **no
restorable archive** (an empty catalog with no stable pointer is only a
brand-new tenant; a catalog drained of its archives, or a stable pointer whose
archive is gone, is not), it now throws `TenantStateUnavailableError` instead of
reporting a clean start. Letting the container boot empty would run onboarding
against an empty filesystem and the next snapshot would overwrite the last good
recovery point — silent data loss. The cold start aborts (the gateway never
launches), `restore-needed` is preserved, and the instance stays visibly
unavailable for an operator to recover. When the catalog is empty but the stable
pointer _is_ present, the recorded entry is attempted directly first, so a
transient catalog-read failure still restores rather than falsely failing
closed.

**Restore skip is keyed on container state, not runtime liveness.** The
cross-isolate cold-boot dedupe previously skipped a restore when
`isRuntimeActive()` returned true ("Runtime already active; skipping duplicate
restore"). But a freshly provisioned container — started by the prelaunch port
probe, or replaced by an image rollout — is already "active" while its
filesystem is **empty**, so that check let a replaced container skip its
restore and come up empty (observed on `tenant-client-dev`: the skip logged 34×
alongside legacy-stage bridge timeouts). `restoreIfNeeded` now probes the
container's actual state (`containerHasState`, the restored/onboarded
`openclaw.json` under the legacy `/root` root or the v1 `/home` root) instead:
absent state means a fresh container that must restore. It is always safe to
attempt an idempotent restore; it is never safe to infer state from a running
process. A probe failure is treated as "no state" (restore), never as "already
restored". `isRuntimeActive()` remains the correct read for the non-waking
health/sleep paths, where the question really is "is the runtime up?", not "is
the tenant state present?".

**Pre-replacement checkpoint (`X-FourIA-Control: checkpoint`).** The `/control`
route gained a **synchronous, verified** `checkpoint` action. The control plane
(Lerma) calls it immediately before an image rollout / container replacement, so
the state about to be discarded is durably captured first. Unlike `backup`
(interval-gated, `202`-accepted, best-effort) it bypasses the interval gate, runs
the full snapshot orchestration, and **fails closed** (non-2xx) unless the
committed archive's `meta.json` is present and restorable. A sleeping/destroyed
instance has no live state and is reported as `skipped: not-running` rather than
woken. The Dispatch Worker forwards `checkpoint` with the same raised CPU budget
as `backup` (`BACKUP_CONTROL_CPU_MS`). In Lerma, both upgrade paths
(`TenantProvisioner`'s `@upgrade_pipeline` `:checkpoint` step and
`Instances.do_upgrade`) invoke `Cloudflare.Containers.checkpoint/1` before the
rollout. The step is **best-effort**: a control-plane failure is logged and the
upgrade proceeds, so a checkpoint problem can never brick a valid upgrade.

**Control-plane reachability (Cloudflare Access).** The Dispatch Worker's control
endpoints (`POST /_control/:slug/{freeze,unfreeze,backup,checkpoint}`,
`GET /_invalidate/:slug`) sit behind Cloudflare Access on the platform domain
(`dev.fouria.io` staging, `dash.fouria.io` production). Without a path-scoped
Access application those headless calls are redirected to the Access login and
return HTML, so the checkpoint is skipped. `apps/fouria/iac/access.tf` declares
`public_control_apps` with a **bypass** policy (same posture as the Lerma
`/api/*` CI bypass), so the path is reachable without an Access login; the
Worker still enforces its own internal bearer token on top. The bearer is the
Worker secret `INTERNAL_DISPATCH_INVALIDATE_TOKEN` (required, no hardcoded
fallback); the Worker compares SHA-256 digests in constant time and answers
`403` when the secret is unset or the credential does not match. Lerma must send
the same value (`deploy-dashboard` passes the production value, or
`STAGING_INTERNAL_DISPATCH_INVALIDATE_TOKEN` for staging) — without it the
internal calls 403 and the checkpoint cannot run. Roll out the Dispatch Worker
before Lerma in each environment, because a Lerma deploy holding the new value
is rejected until the Worker has it (and vice versa).

**Checkpoint outcome telemetry.** A successful tenant checkpoint returns the
archive `backupId`, `quiesced` (whether `killGateway` completed), and
`maintenanceWindow` (whether the cross-isolate R2 guard was opened). Dispatch
forwards only those checkpoint fields, and Lerma logs them as
`backup_id`/`quiesced`/`maintenance_window`; `skipped` is reported separately
when the tenant is not running. Require both `quiesced: true` and
`maintenanceWindow: true` to claim the guarded quiesce path ran; a successful
snapshot by itself only proves archive creation.

**The checkpoint takes the R2 lease before quiescing.** A live `createBackup`
can race the gateway's own writes, so after acquiring the shared `backup-lock`
lease the action records a durable, **cross-isolate** maintenance window
(`src/maintenance.ts`, `maintenance/replacement.json` in `BACKUP_BUCKET`) and
then stops the gateway (`killGateway`) before snapshotting. A busy lock fails
before maintenance is opened or the gateway is stopped; it must never drain the
tenant while waiting for another snapshot. While the window is open,
`ensureGateway` refuses to start the gateway
(`GatewayMaintenanceError`) so a request served by another isolate cannot
resurrect it mid-snapshot; the window has an explicit expiry (15 min) so a
crashed controller can never wedge a tenant, and a malformed/expired record is
discarded (fail open). The `checkpoint` action always restores service and closes
the window in a `finally`, even when the snapshot fails, so a failed checkpoint
never leaves the tenant drained. Its recovery restarts the **existing live
filesystem** with `skipRestore: true`: the snapshot attempt did not replace the
container, and trying to restore while recovering from a lock timeout would
enqueue another waiter behind the same lock. The proxy maps maintenance refusal
to a retryable `503`.

`restoreIfNeeded` also coalesces concurrent calls within a Worker isolate by
R2 bucket. The startup WebSocket allows only **one in-flight**
`ensureGateway({waitForReady:true})` warm-up, retrying only after a failed attempt
settles and with bounded exponential backoff. Its recurring health snapshots are
**passive/observation-only**, and startup-progress shell reads run only when the
startup wrapper is actually visible; it does not start a fresh restore attempt
or shell RPC on every poll. Together these prevent a long-lived startup stream
from multiplying 120-second waiters behind one in-flight persistence operation
or waking an empty container ahead of the restore.

Checkpoint recovery restarts the existing container filesystem with
`ensureGateway({ bypassMaintenance: true, skipRestore: true })`. A checkpoint
does not replace or clear that filesystem; if its snapshot cannot acquire the
shared backup lock, recovery must not call `restoreIfNeeded` and enqueue another
waiter on the same lock it just timed out waiting for.

**Only the snapshot is fail-closed.** The maintenance window and the gateway stop
are _best-effort guards_: a failure to open the window or to stop the gateway
falls back to the previously-shipped **live snapshot** instead of failing the
replacement (a broken guard must never block an upgrade). The post-snapshot
`meta.json` check is retried briefly before it is treated as a real failure, so a
transient R2 visibility gap cannot fail an otherwise-successful snapshot. The
checkpoint returns `quiesced: true` only when the gateway was actually stopped.

This closes most of the planned-replacement loss window, but it is **not** yet a
full drain-to-replacement fence: the gateway is restarted after the checkpoint,
so writes between the checkpoint and the rollout are still outside the archive,
and the shared control-plane coordination record is not implemented. Durable
primary storage for **unexpected** container loss remains the gated next step in
`docs/plans/legacy-sandbox-zero-data-loss.md`.

### Rollout park (`X-FourIA-Control: checkpoint-park` / `resume`)

The plain `checkpoint` restarts the gateway before the rollout, so anything
written in between is discarded with the replaced container. The **park** variant
is the zero-loss sequence: it is the same verified snapshot, but the tenant is
left drained and write-blocked until the control plane calls `resume` after the
container has been replaced.

- **Ordering.** `checkpoint-park` acquires the R2 lease, opens the maintenance
  window with `parked: true` (`operation: "rollout"`), arms the exec lock, and
  only then stops the gateway and snapshots. The archive therefore describes a
  filesystem no writer can touch afterwards.
- **Two enforcement layers** (`src/tenant-park.ts`):
  - _Request_: `tenantParkGuard` middleware refuses mutating (`POST`/`PUT`/
    `PATCH`/`DELETE`) `/api/admin/*` and `/debug/*` requests with
    `409 { code: "TENANT_PARKED" }` while the durable park record exists. Reads
    stay available so the dashboard can still show state. This is the
    authoritative, cross-isolate check (R2 read per mutating request).
  - _Exec_: `startShell` calls `assertTenantExecAllowed` before the launch RPC, so
    background/spawned work that never passes a route (schedulers, promotion
    helpers, fire-and-forget tasks) cannot start a process either. The park's own
    path runs inside `withTenantExecBypass` (exception-safe, nestable), which is
    the only way past the lock. That lock is **bounded** (`PARK_EXEC_LOCK_TTL_MS`,
    60 s): it cannot be released cross-isolate — `resume` lands in whichever
    isolate receives it — so it closes the in-isolate race during the park and
    then lets the durable record (middleware + `ensureGateway` refusal) be the
    only gate. Background exec is additionally gated by `ensureGateway`, which
    refuses to run during any maintenance window.
- **The gateway is the other writer**, and it is gone: automations and channels
  run inside the gateway process the park stopped. `ensureGateway` already
  refuses to launch during any maintenance window, so a request in another
  isolate cannot resurrect it.
- **Live agent sessions are switched to read-only first.** Before the gateway is
  stopped, `checkpoint-park` calls `drainLiveSessions`
  (`src/gateway/session-drain.ts`): it reads `sessions.list` over the Gateway
  WebSocket (`operator.read` + `operator.write`), patches every session to
  `permissionMode: 'read-only'` (exec denied, managed mutation tools omitted),
  and waits for the turns that were already running to finish — so the next turn
  picks the mode up instead of being interrupted mid-write. The previous mode per
  session is recorded durably (`maintenance/session-modes.json`, never inside the
  container) and handed back by `resume`/the post-restore path, because the mode
  lives on the session row and is therefore part of the snapshot. A session that
  is already read-only is skipped (nothing to restore), and **new** sessions are
  untouched — they inherit the configured policy, so nothing here becomes a
  default. The drain is bounded (`SESSION_DRAIN_TIMEOUT_MS`, 120 s): a turn that
  outlives it is reported as `sessionsDrained: false` and the park proceeds, since
  the quiesce aborts it cleanly and every write it already made is inside the
  snapshot. A long agent turn can therefore never block a release. `resume`
  waits (`SESSION_MODE_RESTORE_WAIT_MS`) for the replacement gateway, applies the
  map and clears it; if the gateway is not up yet the map stays and
  `launchRestoredState` applies it once the restored state is serving (so an
  operator restore of a parked archive also puts the modes back).
- **Telegram users are told, by the ingress, not the model.** While the tenant is
  parked, `POST /telegram-webhook/:account` answers the sender directly with
  `TELEGRAM_PARKED_NOTIFY_MESSAGE` (Bot API `sendMessage`, no gateway call, no AI
  turn), records a `parked` ingress event and returns `200` **without**
  enqueuing or waking — the update could not be delivered during the park
  anyway. The queue consumer does the same for an update that was already
  queued: it answers and acks instead of retrying for the whole rollout.
- **`resume`** closes the park, disarms the lock, clears the stale in-memory
  restore barrier (`clearPersistenceCache`, as `freeze` does), and requests the
  boot. It deliberately does **not** pass `skipRestore`: the replacement container
  is empty and must rehydrate from the parked checkpoint. A `resume` with nothing
  parked is a no-op (`resumed: false`) and never wakes a sleeping tenant.
- **Failure behaviour.** An aborted park (snapshot/lock failure) un-arms the lock,
  closes the window, and restores service exactly like a failed plain checkpoint,
  so a failed rollout never leaves the tenant drained. The 15-minute window TTL
  remains the crash backstop: a controller that dies mid-park cannot wedge a
  tenant.
- **Control surface.** `POST /control` accepts `checkpoint-park` and `resume`;
  every other action (`backup`, `freeze`, `unfreeze`) is refused with `409
  TENANT_PARKED` while the tenant is parked. The Dispatch Worker allows and
  forwards both actions (`CONTROL_ACTIONS`/`isControlAction`), gives
  `checkpoint-park` the inline snapshot CPU budget, and echoes only the
  whitelisted `parked`/`resumed` fields.

Lerma drives the sequence from `@upgrade_pipeline`: `:checkpoint` calls
`Cloudflare.Containers.checkpoint(instance, park: true)` and is **fail-closed**
(a checkpoint that cannot be taken aborts the upgrade _before_ the container is
replaced; the only exemption is a sleeping instance, which has nothing live to
lose), `:dispatch_deploy` performs the script upload + image rollout, and the
`:resume` step (immediately after) releases the park. Both error paths call
`Containers.resume/1` best-effort so a failed upgrade cannot leave a tenant
parked.

What is still outstanding is the **post-rollout readiness gate** (waiting for
`/api/health` to report a running gateway and verifying the deployed image before
declaring success, plus a fleet-level cohort/abort policy) — see the rollout
section above.

The pre-snapshot **legacy-stage shell** runs the node:sqlite online backup plus a
symlink-preserving `tar` pipe of the legacy roots, with
`--exclude='*/node_modules/openclaw'`. It deliberately **preserves symlinks**
rather than dereferencing them (`cp -aL`) and **excludes the image-provided
shared OpenClaw package** (`extensions/<id>/node_modules/openclaw` → the image's
`/usr/local/lib/node_modules/openclaw`). `-L` expanded that package into every
archive (three full copies), which pushed the stage past its bound and — worse —
made the **restore** materialization a multi-minute copy (observed on
`staging-client-dev`: `cp -a … /root/openclaw/.openclaw/` running 8m45s, starving
the gateway launch, leaving `restore-needed` set, and re-running the restore
every boot). The restore bridge (`materializeLegacyRootsAfterRestore`) uses the
same `tar --exclude` pipe, so an old backup's dereferenced copy is never
materialized; a copy-then-`rm -rf` is equally slow because deleting a
squashfs-lower tree requires copying each file up first. `restoreIfNeeded` clears
the `restore-needed` marker immediately after a successful materialization
(before the analytics bookkeeping) so an isolate eviction cannot leave a stale
marker. On a large tenant the stage can exceed the
old 300s bound (observed killing the `tarjetero` stage at exactly 300s — issue
#344 follow-up). The bound is `LEGACY_STAGE_TIMEOUT_MS` (default `600000`), the
idempotent stage is retried **once** on a transient timeout/transport failure, and
a failure logs `exit`/`timedOut`/stderr plus a structured process event with the
Sandbox `processId` for cross-dataset correlation. The stage wait is
**transport-resilient** (`runShellResilient` → `waitForExitResilient`): a Sandbox
DO↔container drop mid-stage reattaches by process id and resumes within the stage
deadline, so a mid-stage DO reset no longer ends the request. The Dispatch
`backup` CPU budget (`BACKUP_CONTROL_CPU_MS`) is aligned to the same 10 minute
bound.

The scheduled `backup` control action acknowledges immediately (`202
{ ok: true, accepted: true, jobId }`) and enqueues the snapshot as a durable job
on the per-tenant `ActivityScheduler` DO, so neither the dispatch cron nor an
operator request is bounded by it. The DO alarm runs the job past the request
and retries under a bounded lease — the previous `waitUntil` approach was
cancelled ~30s after the response and could drop a snapshot that was still
cold-booting. Combined with the crash-consistent catalog commit and orphan
reconciliation (#344) the archive stays durable even if the isolate is replaced
mid-snapshot. `POST /debug/backup/async` and `POST /debug/backup/sync`
(DEBUG_ROUTES, ROOT) expose the two modes on demand.

**Config-version snapshots.** Alongside the archive, each snapshot captures
`/home/openclaw/.openclaw/openclaw.json` into R2 under `configs/<timestamp>.json`
(30-day retention) for the Backups page's Config Version History / diff. The
history is rendered with client-side pagination (10 versions per page, newest
first) so a long-lived tenant does not paint thousands of rows. The
capture normalizes `sandbox.readFile()` across SDK generations
(`readSandboxFileText` in `persistence.ts`): the legacy SDK returns
`{ content, … }`, SDK 1.0 returns a `Response`, and older helpers return a bare
string/`Blob`. Calling `String()` on the structured result collapsed it to
`"[object Object]"`, which was stored verbatim and then rejected by the diff with
`"[object Object]" is not valid JSON`. A `JSON.parse` write-time guard now
refuses non-JSON config versions, and the diff viewer (`DiffTool`) shows a clear
"corrupted" message if a pre-existing bad snapshot is opened.

`POST /api/admin/storage/test-connectivity` (ADMIN/ROOT) is a read-only
diagnostic that verifies the R2 backup bucket binding is reachable by HEADing the
backup catalog handle — a 404 (no handle yet) still counts as connected. It is
whitelisted for ROOT through the diagnostics gate (`DIAGNOSTICS_PATHS` in
`src/routes/api.ts`) so platform operators can test backup connectivity from the
Diagnostic Dashboard without explicit access to the ADMIN-gated Backups page.

Hard restart (`POST /debug/restart-openclaw`) takes the shared lease, then opens a
`restart` maintenance window and kills the gateway **before** cutting the archive, so
the checkpoint is consistent. It sets `restore-needed` before closing the window, so no
request can relaunch the gateway on pre-restart state in between. If the checkpoint
fails after the stop, the gateway is restarted in place on the untouched filesystem and no restore is armed. It refuses with `409` while a restore is in flight
or a maintenance window is open, and while a restore is pending over the recovery point.
After a successful checkpoint, the next activation restores the committed snapshot.

**Restore outcome.** `sandbox.restoreBackup` is bounded by `RESTORE_BACKUP_TIMEOUT_MS`.
Because `withTimeout` only stops waiting, a timeout is recorded in
`restore-target.json` as `outcome: 'unknown'`, not as a failure. While that record is
younger than `GATEWAY_RESTORE_PHASE_BUDGET_MS`, `getRestoreInFlightReason` blocks every
restore (`RestoreInFlightError`, `409` on the admin route). A definitive failure is
recorded as `outcome: 'failed'` and does not block. The in-lease checks (restore
entry, `restoreIfNeeded`, `restoreBackupById`) re-read this under the lease, and
`restoreIfNeeded` also re-checks the maintenance window after it waits for the lease.

**Destructive guard.** `getPersistenceBusyReason` is the single busy check for freeze
(`/control`), `POST /debug/destroy-container`, and `POST /debug/reprovision`. It is
busy while any maintenance window is open or a restore is in flight. Reprovision also
re-checks after its checkpoint. A snapshot-only lease is **not** busy: an abandoned
snapshot leaves no half-applied filesystem behind, whereas an abandoned restore does.

```mermaid
flowchart TD
  DC["Dispatch Worker cron<br/>backup control action"] --> RC{"runtime active?<br/>isRuntimeActive()"}
  RC -- no --> SKIP["skip (never wake)"]
  RC -- yes --> S["scheduleBackupIfNeeded"]
  S --> G{"credentials + interval + lock"}
  G -- no --> N["no-op"]
  G -- yes --> L{"backup-lock free?<br/>(non-blocking)"}
  L -- busy --> SKIP["scheduler skips this tick"]
  L -- free --> HOLD
  MAN["POST /api/admin/storage/sync"] --> W["acquire lease<br/>wait ≤120s w/ backoff"]
  HR["POST /debug/restart-openclaw"] --> CP["checkpoint first"]
  CP --> W
  W -- busy > 120s --> ERR["error (no snapshot)"]
  W -- acquired --> HOLD["lease held over legacy stage<br/>+ sandbox.createBackup"]
  HOLD --> A["Sandbox snapshot<br/>/home/openclaw"]
  A --> E["optional F4E1 encryption"]
  E --> P["commit backup-handle.json"]
  P --> REL["release lease (CAS tombstone)"]
  CP --> K["graceful kill + restore marker"]
  K --> R["restore committed snapshot<br/>on next activation"]
  RP["POST /debug/reprovision"] --> RCP["checkpoint first<br/>(or require existing backup)"]
  RCP --> RD["destroy instance<br/>409 unless restorable backup or force"]
  SR["POST /api/admin/gateway/restart"] --> SOFT["SIGUSR2 in-process reload<br/>no snapshot"]
```

Do not perform repeated hard restarts without verifying a successful
checkpoint. The checkpoint protects only files inside `/home/openclaw` and only
after the backup catalog has been committed.

#### Durable async backup/restore jobs + stable recovery point

`POST /api/admin/storage/sync?async=1` and
`POST /api/admin/storage/restore/:id?async=1` no longer rely on request
`waitUntil()` (Cloudflare only guarantees ~30s of background time after a
response). Instead the route commits a job record to the per-tenant
`ActivityScheduler` Durable Object (`src/activity/backup-jobs.ts`) and returns
`202 { accepted: true, jobId }`. The DO alarm performs the work and retries with
a bounded lease (`BACKUP_JOB_LEASE_MS`) so a multi-minute snapshot survives
isolate eviction and Worker redeploys. `GET /api/admin/storage/jobs/:id` reads
the persisted status. The `ActivityScheduler` owns one alarm, so the DO merges
the next automation wake and the next backup job and arms the earliest
(`scheduleCombinedAlarm`); a queued job is never dropped when the activity
schedule clears the alarm.

**Every durable job runs one bounded step per alarm invocation.** A Durable
Object alarm handler has a hard 15-minute wall limit, while a snapshot is
`stage (≤10 min) + createBackup + config capture + prune` — and the alarm also
runs the boot, maintenance and activity queues. A single-invocation job could
therefore be killed by the platform, stay `running` until its lease expired, and
restart the whole operation from zero (the orphaned-archive class of incident).
`src/activity/job-steps.ts` persists a `cursor: { step, attempts, startedAt }`
on the job record; each alarm runs at most one step, commits the cursor, and
re-arms one second later so the other queues get a fair turn.

| Job                  | Steps                                                  |
| -------------------- | ------------------------------------------------------ |
| snapshot (manual)    | `repair` → `prepare` → `archive` → `finalize`          |
| snapshot (scheduled) | `repair` → `gate` → `prepare` → `archive` → `finalize` |
| deprovision          | `repair` → `prepare` → `archive` → `finalize` → `stop` |
| restore              | `restore` → `launch`                                   |
| boot                 | `restore` → `launch`                                   |
| maintenance          | `preflight` → `execute`                                |

- `repair` brings the catalog back to a consistent state before any snapshot
  work: it **sweeps** archive objects that no catalog entry can use (the SDK
  writes `backups/<id>/meta.json` only _after_ a successful upload, so an
  upload that lost its metadata write is invisible to both the reconciler and
  the retention prune — it would sit in the bucket forever, billed and
  unrestorable) and it **encrypts** the archives that are not encrypted at rest.
- The lease covers one **step**, not the whole operation (`STEP_LEASE_MS` =
  6 min). A Durable Object alarm has a hard 15-minute wall limit, so the
  previous 14-minute lease meant a step killed at minute 3 stalled the job for
  another 11 minutes before it was retried. Alarm invocations are serialized per
  object, so a healthy step is never duplicated.
- A step recovered after its lease expired had an **unknown** outcome: the job
  says so (`detail: 'step-interrupted'`, or `'step-deadline-exceeded'` past the
  12-minute soft deadline) and emits a `job-step.interrupted` event instead of
  silently replaying. Every step emits a `job-step.completed` /
  `job-step.failed` event with its duration — the measurement the step budget is
  derived from (`max(configured, 2 x the last snapshot duration)` for the
  cadence, 12 min for the per-step deadline).
- `prepare`/`archive`/`finalize` are the phased snapshot
  (`prepareSnapshot`/`archiveSnapshot`/`finalizeSnapshot` in
  `src/persistence.ts`). The catalog entry is still committed in `archive`,
  immediately after encryption, so a crash before `finalize` leaves a restorable
  archive (and `finalize` re-adopts it if the catalog write was lost).
- Each step takes the shared R2 lease on its own (`withSnapshotLease`); the stage
  is idempotent and `archive` re-checks the `restore-needed` latch, so a
  hand-off between steps cannot capture a half-restored container. A **scheduled**
  snapshot refuses to archive behind a pending restore; a manual/deprovision
  checkpoint deliberately still archives (without promoting), which is the
  documented recovery case.
- The retry budget is **per step** (`isStepExhausted`) as well as per job, so a
  permanently broken step fails the job instead of consuming attempts that
  belong to healthy steps. A completed step is never re-run: a boot that is
  interrupted during `launch` resumes at `launch`, not at the destructive
  `restore`.
- The legacy stage runs with `maxAttempts: 1` inside a durable step (the step
  cursor owns the retry), so its worst case stays at the 10-minute
  `LEGACY_STAGE_TIMEOUT_MS` instead of 20 minutes.
- The activity alarm now **enqueues** a scheduled snapshot instead of running
  `scheduleBackupIfNeeded` inline, so a 10-minute stage no longer shares an
  invocation with the automation wake.

**Archives that are not encrypted at rest are visible and repaired.** While a
master key is configured, an archive whose `meta.json` does not carry
`encrypted: true` (a legacy plaintext archive, or one whose upload finished
while the encryption step never ran) is catalogued with `unencrypted: true` and
queued in `pendingEncryption`. It is **never dropped** (it stays restorable) and
**never promoted** to the recovery point; the durable `repair` step encrypts it
in place and clears the flag. The dashboard shows a "not encrypted" badge for
such an archive.

**Scheduled cadence is a floor, not a promise.** `BACKUP_INTERVAL_MINUTES`
(default 5) is the minimum; `resolveEffectiveIntervalMs` stretches the effective
interval to at least **2× the last observed end-to-end snapshot duration**
(recorded as `lastSnapshotDurationMs` in the catalog). Without that, a tenant
whose snapshot takes 10 minutes would start a new one the moment the previous
committed: continuous R2 lease contention, continuous container work, and a
starved automation wake. A fast tenant is unaffected.

`createSnapshot` promotes a **stable recovery point** only after the SDK
archive, optional F4E1 encryption, and the catalog commit have all succeeded
(`promoteStableBackup` → `stable-backup.json`). A cold restore prefers that
pointer and never restores a newer archive whose promotion was interrupted;
retention (`cleanupOldBackups`) always keeps the stable entry, so a
just-uploaded-but-unpromoted archive cannot evict the last committed recovery
point. The pointer is also **monotonic** (an older or replayed archive never
moves it backwards) and **fails closed**: a malformed `stable-backup.json`
aborts restore and retention rather than falling forward to an unpromoted
archive. Unit-tested in `src/persistence.test.ts` (the “stable recovery point
promotion” block). The latest validated config is also mirrored to a fixed-key
hot copy (`stable/config.json`), kept out of the version-history listing and
30-day prune.

A **targeted restore** (`POST /api/admin/storage/restore/:id`) force-promotes the
restored archive (the exception to the forward-only pointer) so the next cold
boot restores the same archive. `?setCurrent=0` (sync or `?async=1`) threads
`{ setRecoveryPoint: false }` through `restoreBackupById` (and the durable
restore job) to apply an archive **without** moving the recovery point — the
next fresh container then reverts to the unchanged pointer. A restore also opens
a cross-isolate `maintenance/replacement.json` window (operation `restore`) and
stops the gateway right after taking the shared restore lease, closing the
window in a `finally`; `restoreIfNeeded` and `ensureGateway` both refuse to run
under it, so a refetch cannot restart the gateway and remount the latest
recovery point mid-restore.

This makes the async workflow durable and the recovery point monotonic. It does
**not** turn interval snapshots into zero-RPO replication: writes acknowledged
after the last committed snapshot are outside the guarantee. Closing that gap
requires a write-through storage path and is tracked in issue #377.

### Sandbox → R2 Connectivity (`outboundByHost`)

Sandbox code can reach Worker bindings through a **virtual hostname** without
any SDK/client library or credentials inside the container. `@cloudflare/sandbox`
(via `@cloudflare/containers`) exposes the canonical per-host outbound handler:

```ts
// src/index.ts
Sandbox.outboundByHost = {
  [RELEASES_R2_HOST]: (request, env, ctx) => handleReleasesR2Request(request, env),
};
// catch-all: internet passthrough
Sandbox.outbound = async (request) => fetch(request);
```

The handler runs **in the Workers runtime** with full `env` bindings, so the
sandbox never sees R2 credentials. Unmatched hosts fall through to the catch-all
`Sandbox.outbound`.

**Release bridge (`releases.r2.internal`).** `src/sandbox-r2-bridge.ts` maps
plain HTTP from the sandbox onto the tenant's `BACKUP_BUCKET` under a fixed
`releases/` prefix (the bucket is per-tenant, so the prefix is already
tenant-isolated):

| Method   | Path     | Effect                          |
| -------- | -------- | ------------------------------- |
| `GET`    | `/`      | JSON list of release objects    |
| `GET`    | `/<key>` | Stream an object (200) or 404   |
| `HEAD`   | `/<key>` | Size/etag metadata (200) or 404 |
| `PUT`    | `/<key>` | Upload an object (201)          |
| `DELETE` | `/<key>` | Remove an object (204) or 404   |

Keys are sanitized against traversal before touching R2; uploads use a
`FixedLengthStream` (bounded memory) when `content-length` is advertised.

**Docs bridge (`docs.r2.internal`).** `src/sandbox-docs-bridge.ts` exposes the
tenant's client documentation (`customers/<slug>/docs/`) for on-demand fetch by
the Fouria Builder agent / sandbox tooling, reusing `R2ClientDocsStore`
sanitization. The tenant slug is resolved via the shared `resolveTenantSlug`
(`src/sandbox-tenant.ts`): an explicit `customer-<slug>` container id wins,
otherwise it is derived from the per-tenant Worker env (`CUSTOMER_SLUG`, else
`WORKER_NAME` / the always-provisioned `WORKER_URL`, with the `tenant-` /
`staging-` prefixes and `-dev` suffix stripped), since production provisions the
sandbox under the fixed id `openclaw`. `GET /` lists,
`GET/HEAD /<name>` reads, `PUT /<name>` writes, `DELETE /<name>` removes. This
lets the sandbox read docs without R2 credentials and without waiting for
Worker-side materialization.

**Software bridge (`software.r2.internal`).** `src/sandbox-software-bridge.ts`
exposes the tenant's declared-software manifest (`customers/<slug>/software.json`
— the same key the admin upload and the boot-time materialization use) so the
dev-only Fouria Builder can read and declare packages **without the
Access-protected admin API** (from the container, `/api/admin/*` answers `302`
to the Access login page). `GET /` returns a discovery summary (`present`,
`path`, `containerPath`, `writeEnabled`); `GET/HEAD /manifest` reads; `PUT
/manifest` validates (the same `validateSoftwareManifestInput` as the route, with
a 256 KiB cap) and replaces; `DELETE /manifest` removes. The slug comes from the
shared `resolveTenantSlug`, so a sandboxed agent can only reach its own
manifest. Writes are **dev-only** (`isDevAgentsEnabled`): a manifest entry is
arbitrary shell executed inside the container at provisioning time, so
production stays read-only through the bridge and operators declare software
from the dashboard / admin API. The dev-only `software-manifest` workspace skill
(`packages/moltlazy/workspace/skills/software-manifest.ts`) documents the bridge
and is seeded for the Fouria Builder; because workspace skills outrank Skill
Workshop skills, it shadows any Workshop copy of the same name.

**Leads bridge (`leads.r2.internal`).** `src/sandbox-leads-bridge.ts` exposes the
**global** `paso4-leads` proposal bucket (the `fouria-leads` skill) over
`LEADS_BUCKET`. The bucket folder layer **is the tenant slug**, and the bridge
resolves the caller's slug via the shared `resolveTenantSlug` in
`src/sandbox-tenant.ts` (explicit `customer-<slug>` container id, else the
per-tenant Worker env `CUSTOMER_SLUG` / `WORKER_NAME` / `WORKER_URL`). Every path
the agent sends is **relative to its own `<slug>/` root** — the handler prepends
the slug,
so a sandboxed agent cannot list, read, or write another client's proposal. The
only bucket-root object a tenant may read is the shared, proposal-agnostic
`build-pdf.sh` (`SHARED_LEADS_READ_KEYS`); writes/deletes of it resolve under the
tenant's own folder. Because the bucket is **global**, the bridge refuses the
bare fixed sandbox id (`openclaw`) as a fallback namespace when no env identity
resolves: a worker that lost its identity env must not share `openclaw/` with
another misconfigured worker. Routes: `GET /` (or `GET /?prefix=<p>`) lists **relative**
keys, `GET/HEAD /<key>` reads, `PUT /<key>` writes, `DELETE /<key>` removes. Keys
reuse the shared `sanitizeLeadsKey` / `sanitizeLeadsPrefix` rules from
`leads/operations.ts`. This is how the sandbox reaches proposal docs **without a
public route** — the legacy `wss://<worker>/api/leads/ws` proxy
(`src/routes/leads.ts`) is intercepted by Cloudflare Access on tenant hostnames
(a per-tenant Access app shadows the `*.fouria.io/<path>` apps) and is now used
only by the host-side E2E fixture (`test/e2e/o_leads_proxy.txt`, which
authenticates with the `CDP_SECRET` persisted by `server/start`).

> **Binding requirement.** `LEADS_BUCKET` must be provisioned as an
> `r2_bucket` binding (lerma `instances.ex` `instance_bindings/2`, config
> `LEADS_R2_BUCKET`, default `paso4-leads`). A plain-text var holding the bucket
> name makes `env.LEADS_BUCKET` a `string`, so `bucket.list()` throws inside the
> `outboundByHost` handler and the container sees an opaque `ECONNRESET`. All
> the R2 bridges validate the binding with `isR2Bucket` (`src/r2-binding.ts`) and
> wrap their handler body in a try/catch, returning a structured JSON `503`/`500`
> instead of resetting the socket.

**R2 S3 credentials are deprecated for the container.** `buildEnvVars` no longer
forwards `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` into the sandbox: the
bridges run in the Workers runtime with the `BACKUP_BUCKET` / `LEADS_BUCKET`
bindings, so the container never sees credentials. The S3 credential remains a
**Worker binding** for the Sandbox SDK presigned-URL backup path only.

> **SDK 1.0 (preview).** The canonical `static outboundByHost` API is unchanged
> in the 1.0 preview, so these bridges run as-is on
> `@cloudflare/sandbox@0.13.0-next`; see the Sandbox SDK 1.0 section below.

### Client Docs (dev-only admin surface)

Client documentation is managed from the Admin **Client Docs** page
(`/_admin/docs`, ADMIN-only, dev instances only) backed by
`src/routes/api/fouria-docs.ts` (dev-gated by `isDevAgentsEnabled`, so it 404s in
production). It manages two tenant-scoped namespaces **directly from the Worker**
— no sandbox/CLI round-trip:

- **Documentation** — `customers/<slug>/docs/` in `BACKUP_BUCKET` (markdown,
  text, and binary files like PDFs/images; content type from R2 metadata or the
  extension). Store: `R2ClientDocsStore` (`listDocs` / `getDoc` / `getDocBinary` /
  `putDoc` / `putDocBinary`).
- **Proposals (leads)** — `<slug>/` in the global `LEADS_BUCKET` (PDFs and
  assets), scoped to the caller's tenant.

The SPA renders markdown client-side (`marked` + `DOMPurify`, sanitized) and opens
PDFs with the browser's native viewer through the `/raw` endpoints. Raw responses
set `X-Content-Type-Options: nosniff`; HTML/SVG are forced to `attachment` with a
sandbox CSP so client-authored files can never execute in the admin origin.

**Versioning (`DocsGitStore`).** Every doc write is committed to a Worker-side,
git-like object store in `BACKUP_BUCKET`:

```
customers/<slug>/docs-git/objects/<blobSha>    content-addressed file bytes
customers/<slug>/docs-git/commits/<commitSha>  commit JSON ({ tree, parents, … })
customers/<slug>/docs-git/refs/heads/dev       mutable HEAD
customers/<slug>/docs-git/refs/promoted        promoted ref
```

The working namespace always mirrors HEAD. History is exposed in the dashboard's
commit-tree shape (`GET /commits`); `POST /rollback/:sha` restores an older commit
as a **new, non-destructive revert commit**.

**Backups (`DocsBackupStore`).** `POST /backup` snapshots the working namespace to
`customers/<slug>/docs-backups/<id>/` (files + `manifest.json`), independent of the
git store; `GET /backups` lists (retention keeps the newest 10), and
`POST /backups/:id/restore` re-applies a snapshot as a new commit.

### Versioned documentation channels (platform docs)

The platform docs site (`webs/fouria-docs`, served at `fouria.io`) is published
per **channel** so an instance reads documentation and release notes that match
the code it runs:

| Channel   | Source                  | URL                               |
| --------- | ----------------------- | --------------------------------- |
| `latest`  | newest `v*` release tag | `/latest/` (the root aliases `/`) |
| `staging` | `develop` head          | `/staging/`                       |

`buildEnvVars` forwards `FOURIA_DOCS_CHANNEL` (`latest` only for a
production-platform production instance; `staging` for the staging platform and
every `-dev` environment — `src/docs-channel.ts`), `FOURIA_DOCS_BASE_URL`
(`https://fouria.io/<channel>`), `FOURIA_DOCS_FALLBACK_URL`
(`https://fouria-docs.pages.dev/<channel>`), and `FOURIA_VERSION` (the Worker
bundle's CalVer). The `fouria-docs` skill and the dev-only Fouria Builder/Q&A
personas read those variables, then confirm the build via
`<base>/versions.json` before quoting release notes. `versions.json` records the
channel, source commit, and component versions; `release-notes/` is compiled from
each component's `CHANGELOG.md`, so a channel page only lists what is in that
channel's code.

Deployment (`.github/workflows/deploy-pages.yml`): both channels are built from
their own checkouts and assembled by `scripts/build-deploy-tree.ts` — the
release tag supplies `/` and `/latest/`, `develop` supplies `/staging/` — and the
combined tree is deployed to `fouria-docs` on a release and on every develop
docs change. The staging project previews the staging channel at its root.

### Sandbox SDK 1.0 side-by-side foundation (`SandboxV1`)

Alongside the default legacy `Sandbox` path, the Worker carries an **additive,
not-routed** Sandbox SDK 1.0 foundation so the side-by-side move can be built
without a big-bang cutover. It is a foundation only — **no tenant is on it and
it carries no durability claim.**

- **Class** — `SandboxV1` (`src/sandbox-v1.ts`), a user-owned `DurableObject`
  that drives `this.ctx.container` (the 1.0 model). Exported from `src/index.ts`
  together with `DirectoryBackupGateway` (needed for the DO to pass
  `ctx.exports.DirectoryBackupGateway` to `DirectoryBackup`).
- **Registration (class + migration only)** — `wrangler.jsonc` migration `v7`
  (`new_sqlite_classes: ["SandboxV1"]`). There is **no deploy surface**: the
  `SandboxV1` DO binding and its `scheduling_policy: "durable_object"` container
  application were removed in every environment (durable-agents audit finding
  F9), because they served no traffic while every deploy still built/pushed an
  extra image and registered an extra application. `Dockerfile.v1` and the class
  are kept; restoring a container entry plus the binding is the first step of the
  cutover (see `docs/plans/sandbox-v1-scaffold.md`).
- **Default routing unchanged** — the legacy `Sandbox` class + `getSandbox()`
  still serve every tenant. No route selects `SandboxV1`; there is no customer
  routing change and no cutover.
- **Package / image pairing** — root catalog installs
  `@cloudflare/sandbox@1.0.0` (this DO) and
  `@cloudflare/sandbox-legacy@0.13.0-next.776.1` (the still-default legacy class,
  imported via the `-legacy` alias). `Dockerfile.v1` uses `node:24-trixie-slim`
  and COPYs the `sandbox-shim` helper from the matching `cloudflare/sandbox:1.0.0`
  image (the shim tag must track the package version); the legacy base image no
  longer applies because 1.0 removes the in-image SDK server.
- **TTL (derived, capped at 6h) and keep-alive** — `resolveInactivityPolicy()`
  reuses `resolveSleepAfter()`: non-billable instances get a finite idle timeout
  (staging 15m, dev 2h) clamped to the 6h platform maximum accepted by
  `setInactivityTimeout()`. **Billable production sets the 6h maximum and renews
  it from a recurring alarm on the same DO** (`keep-alive` purpose): SDK 1.0 has
  no "never stop" timeout, so `timeoutMs: null` would mean _stop shortly after
  the DO goes inactive_, not _stay awake_. The renewal alarm keeps the DO active
  (which resets the inactivity timer), re-applies the timeout in case the DO
  incarnation lost it, and never starts a stopped container. The snapshot
  cadence and a queued conversion share the same alarm slot, so clearing one
  purpose never deletes another's alarm. There is no `SANDBOX_SLEEP_AFTER`
  binding, and the provisioned instance size is carried in
  `CONTAINER_INSTANCE_TYPE` (a `durable_object` container chooses its size in
  `ctx.container.start()`, including every wake).
- **Real `DirectoryBackup` with F4E1 at-rest wrapping** — the additive DO
  instantiates the real `DirectoryBackup` (from `@cloudflare/sandbox@1.x`)
  against `ctx.container` + `ctx.exports.DirectoryBackupGateway` with a
  tenant-scoped `sandbox-v1/<encoded-do-name>/` R2 prefix in `BACKUP_BUCKET`.
  The SDK writes one flat plaintext object per record
  (`<prefix><id>.tar.zst`); the Worker then post-encrypts it in place to
  `<prefix><id>.tar.zst.enc` (F4E1/AES-GCM) plus a readable
  `<prefix><id>.meta.json` sidecar, and deletes the plaintext canonical object.
  Restore transiently decrypts `.enc` back to the canonical key for the SDK,
  then always deletes that plaintext in a `finally` block; delete removes all
  three objects (strict for explicit user deletion, best-effort on cleanup).
  Plaintext therefore exists only in the short post-upload window and the
  restore/decrypt window. The master key stays Worker-only
  (`env.BACKUP_ENCRYPTION_KEY`, never passed to the container); the record's
  `sha256`/`size` remain plaintext and the SDK `DirectoryBackup.restore()`
  verifies them on restore. With `BACKUP_ENCRYPTION_REQUIRED=true` a missing
  key/meta fails closed before any upload or restore.
- **Still no scheduling, conversion, or cutover** — Worker-owned periodic
  snapshot scheduling + retention is **not implemented** (no approved defaults
  wired; the intended targets are a 5m snapshot interval, retaining the latest
  interval snapshots plus a 24h rolling window and daily 30d history, once a
  scheduler exists). There is **no** 0.13 `.sqsh.enc` → DirectoryBackup
  conversion, **no** customer cutover, and **no** default-routing change: the
  legacy `Sandbox` class + `getSandbox()` still serve every tenant, no route
  selects `SandboxV1`, and the complete restore-before-Gateway path is not wired
  for the v1 line. This is not a final durability/no-loss guarantee. See
  `docs/plans/sandbox-v1-scaffold.md`.

### Sandbox SDK runtime lines

The default tenant runtime is the **legacy** `Sandbox` class, installed via the
`@cloudflare/sandbox-legacy` catalog alias of `npm:@cloudflare/sandbox`
`0.13.0-next.776.1`, paired with the `docker.io/cloudflare/sandbox:0.13.0-next.776.1`
image. The Worker package and the container image **must stay on the same line** —
the stable and 1.0 control protocols are mutually incompatible.

The side-by-side `SandboxV1` Durable Object uses the stable `@cloudflare/sandbox`
`1.0.0` package (see the Sandbox SDK 1.0 section above). The intended runtime is
selected by the `SANDBOX_RUNTIME` binding (`legacy` default | `v1`); until V1
implements the full `Sandbox` surface the legacy class keeps serving and
`resolveSandboxRuntime()` reports the requested V1 as `pending` (see
`src/sandbox-runtime.ts`).

Every tenant-facing site that needs a `Sandbox` obtains it through the single
runtime acquisition boundary `acquireSandbox()` in `src/sandbox-contract.ts` —
the request middleware (`src/index.ts`), the task buffer (`src/tasks/index.ts`),
the `ActivityScheduler` Durable Object (`src/activity/scheduler.ts`), and the
per-customer gateway launcher (`src/gateway/process.ts`). The boundary resolves
the runtime but always returns the legacy stub for now (a `v1` request is still
`pending`), preserves the container identity (`openclaw` for the single-tenant
instance, `customer-<slug>` for a per-customer gateway) and the legacy idle
policy (defaults to `buildSandboxOptions(env)`), and never starts a container —
`getSandbox()` returns a Durable Object stub, so acquiring a runtime never wakes
a sleeping tenant or touches the v1 DO namespace. This is the single flip point
for the cutover.

#### Sandbox v1 migration gate (S1)

The first cutover slice (`S1`, contract C1/C2/C10) adds the **authority and
identity gate**, not the conversion (S2 owns `convertToV1` and the DO-owned
restore). The gate is `src/sandbox-migration/platform.ts`:

- `canMigrate(env)` is true **only** when the trusted `WORKER_PLATFORM` binding
  is exactly `staging`. Absent, empty, whitespace, `production`, `dev`, `local`,
  `STAGING`, and a request carrying `Host: staging-<slug>` or
  `X-FourIA-Role: root` all refuse.
- `migrationRefusal(env)` resolves the canonical tenant with
  `tenantSlugFromEnv` (never `ctx.containerId`, never `resolveTenantSlug`'s
  context fallback) and refuses `null` and the reserved `client` slug.
- The `/control` entry runs the gate in a middleware **before** the sandbox stub
  is acquired, so a refused migrate performs no R2 write and no DO/container
  addressing. It returns `403 MIGRATION_PLATFORM_FORBIDDEN`,
  `409 MIGRATION_SLUG_UNRESOLVED`, or `409 MIGRATION_RESERVED_CLIENT`. An
  allowed staging request is served by the **queue-only** entry (S5): it
  enqueues the durable conversion on the `SandboxV1` DO and answers `202`
  (queued/running), `200` (committed + `recordId`) or `502`
  (`SANDBOX_V1_CONVERSION_FAILED`); the read-only `migration-status` action
  reports `absent`/`queued`/`running`/`committed`/`failed` and can never start
  work. See § "Sandbox v1 conversion orchestration" below.
- The gate is **migrate-only**: `freeze`, `unfreeze`, `backup`, `checkpoint`,
  `checkpoint-park`, and `resume` keep their existing behavior on every
  platform and on the reserved `client` tenant.
- `sanitizeCustomerSlug` rejects the `default` sentinel in every resolver branch
  (C10), and `SANDBOX_V1_CUTOVER` defaults to `false` — unselected tenants stay
  byte-identically on the legacy runtime.

Lerma's **provisioning** and **upgrade** paths now both add the platform
bindings (`WORKER_PLATFORM` from `cf_environment`, `SANDBOX_V1_CUTOVER=false`)
and, on the staging platform only, the `SandboxV1` DO binding
(`Lerma.Cloudflare.SandboxV1`). The per-tenant v1 container application is
created by `Containers.deploy_script_and_container/1` from the same
platform/image predicate, with the v1 image supplied as a digest-pinned named
image (`ctx.container.images.sandbox`) because the `durable_object` policy takes
no application-level image. The upgrade path also re-asserts the tenant's full
binding set, so it can rebuild a script that no longer exists, and an upgrade
rollback never deletes the live tenant's script. Verified on staging
(2026-11-05); the staging Lerma was then restored to `:latest-pre`, so a
promotion from `develop` Lerma would drop the v1 bindings until the branch
lands. See
[`sandbox-v1-staging-observations.md`](/llms/platform/reference/plans/sandbox-v1-staging-observations/index.md).

#### Sandbox v1 conversion (S2)

`SandboxV1.convertToV1()` (`src/sandbox-v1.ts`) owns the data movement and the
conversion barrier; the Worker never runs the primitives in a request context.

- **Intent (C5, C6).** `src/sandbox-migration/intent.ts` writes
  `sandbox-migration/<tenant>/intent.json` **before** the first converting step.
  Reads are tri-state (`absent` / `present` / `unknown`); only an explicit
  `absent` permits legacy. `present` **or** `unknown` means v1, permanently.
- **Lease (C4).** `src/sandbox-migration/lease.ts` is a per-tenant R2 lease:
  create-only acquire (`etagDoesNotMatch: '*'`), etag-CAS takeover of an
  expired/released tombstone, owner-scoped renew/release, and fail-closed
  parsing. The TTL is a liveness bound only.
- **Conversion (C7).** `convertToV1()` acquires the lease, writes/validates the
  intent, opens the legacy recovery point (`stable-backup.json`, F4E1-decrypting
  `data.sqsh.enc` when needed), streams the plaintext squashfs into the v1
  container, extracts it with `unsquashfs -f -d /home/openclaw`, takes a v1
  `DirectoryBackup` (`tar+zstd/1`), writes
  `sandbox-migration/<tenant>/conversion.json`, and publishes
  `sandbox-migration/<tenant>/stable.json`. `Dockerfile.v1` installs
  `squashfs-tools`.
- **Barrier (C8).** `verifyConversionRecord()` completes only when the stable
  pointer names a shape-valid committed record. Readiness is a separate input
  (S3/S4); `startGateway(env)` is the DO-owned owner restart and waits for a real
  `/health`, but S2 only calls it when the caller supplies the launch env.
- **No fallback (C5).** Once the intent exists, the Worker acquires the v1
  runtime (S3 below); it never silently falls back to the legacy gateway.
  `ensureGateway` on a legacy stub still throws `SandboxV1RuntimePendingError`
  as a defense-in-depth check.

#### Sandbox v1 serving (S3 / OCM-1)

Runtime selection is the durable conversion intent, resolved once per request in
the global middleware (`resolveRuntimeIntent`). `acquireSandbox(env, id, options,
runtime)` then returns the matching runtime:

- `legacy` → the 0.12 `getSandbox()` stub (unchanged, byte-compatible).
- `v1` → `V1SandboxClient` (`src/sandbox-v1-client.ts`) wrapped around the
  `SandboxV1` DO. `unknown` → `503 SANDBOX_V1_RUNTIME_UNKNOWN` before any
  acquisition. Control actions on a converted tenant fail closed with
  `503 SANDBOX_V1_CONTROL_UNSUPPORTED` (S4 owns them) rather than acting on the
  wrong container.

**Transport.** The DO exposes a `fetch()` entrypoint: the Worker client sends the
request to the DO stub with `X-FourIA-Sandbox-Port`, and the DO rewrites it to
`http://localhost:<port>` before calling
`ctx.container.getTcpPort(port).fetch(...)`. The entrypoint (not an RPC method)
is what carries a WebSocket upgrade Response back to the Worker. It never starts
the container: a stopped container answers `503 SANDBOX_V1_SLEEPING`, so a probe
cannot wake an idle tenant.

**Process RPC.** `ExecProcess` handles cannot cross RPC, so the DO keeps them and
exposes `v1Exec` / `v1ProcessOutput` / `v1ProcessWait` / `v1ProcessSignal` /
`v1ProcessState` / `v1ListProcesses` (output is decoded to strings inside the
DO). `V1SandboxProcess` presents that as the `SandboxProcess` shape, so
`startShell` / `runShell` / `waitForProcess` and the CLI-backed admin routes keep
working.

**Lifecycle.** `ensureGateway` detects the v1 client and delegates to
`ensureV1Gateway` → DO `v1EnsureGateway`, which (1) adopts a gateway already
answering `/health`, (2) restores the committed v1 `DirectoryBackup` when the
container holds no tenant state (`OPENCLAW_RESTORED=true` for the launch), and
(3) runs `start-openclaw.sh start` as the single supervised owner, polling a real
`/health` before returning. The DO is the serialization point, so there is no
second owner even across Worker isolates. `killGateway` delegates to
`v1KillGateway`.

**Health.** `getHealthSnapshot` branches on the v1 client: `v1Status()` is the
non-waking `ctx.container.running` read (a passive probe on a stopped runtime is
`sleeping`, HTTP 200 at the route), readiness is a real `/health` response, 503
until ready, and `/healthz` stays the sub-second non-waking liveness read.

**Not yet implemented on the v1 line** (each fails with
`SANDBOX_V1_UNSUPPORTED_OPERATION` instead of an undefined-method crash):
`createBackup` / `restoreBackup` / directory backups (PER-2), file tools such as
`writeFile` / `readFile` / `mkdir` (OCM-3), and the lifecycle/eviction hooks
`stop` / `stopIfIdle` / `onActivityExpired` (OCM-2). Durable boot jobs and
scheduling still acquire the legacy stub (PER-3).

#### Sandbox v1 gateway proxy attribution

OpenClaw attributes **proxy-shaped** traffic (any `Forwarded`, `X-Forwarded-*`
or `X-Real-IP` header) against `gateway.trustedProxies` _before_ Gateway auth.
It accepts the request only when the **source address is trusted** _and_ the
forwarded client headers resolve to a non-loopback client; otherwise the
Gateway-authenticated routes answer `403`
`{"error":{"type":"proxy_attribution_required", …}}`.

The Worker always rebuilds `X-Forwarded-For` / `X-Real-IP` from
`CF-Connecting-IP` (see `rebuildForwardedClientHeaders` in `src/index.ts`), so the
second condition holds. The first one depends on the ingress path, which a tenant
cannot configure:

| Runtime  | Gateway sees the request from                                  | In `trustedProxies`?                  |
| -------- | -------------------------------------------------------------- | ------------------------------------- |
| `legacy` | the Cloudflare Sandbox container bridge (`10.1.0.x`)           | yes — moltlazy unions `10.0.0.0/8`    |
| `v1`     | the `ctx.container.getTcpPort()` ingress owned by the platform | **no** (unknown to any tenant config) |

That made **every** proxied request on a converted/born-v1 tenant fail (the
reported symptom: `proxy_attribution_required` on all dev instances deployed
with the v1 runtime), including WebSocket upgrades and the admin-RPC reads,
because `gateway.trustedProxies` can only list addresses the tenant can know.

`src/gateway/proxy-attribution.ts` owns the fix. A request the Gateway refuses
cannot be attributed as a proxy, but a **direct-remote** request is always
attributed (OpenClaw only rejects proxy-_shaped_ traffic it cannot resolve), and
it is served with the same token/device auth. `SandboxV1.fetch()` therefore:

1. probes attribution **once per gateway incarnation** with a `GET /` carrying
   the same rebuilt forwarded headers (`X-Forwarded-For: 203.0.113.1`). `/` is
   the Gateway-authenticated route that fails with the rejection, so its 403 is
   the definitive signal. A probe that cannot reach a still-booting Gateway
   leaves the mode `unknown` (traffic keeps its headers) and is retried by the
   next forward;
2. when the Gateway refuses attribution, strips every forwarded header from the
   request before forwarding it, so OpenClaw attributes it as `direct-remote`;
3. caches the verdict for the Durable Object's lifetime and **re-probes after a
   gateway restart / container destroy** (`v1KillGateway`, `v1Destroy`,
   `startGatewayOwner`), so a healed `gateway.trustedProxies` restores
   client-IP fidelity without a Worker deploy.

Requests that carry no forwarded headers (container health probes, CLI reads)
skip the probe entirely, and the probe result is single-flighted so concurrent
requests at boot cannot fan out probes. Structured event:
`sandbox.v1.proxy_attribution.direct` (warn) — the direct fallback is active for
this gateway.

Trade-off: in direct mode the Gateway attributes the container-internal source
instead of the real client IP (rate-limit subject and audit client IP). That is
acceptable for a single-tenant gateway whose credential is the shared token, and
it is strictly better than a 403. When attribution is accepted, nothing changes.

#### Provisioning an instance directly on Sandbox v1

A tenant can skip the legacy conversion entirely: Lerma provisions it **on v1**
("born v1"), the Worker serves it on `SandboxV1` from the first request, and
the tenant onboards inside the v1 container. The decision is durable in R2 and
written once, at provisioning time.

**Marker contract.** `sandbox-migration/<canonical-slug>/runtime.json` in the
tenant's `BACKUP_BUCKET`:

```json
{
  "version": 1,
  "tenant": "<canonical-slug>",
  "runtime": "v1",
  "mode": "fresh",
  "provisionedAt": "<ISO8601>"
}
```

- Lerma writes it **create-only, before the script deploy**; the Worker only
  ever reads it (`src/sandbox-migration/runtime-marker.ts`). There is no env
  var, no endpoint and no Worker-side write path for it.
- The read is tri-state like every other migration record (`readJsonObject`):
  a transport error, malformed JSON, a non-matching `tenant`, a `version` other
  than `1`, another `runtime`, a `converted` mode, a missing/empty tenant or a
  `provisionedAt` that is not ISO 8601 **with an explicit zone** (`Z` or
  `±hh:mm`) is `unknown`, never `absent`.
- `readFreshRuntime(env)` is the route-facing convenience: it resolves the
  canonical tenant with `tenantSlugFromEnv` and reads the marker; an unresolved
  identity is `unknown`, and **no `BACKUP_BUCKET`** is `absent` (a marker cannot
  exist without the bucket, so local/dev instances without R2 keep the
  historical behavior).

**Resolution matrix** (`resolveRuntimeIntent`,
`src/sandbox-migration/intent-gate.ts`):

| Platform      | Conversion intent                                | Marker                         | `SANDBOX_RUNTIME` | Runtime                                                            |
| ------------- | ------------------------------------------------ | ------------------------------ | ----------------- | ------------------------------------------------------------------ |
| not `staging` | any                                              | any                            | any               | `legacy` (unchanged; the platform is the only migration authority) |
| `staging`     | `present`/`unknown` + C8 record+pointer complete | any                            | any               | `v1`                                                               |
| `staging`     | `present`/`unknown` + barrier incomplete         | any                            | any               | `pending`                                                          |
| `staging`     | `absent`                                         | `present` (valid fresh marker) | any               | `v1`                                                               |
| `staging`     | `absent`                                         | `unknown`                      | any               | `unknown` (fail closed)                                            |
| `staging`     | `absent`                                         | `absent`                       | `v1`              | `pending` (fail closed)                                            |
| `staging`     | `absent`                                         | `absent`                       | unset/`legacy`    | `legacy`                                                           |

The conversion intent wins over a marker: a tenant that was converted (and
historically also carried a marker) stays behind the C8 record+pointer barrier.

**Fail-closed `pending`.** `SANDBOX_RUNTIME=v1` with **no marker** means the
instance asked for v1 but was never provisioned for it (the marker write and the
binding disagree). The Worker answers the existing
`503 SANDBOX_V1_CONVERSION_PENDING` and never falls back to legacy — a legacy
boot would create the exact legacy state the tenant is not supposed to have.
There is no "assume the marker" path and no automatic recovery: re-provision or
remove the binding.

**Control plane.** Neither migration action applies to a fresh tenant:

- `POST /control` `X-FourIA-Control: migrate` → `409`
  `{ ok: false, action: 'migrate', code: 'MIGRATION_ALREADY_V1', error: 'tenant
  is provisioned on SandboxV1; there is nothing to convert' }`, without
  addressing the DO or enqueuing (an unreadable marker → `503
  SANDBOX_V1_RUNTIME_UNKNOWN`, since converting a tenant that may already be v1
  would be destructive). The platform/slug gate still runs first, and a
  converted tenant has no marker, so it keeps its idempotent enqueue behavior.
- `POST /control` `X-FourIA-Control: migration-status` → `200`
  `{ ok: true, action: 'migration-status', state: 'absent', mode: 'fresh' }`
  without touching the DO (never enqueues, never wakes). A converted tenant
  answers `mode: 'converted'` alongside its durable `state`.
- `backup` / `checkpoint` / `freeze` / `unfreeze` / `checkpoint-park` / `resume`
  run the existing v1 branches (snapshot through the DO, park/resume), so a
  fresh tenant is a first-class v1 tenant for the rollout machinery.

**Boot and persistence.** `SandboxV1.restoreTenantStateIfNeeded()` returns
`false` when the container holds no config and the recovery point is
`pointer-absent` with no conversion record, so a fresh tenant **onboards**
(`start-openclaw.sh start` runs the setup body) with `OPENCLAW_RESTORED` unset.

**The first recovery point is taken right after that first successful boot.**
`v1EnsureGateway` dispatches `runInitialSnapshot()` detached (registered with
`ctx.waitUntil`, never awaited), so the tenant no longer waits for the first
idle-cadence tick to become durable:

- **When.** Only for a tenant with no recovery point: `verifyRecoveryPoint` must
  report `pointer-absent` **and** `conversion.json` must be absent. A converted
  tenant, a tenant that already snapshotted (cadence, `backup`,
  `checkpoint-park`), an unreadable pointer and any inconsistent state are left
  alone — the boot never snapshots blindly over an existing point. A parked
  tenant is skipped too (its park already committed the rollout record) and a
  container that is not running is never woken.
- **Guards.** A single-flight promise (`#initialSnapshotRun`) makes concurrent
  boots (the owner run and the adopted-gateway path both schedule it) fold into
  one attempt, a settled flag short-circuits the check once there is nothing to
  do (at most one R2 read per DO incarnation), and `takeOwnedSnapshot()` gives
  the idle cadence and this attempt one shared owner so they cannot snapshot the
  directory concurrently — the loser skips and the cadence keeps its alarm
  armed.
- **Best effort.** Any failure is caught and logged as
  `sandbox.v1.initial_snapshot_failed` (bounded message); the boot stays `ready`
  and the container keeps running. A failure is retried on a later boot: a new
  DO incarnation retries immediately, the same incarnation after
  `INITIAL_SNAPSHOT_RETRY_MS` (10 min), so a broken backup cannot be hammered on
  every request. A success logs `sandbox.v1.initial_snapshot` with
  `recordId`/`size`/`took`.

From then on the normal v1 persistence path applies: `v1CreateSnapshot` writes
the `DirectoryBackup`, catalogs it, advances
`sandbox-migration/<slug>/stable.json` (pruning by retention), and that pointer
is the recovery point everything restores (rollout park, restore by record id,
cold boot). A conversion record without a pointer still refuses to onboard over
it.

**Residual window.** The snapshot is detached, so a tenant that dies between
`/health` becoming ready and the archive committing (seconds to minutes on a
large directory, and longer on a failed + retried attempt) still has no durable
copy; the cadence remains the backstop for that sliver, and readiness is never
delayed by the backup.

**Policy.** Provisioning a fresh v1 tenant is limited to the **staging platform
(including dev instances on it)**, like every other v1 lane: the resolution
matrix returns `legacy` outside `WORKER_PLATFORM === "staging"`, so a marker
written on a production tenant is inert. The reserved `client` slug never
migrates and Lerma refuses to write a marker for it; the Worker deliberately
does not special-case `client` here — a marker is authoritative for serving.

#### Sandbox v1 idle snapshot cadence (PER-2)

A converted tenant's container belongs to `SandboxV1` (`ctx.container`), and raw
`ctx.container` has **no eviction hook**: the legacy `onActivityExpired` →
archive → stop path does not exist on SDK 1.0. The platform stops the container
after its inactivity timeout (`applyInactivityTimeout`, derived from
`resolveSleepAfter`), so without a cadence everything written since the last
committed record would be lost. The DO therefore owns a periodic snapshot:

- **Arming.** `startContainer()` arms the cadence after the display/outbound
  setup succeeds (best-effort: a storage error is logged as
  `sandbox.v1.snapshot.cadence_arm_failed` and never fails the boot). Arming
  writes a durable purpose marker (`snapshot:cadence-pending`) plus
  `ctx.storage.setAlarm(now + interval)`. The DO has one alarm slot and a queued
  conversion arms `now + 1s`, so arming never postpones an earlier alarm — the
  marker, not the alarm time, tells the handler what to do. A restarted DO keeps
  the alarm (it is durable) and `startContainer()` re-arms on the next boot.
- **Interval.** `resolveSnapshotIntervalMs(env, inactivityTimeoutMs)`:
  `SANDBOX_V1_SNAPSHOT_INTERVAL_MS` (ms, `0` disables) → else the configured
  `BACKUP_INTERVAL_MINUTES` → else **1h**. The result is capped at **half** the
  instance's inactivity timeout when one is applied, because a snapshot due after
  the container has already idle-stopped is pointless (and the cadence must never
  wake it): staging (15m) → 7.5m, dev (2h) → 1h. The legacy 5-minute _default_ is
  deliberately not inherited — it was tuned for the interval-gated cron path, and
  a 5-minute v1 cadence would snapshot back-to-back.
- **The alarm turn** (`alarm()` → `runConversionAlarm()` then
  `runSnapshotCadence()`; neither throws out of the handler):
  1. disabled → clear the marker and stop;
  2. `#restoring` (a manual restore owns the directory) → skip this turn, keep
     the cadence armed, so one overlap cannot silently switch the loss bound off;
  3. container stopped → **never wake it**: clear the marker (the next
     `startContainer()` re-arms);
  4. the recovery point (`sandbox-migration/<tenant>/stable.json`) is newer than
     the interval → skip (this dedupes with the `ActivityScheduler`'s own
     interval-gated v1 snapshot);
  5. otherwise `v1CreateSnapshot({ trigger: 'backup' })`, logging
     `sandbox.v1.snapshot.cadence` with `recordId`/`took`;
  6. re-arm for the next interval.
- **Teardown.** `v1Destroy()` clears the marker and the alarm — unless a
  conversion is queued on the same slot (deleting that alarm would strand it).
- **Parking.** The cadence is deliberately park-agnostic: a parked tenant is
  drained, so a fresh snapshot is unnecessary but harmless (the catalog write is
  idempotent and touches R2 only).

Failures log `sandbox.v1.snapshot.cadence_failed` and re-arm; the previous
recovery point stays intact and the container keeps running.

#### Sandbox v1 restore by record id (PER-2)

`POST /api/admin/storage/restore/:id?setCurrent=0` on a converted tenant applies
a **catalogued v1 record** instead of the legacy `.sqsh` archive:

- the route opens the durable maintenance window (`restore`) **before** the DO
  touches the container, so a request arriving mid-restore cannot restart the
  gateway under the swap (`ensureGateway`'s v1 branch honors the window);
- `SandboxV1.v1RestoreRecord` stops the single gateway owner, restores the
  record's directory into `HOME_DIR` (the SDK swaps it atomically), and — by
  default — advances the stable pointer so a cold boot restores what the operator
  chose (`?setCurrent=0` applies the record **without** promoting it);
- an unknown or unreadable record fails closed (`404`) and never touches the
  container; a restore already in progress is refused by an in-DO flag;
- the Worker then boots the restored state (`OPENCLAW_RESTORED` semantics) and
  reports `gatewayReady`. A boot failure does not undo the restore: the next
  request retries the boot, and the caller must not retry the destructive step.

v1 restores are synchronous: there is no legacy backup-job queue for the record
format, and the SDK restore is bounded by the archive size.

#### Sandbox v1 outbound virtual hosts (OCM-1 / BIL-1)

The container reaches the Worker-owned services through virtual hosts instead of
the public internet: `releases.r2.internal`, `docs.r2.internal`,
`leads.r2.internal`, `software.r2.internal` (R2 namespaces resolved with the
Worker's own binding), `otel.internal` (OTLP ingest from the tenant's exporter),
`cdp.internal` (the container's local CDP proxy forwarding Browser Rendering
commands to the tenant's Sandbox DO through `src/sandbox-cdp-bridge.ts` →
`src/cdp-session.ts` — the browser credential never enters the container), and
`ai.internal` (the inference proxy in `src/sandbox-ai-bridge.ts`). The map lives
in `src/sandbox-outbound.ts`
(`SANDBOX_OUTBOUND_ROUTES`), the single source of truth for both runtimes:

- the **legacy** `Sandbox` class installs it as its `outboundByHost` statics;
- **`SandboxV1`** registers each host with
  `ctx.container.interceptOutboundHttp(host, ctx.exports.SandboxOutboundGateway({
  props: { host } }))` in `startContainer()`. SDK 1.0 has no class statics — an
  intercept belongs to the container instance and disappears when it is
  replaced — so the routes are installed on **every** start, including a
  restarted DO that finds its container still running. A Worker image without the
  export logs a warning and leaves the container on direct egress instead of
  failing the boot.

Without this, a converted tenant silently loses OTLP export (AI cost
attribution), the R2 bridges and the agent browser (`cdp.internal`); the
container has no other way to resolve those names.

#### AI Gateway inference proxy (`ai.internal`)

AI Gateway capacity is account-scoped (20 gateways on the paid plan), so tenants
share **pools** rather than owning a gateway. A gateway total is therefore never
a client's spend, and the account-scoped inference credential must not sit in a
tenant container.

The route is **opt-in** (`AI_GATEWAY_PROXY=true`). It only takes effect when the
tenant's primary model resolves under `cloudflare-unified-billing/*`; the
official `cloudflare-ai-gateway` provider ignores
`CF_AI_GATEWAY_CUSTOM_DOMAIN` and would send the derived token to Cloudflare,
which rejects it. Confirm the active model ref (Agent Studio's model pane, or
`openclaw models status`) before enabling it on a tenant. When enabled, tenant
inference is routed through the Worker:

1. `buildEnvVars` sets `CF_AI_GATEWAY_CUSTOM_DOMAIN=ai.internal` and hands the
   container `CLOUDFLARE_AI_GATEWAY_API_KEY` / `CLOUDFLARE_API_TOKEN` set to
   `HMAC-SHA256(MOLTBOT_GATEWAY_TOKEN, "fouria-ai-proxy/v1")`
   (`src/gateway/ai-proxy-token.ts`). The unified-billing plugin then calls
   `https://ai.internal/compat/...`.
2. The sandbox intercepts that host and runs
   `handleAiRequest` (`src/sandbox-ai-bridge.ts`) in the Worker.
3. The handler verifies the derived token, resolves the tenant from the Worker
   env (`resolveTenantSlug`, never the request), maps `/compat/*` →
   `/ai/v1/*`, and forwards to
   `https://api.cloudflare.com/client/v4/accounts/{account}/ai/v1/...` with the
   real credential.
4. It always sets `cf-aig-metadata` to the trusted `tenant_id`, `environment`
   and `platform` (plus the plugin's `openclaw_run_id`), and forces
   `cf-aig-gateway-id` from the trusted binding. A tenant cannot override either
   by editing its config.

Consequences:

- **Attribution.** Lerma's development-expense report sums only log entries
  whose `metadata.tenant_id` matches the client and whose `environment` is one
  of that client's non-billable environments; legacy entries without metadata
  are counted as `ai_gateway_unattributed_requests` instead of being assigned to
  a client. Cloudflare's Analytics API can group the same metadata with
  `metadataValue(key: "tenant_id")`, and spend-limit rules can split budgets by
  it.
- **No Access credential pair.** `CF_ACCESS_TOKEN`,
  `CF_ACCESS_CLIENT_ID` and `CF_ACCESS_CLIENT_SECRET` are no longer forwarded to
  containers. An Access-protected custom domain needs a per-tenant service token
  and service tokens do not produce `cf.user_id`, so the pair could not deliver
  the identity attribution it was meant for.
- **Rollout to deployed instances.** Lerma persists the opt-in on
  `instances.ai_gateway_proxy` and re-deploys an already-running instance's
  current bundle from the instance page (**AI proxy** toolkit action →
  `Instances.set_ai_gateway_proxy/2`). Only the binding changes; the image,
  gateway and container application are untouched, and the instance status is
  preserved. Upgrades carry the flag forward.
- **Custom-domain opt-out.** Setting `CF_AI_GATEWAY_CUSTOM_DOMAIN` explicitly
  keeps the direct path with the real credential; that domain must not be
  Access-protected.
- **Tests.** `src/sandbox-ai-bridge.test.ts` pins the identity, metadata,
  gateway-forcing and path-mapping contract; `src/gateway/env.test.ts` pins the
  derived-token wiring and the removal of the Access credentials.

#### Sandbox v1 rollout park (SEC-3)

A v1 tenant can now be parked for a container replacement, with the same
guarantees as the legacy lane and one deliberate difference in ordering:

- **`POST /control { X-FourIA-Control: checkpoint-park }`** on a converted tenant:
  1. switches live sessions to `read-only` and waits for their turns to end
     (`drainLiveSessions` over the v1 gateway WS bridge; the previous modes are
     recorded durably with `recordPendingSessionModes`),
  2. commits the recovery point with `v1CreateSnapshot` — **fail-closed**: a
     failed snapshot returns `502 SANDBOX_V1_PARK_SNAPSHOT_FAILED` and the
     rollout never starts, because the replacement container has no other copy,
  3. only then opens the durable park (`maintenance/replacement.json`
     `{ parked: true }`) and arms the exec lock.
     A stopped container is already parked (`skipped: 'not-running'`).
- **The park blocks writes** through the existing layers: `tenantParkGuard`
  refuses mutating `/api/admin/*` and `/debug/*` requests with `409
  TENANT_PARKED`, the `/control` entry refuses every action except `resume` and
  `checkpoint-park`, and `assertTenantExecAllowed` stops Worker-initiated exec.
  Reads stay available.
- **`resume`** clears the park record, unlocks exec, requests the boot through
  the v1 DO (which restores the committed record) and applies the recorded
  session modes once the gateway answers a real `/health` probe; if it is not
  ready yet the response says `sessionModes: "pending"` and the durable map stays
  for the next boot. Resuming an unparked tenant is a no-op that never wakes it.

`killGateway` remains process-scoped on v1 (`v1KillGateway` signals the gateway
process); nothing in the park path destroys or recreates the container.

#### Sandbox v1 boot, owner and repair (OCM-2)

The durable lanes that run **outside** a request must resolve the tenant's
runtime themselves — they have no request-scoped sandbox to inherit:

- **The `ActivityScheduler` resolves the runtime on every operation.** Its
  `sandbox()` helper calls `resolveRuntimeIntent` and then
  `acquireSandbox(..., runtime)`. Before this, the scheduler always acquired the
  **legacy** client: after a conversion, an alarm-driven wake or a scheduled
  snapshot would have booted the old container, archived the legacy filesystem
  and advanced the legacy `stable-backup.json` — silent divergence from the v1
  recovery point. `pending`/`unknown` tenants are **skipped** (logged as
  `activity.sandbox.skipped`), never guessed.
- **Scheduled snapshots on v1 write a v1 record.** A `snapshot` backup job on a
  converted tenant calls `SandboxV1.createSnapshot({ trigger: 'backup' })` (one
  call: DirectoryBackup + catalog + pointer + retention) instead of the legacy
  `stage → archive → finalize` pipeline. `restore`/`deprovision` jobs on v1 are
  reported as `skipped` with an explicit reason until those paths exist.
- **`restoreIfNeeded` no-ops on v1.** The DO restores the committed
  `DirectoryBackup` (or onboards a fresh tenant) inside
  `v1EnsureGateway`/`launchRestoredState`. The legacy state probe (`exists`) is
  intentionally unsupported on the v1 client, so running the legacy restore
  step would read `unknown` and wedge the durable boot job. The boot job's
  `restore` step therefore returns immediately for v1 and the `launch` step
  performs the restore; this is what lets the automation wake reuse the durable
  boot job instead of blocking the alarm on `ensureGateway`.
- **The DO's launch supervisor mirrors the legacy boot guards.** `v1Exec`
  returns a process handle, so `startGatewayOwner` races the supervised launch's
  exit against the `/health` poll: a **non-zero** exit before the port opens is
  a terminal launch failure reported in seconds (a zero exit stays ignored — the
  "lost the startup-lock race" path is judged by the health probe). On failure it
  tails `/tmp/gateway-stdout.log`; when the text matches a SQLite corruption
  signature it queues the durable `doctor-repair` job
  (`gateway-launch-state-database`) and fails with a retryable error, exactly
  like the legacy path. Relaunching cannot fix a malformed page, and the job
  re-queues the boot when the repair succeeds.

#### Sandbox v1 snapshots and the recovery point (PER-2)

On the legacy lane a snapshot is a `.sqsh` archive catalogued by
`backup-handle.json`; on v1 it is a `DirectoryBackup` record. The tenant needs
the same guarantees, so the v1 lane has its own catalog under the tenant's
migration scope:

| Object              | Purpose                                                                             |
| ------------------- | ----------------------------------------------------------------------------------- |
| `records/<id>.json` | One catalog entry per committed record (size, sha256, format, trigger)              |
| `stable.json`       | The stable pointer naming the committed recovery point (shared with the conversion) |
| `conversion.json`   | The migration artifact; doubles as the first catalog entry's fallback               |

- **`v1CreateSnapshot()` (DO).** `backupDirectory({ dir: '/home/openclaw' })`
  writes the record and post-encrypts it in place with F4E1 (failing closed when
  `BACKUP_ENCRYPTION_REQUIRED=true` and no key is set). The catalog entry is
  written **immediately** after the archive, so a killed invocation leaves a
  findable record instead of an unlisted orphan, then the stable pointer
  advances and retention prunes oldest-first beyond `BACKUP_MAX_VERSIONS`
  (default 3) while never removing the record just taken. A failed snapshot
  leaves the container running and the previous recovery point intact.
  **A stopped container is skipped, never started:** a 1.0 instance loses its
  disk on an inactivity stop, so `ensureRunning()` here would snapshot an empty
  `/home/openclaw` and advance the recovery point over real data. `v1CreateSnapshot`
  returns `{ state: 'skipped', reason: 'not-running' }`; `/control`
  `freeze`/`backup`/`checkpoint` report the skip, and the scheduler's snapshot
  job skips the cycle.
- **`/control` routing.** `backup`, `checkpoint` and `freeze` on a v1 tenant are
  served by the DO (the legacy `.sqsh` orchestration never runs for a converted
  tenant). `freeze` takes a snapshot **and only then** destroys the container —
  a failed snapshot keeps the tenant running (`SANDBOX_V1_FREEZE_SNAPSHOT_FAILED`)
  because the container is the only live copy until the record is committed.
  `unfreeze` is a no-op (the next request cold-boots from the recovery point).
  Other control actions still answer `503 SANDBOX_V1_CONTROL_UNSUPPORTED`.
- **Cold boot restores the pointer, not the conversion artifact.** A snapshot
  advances `stable.json`, so `restoreTenantStateIfNeeded` restores whatever the
  pointer names (`records/<id>.json`, falling back to `conversion.json` for
  pre-catalog records) and then launches with `OPENCLAW_RESTORED=true`. A
  conversion record **without** a pointer is an inconsistent C8 state and is
  refused — onboarding over it would silently discard a converted tenant.
- **The C8 barrier is the pointer→record pair.** `verifyRecoveryPoint` is the
  single read used by the runtime gate and the boot path: no pointer is
  `pointer-absent`, a pointer without a readable record is `record-absent`, an
  unreadable record is `record-unknown`, and a pointer naming a different record
  is `pointer-record-mismatch`. Only a matching, shape-valid pair is `complete`.

#### Sandbox v1 conversion orchestration (S5)

`POST /control` with `X-FourIA-Control: migrate` (forwarded by the Dispatch
Worker's `/_control/:slug/migrate`) starts the legacy → v1 conversion. The work
streams the legacy squashfs, extracts it in the v1 container and uploads a v1
DirectoryBackup — minutes of I/O, far longer than a Worker request lives (its
background continuations are cancelled shortly after the response). The control
entry is therefore **queue-only**: it enqueues a durable job on the `SandboxV1`
DO and answers in milliseconds; the conversion runs on the DO **alarm**.

This is deliberate rather than incidental: the idle-snapshot cadence alarm
(§ Sandbox v1 idle snapshot cadence) fired on the live tenant lane and committed
record `4437c278-…` with no operator action, which is the proof that alarms fire
on the Workers-for-Platforms tenant script. Queueing removes the request's
wall-clock, CPU and disconnect exposure from a destructive data migration.

**Contract**

| Outcome                                            | Status | Body                                                                                             |
| -------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------ |
| queued or in flight                                | `202`  | `{ ok: true, action: 'migrate', state: 'queued' \| 'running' }`                                  |
| already committed                                  | `200`  | `{ ok: true, action: 'migrate', state: 'committed', recordId }`                                  |
| durable job recorded `failed`                      | `502`  | `{ ok: false, action: 'migrate', state: 'failed', error, code: 'SANDBOX_V1_CONVERSION_FAILED' }` |
| `SandboxV1` binding missing, or the DO unreachable | `503`  | `{ ok: false, action: 'migrate', error, code: 'SANDBOX_V1_UNAVAILABLE' }`                        |

The staging/reserved-slug gate (`403`/`409`) is unchanged and still runs in
middleware **before** any DO access, so a refused `migrate` performs no R2 write
and no container addressing. The route logs a structured
`sandbox.v1.conversion.requested` event (`state`, `attempt`, `recordId`; no
secrets). A transport failure is reported as `503` and not as `failed`: nothing
was recorded, so the same call is safe to retry, whereas `failed` is a durable,
terminal state an operator must act on.

**Reading the status without starting a conversion (`migration-status`)**

`migrate` is an idempotent **enqueue** (it starts work from `absent`/`failed`), so
a poll that reused it would convert a tenant just by asking. The read-only
action `migration-status` (`X-FourIA-Control: migration-status`, forwarded by the
Dispatch Worker as `/_control/:slug/migration-status`) exists for polling:

| Tenant runtime                        | Outcome                                                                                |
| ------------------------------------- | -------------------------------------------------------------------------------------- |
| `legacy` (no conversion intent)       | `200 { ok: true, action: 'migration-status', state: 'absent' }`, no v1 DO addressed    |
| `pending` / `v1`                      | `200 { state: 'queued' \| 'running' \| 'committed' \| 'failed', recordId?, error? }`   |
| v1 binding missing, or the read fails | `503 { ok: false, action: 'migration-status', error, code: 'SANDBOX_V1_UNAVAILABLE' }` |

It calls **only** `v1ConversionState()` — never `v1RequestConversion()`, so it
cannot enqueue, never starts work and never wakes the container. The one write it
may perform is the lost-alarm re-arm (`ensureConversionArmed()` on a
non-terminal job), which arms the alarm slot but launches nothing. `error` (the
job's bounded failure message) is included only for `state: 'failed'`. The action
is allowed through the runtime middleware for `pending` and `v1` tenants and is
not part of the staging-only gate: reading a conversion state is not an action.

**Durable job**

- The job is the DO storage marker `migration:convert-request`
  (`{ requestedAt, attempt }`); the state the control plane polls is
  `migration:convert-status` (`queued` → `running` → `committed` | `failed`,
  plus the 1-based `attempt`). Both survive eviction and code updates.
- `v1RequestConversion()` is idempotent: `committed` returns as-is (including
  when only the R2 conversion record survives a DO storage reset), an in-flight
  job returns `queued`/`running`, and only `absent`/`failed` enqueues new work —
  `failed` starts the next `attempt`.
- Arming never postpones an earlier alarm: the conversion and the cadence share
  the DO's single alarm slot, so `armConversionAlarm()` only writes when there is
  no pending alarm or the pending one is later. Conversion arming wins.
- `alarm()` runs the **conversion first, then the cadence**, each in its own
  try/catch so one failing never starves the other, and finally re-arms a job
  that is still non-terminal (a backstop for a crash inside a turn).
- A lost alarm is healed on the read paths: `v1ConversionState()` (the poll),
  `v1RequestConversion()`, `v1Status()` and `v1EnsureGateway()` all call
  `ensureConversionArmed()`. It re-arms only when the durable marker exists and
  the state is non-terminal, so a queued job can never sit forever; a terminal
  state with a leftover marker is cleared instead. The read paths themselves
  never wake the container — only the conversion turn does, by design.
- Concurrency: `#conversionRun` is the single owner, mirroring `#gatewayStart`.
  A second alarm turn (or a read path that armed one) folds into the in-flight
  run instead of starting a second destructive conversion. The conversion's own
  lease/intent rules keep a crashed attempt idempotent on the next turn.
- Ordering is the crash contract: `running` is durable before the conversion
  starts, **`committed` is durable before the marker is dropped and before the
  gateway launch is attempted**, and a failure records `failed` (bounded
  message: 500 chars) without throwing. A launch failure therefore can never
  retract a commit — the next `v1EnsureGateway` retries the boot, whereas
  `failed` would re-run the destructive conversion. The container environment is
  rebuilt inside the DO from its own bindings (no secrets in DO storage).
- There is **no synchronous conversion entry point**: it was removed from the DO,
  the RPC surface and the client, so a request cannot await the migration.

The conversion still depends on a committed legacy recovery point
(`stable-backup.json`); a tenant without one fails closed with a recorded
reason and must be checkpointed first. Because the intent is a point of no
return, the runtime gate requires the **C8 barrier** (conversion record + stable
pointer) before a tenant serves on v1: an intent without it is `pending` and the
Worker answers `503 SANDBOX_V1_CONVERSION_PENDING` (only the idempotent
`migrate` entry stays reachable), so a failed conversion can never wake a fresh
empty v1 container over the legacy recovery point.

Key differences from the 0.12 stable line, all funneled through
`src/gateway/sandbox-exec.ts`:

- **No `startProcess` / `execStream` / `getLogs()`.** `sandbox.exec(argv)`
  starts a process and resolves once it is **launched**, not once it finishes;
  it never implies a shell. `startShell(sandbox, script)` wraps a shell command
  line in `['/bin/bash', '-lc', script]` and returns the handle;
  `runShell(sandbox, script)` waits and buffers `{stdout, stderr, exitCode}`;
  `outputOf(proc)` / `collectLogs(proc)` read an existing handle.
- **`listProcesses()` returns `ProcessStatus[]`** (`.state`, argv `.command`,
  `.startedAt`) with no methods. `findExistingGatewayProcess` fetches a live
  handle via `getProcess(id)` and returns a `GatewayProcess` wrapper
  (`.state`, `.command` string, `.refresh()`, `.waitForPort()`, `.kill()`,
  `.getLogs()`).
- **`waitForProcess`** delegates to `waitForExit({ timeout })` and kills the
  process on a local wait timeout (the SDK timeout cancels the wait only).
- **Busy-DO hardening for gateway reads.** A contended container-control channel
  can hold an `exec` launch for the SDK default (120s) and stall the Sandbox DO.
  Read routes therefore bound their sandbox work and degrade instead of hanging
  past the client timeout:
  - `ensureGateway(waitForReady:false)` (every API read route) is bounded by
    `ENSURE_GATEWAY_READ_TIMEOUT_MS` (30s) and returns `null` on expiry; the
    caller serves its normal "not ready" payload. Cold-start (`waitForReady:true`)
    is exempt.
  - A short-TTL positive readiness result (`GATEWAY_READINESS_CACHE_TTL_MS`, 5s)
    collapses concurrent read-path probes onto one; it is cleared on
    kill/restart/recovery.
  - The gateway port probe (`isGatewayPortOpen`) is bounded by
    `PORT_PROBE_TIMEOUT_MS` (10s) with a container-side `curl -m`, and marks the
    expected closed-port exit (`curl` 7 = connection refused) with
    `reportNonZeroExit: false` (operation `gateway.port-probe`) so the normal
    "gateway down, start it" outcome is not logged as a `sandbox.process.failed`
    error on every cold start — matching the `gateway.health-probe` behavior.
  - Expected CLI non-zero exits are not logged as failures either: `runShell`
    (via `reportNonZeroExit`) and `waitForProcess` (same option) let a call site
    mark an exit it handles as normal. The knowledge `wiki status` probe uses it
    (`operation: 'knowledge.wiki-status'`) because an uninitialized vault exits
    non-zero and the handler reports `initialized:false`. Timeouts always log.
  - Route flows that poll the container wrap their whole flow in a budget
    (`/api/admin/devices` 25s, `/debug/container-config` 20s).
- **No transport selection.** `SANDBOX_TRANSPORT` was removed; the preview uses
  RPC automatically.
- **`SandboxOptions.labels` was removed** upstream (worker identity is no longer
  attached at container start).

**Deploy cutover:** because the protocols are incompatible, a production deploy
must roll containers in one step — `wrangler deploy --containers-rollout=immediate`
— with `rollout_active_grace_period` at `0`. In-flight processes/terminals stop
and pre-deploy process IDs become invalid.

#### Debug tools on Sandbox v1 (ROOT diagnostics)

The ROOT `/debug/*` surface (`DEBUG_ROUTES=true`, ROOT only) is the operator's
escape hatch when a tenant misbehaves. A converted tenant is served by
`V1SandboxClient` (`SandboxV1` DO), not the legacy `Sandbox`, so each route was
audited against the v1 surface (`exec`, `getProcess`, `listProcesses`,
`containerFetch`, `wsConnect`, `killGateway`, `destroy`, `status`,
`isRuntimeActive`) and adapted where the legacy call had no v1 equivalent.

| Route                                  | What it calls                                                          | v1               | Notes                                                                                                                                                                                                                                                                                                     |
| -------------------------------------- | ---------------------------------------------------------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /debug/version`                   | `runShell('openclaw --version' / 'node --version')`                    | ✅ works         | `exec` → `waitForExit` → `output()` are all implemented on the v1 client.                                                                                                                                                                                                                                 |
| `GET /debug/processes`                 | `listProcesses()`                                                      | ✅ works         | v1 returns serializable `ProcessStatus`-shaped entries (`.state`, argv `.command`, no methods). The list is the **DO's tracked map**, so it is empty after a DO eviction — not a container inventory.                                                                                                     |
| `GET /debug/processes?logs=true`       | `getProcess()` + logs                                                  | ✅ works         | A finished process is read with `output()` (the v1 handle has no replayable `logs()` stream). A **running** process is never read (`logs_error: 'process running — live log stream disabled'`).                                                                                                           |
| `GET /debug/gateway-api?path=`         | `containerFetch` on the gateway port                                   | ✅ works         | v1 `fetch` forwards to `ctx.container.getTcpPort()`; a stopped container answers `503 SANDBOX_V1_SLEEPING` instead of waking.                                                                                                                                                                             |
| `GET /debug/cli?cmd=`                  | `startShell` + `waitForProcess` + `outputOf` + `status()` + `exitCode` | ✅ works         | Same CLI semantics as legacy; `DEV_MODE` instances still need `OPENCLAW_GATEWAY_TOKEN` forwarded (already done).                                                                                                                                                                                          |
| `GET /debug/logs[?id=]`                | gateway tail / finished-process output                                 | ✅ works         | No `id` → `findExistingGatewayProcess` → `getLogs()` tails `/tmp/gateway-stdout.log`. With `id` → `output()` fallback when `logs()` is unsupported.                                                                                                                                                       |
| `GET /debug/gateway-logs`              | `findExistingGatewayProcess` + `getLogs()`                             | ✅ works         | Always the file tail (a live SDK log stream never settles). The response now carries `path` and `runtime`.                                                                                                                                                                                                |
| `GET /debug/env`                       | Worker bindings only                                                   | ✅ works         | Adds `sandbox_runtime_actual` (the durable per-tenant lane this request was served by) next to the env-level `sandbox_runtime` switch.                                                                                                                                                                    |
| `GET /debug/container-config`          | `exec cat …/openclaw.json` + `outputOf`                                | ✅ works         | 20s end-to-end budget unchanged.                                                                                                                                                                                                                                                                          |
| `POST /debug/sqlite-integrity`         | `runShellResilient` (inline) / `ActivityScheduler` (durable)           | ✅ works         | The durable path resolves the tenant's runtime itself (`scheduler.sandbox()`), so the repair job runs on v1 too.                                                                                                                                                                                          |
| `GET /debug/sqlite-integrity/jobs/:id` | ActivityScheduler read                                                 | ✅ works         | DO-only. `.../jobs/:id/ws` is the streaming variant.                                                                                                                                                                                                                                                      |
| `GET /debug/gateway-token`             | secret binding                                                         | ✅ works         | No container access.                                                                                                                                                                                                                                                                                      |
| `POST /debug/stop-gateway`             | `findExistingGatewayProcess` + `killGateway`                           | ✅ works         | `killGateway` delegates to `v1KillGateway` (process-scoped, with a best-effort `pkill` sweep).                                                                                                                                                                                                            |
| `POST /debug/restart-openclaw`         | `killGateway` + cache resets                                           | ✅ works         | v1 branch: maintenance window → kill → `resetStartupCache` + `clearPersistenceCache` + `resetV1GatewayStart` → close window; the next request relaunches. **No checkpoint and no `restore-needed`**: a v1 restart destroys nothing and the marker is legacy-only. Same response fields + `runtime: 'v1'`. |
| `POST /debug/destroy-container`        | `destroy()`                                                            | ✅ works         | v1 `destroy` tears down the container and clears the DO's tracked processes/alarm; the route also clears the persistence/startup caches and the isolate start dedupe.                                                                                                                                     |
| `POST /debug/reprovision`              | v1 snapshot + C8 recovery-point check + `destroy()`                    | ✅ works         | The checkpoint is `v1CreateSnapshot({ trigger: 'backup' })` (bounded by `REPROVISION_CHECKPOINT_TIMEOUT_MS`); "restorable" is `verifyRecoveryPoint` (pointer→record), not `backups/<id>/meta.json`. Response carries `recordId` and `runtime: 'v1'`; still fails closed (409) without a restorable copy.  |
| `POST /debug/backup/:mode`             | `v1CreateSnapshot` (v1) / `scheduleBackupIfNeeded` (legacy)            | ✅ works         | On v1 the snapshot is one DO call (DirectoryBackup + catalog + pointer + retention) and needs no gateway, so the route never boots one. `sync` returns `recordId`.                                                                                                                                        |
| `POST /debug/rescue-container`         | `containerFetch` probe + Containers API + `destroy()`                  | ⚠️ partial       | Diagnosis and destroy work, but v1 `containerFetch` **never wakes** a stopped container: a sleeping tenant reads as unreachable. Use it for an orphaned/`Error` container, not to inspect a sleeping one.                                                                                                 |
| `POST /debug/r2-put?key=`              | R2 binding                                                             | ✅ works         | No container access.                                                                                                                                                                                                                                                                                      |
| `POST /debug/sleep`                    | durable `deprovision` job (legacy) / `stop()`                          | ❌ not supported | v1 has no stop primitive (the tenant sleeps on the platform inactivity timer). Answers `503 { code: 'SANDBOX_V1_SLEEP_UNSUPPORTED' }` instead of a `202` for work that never happens.                                                                                                                     |

Operational caveats:

- **Access blocks headless `/debug/*` calls on tenant hostnames** (the per-tenant
  Access app answers `302` to the login page). Drive these routes from a ROOT
  **browser session** on `https://<tenant>/_admin/` (the Debug Tools cards), or
  from Lerma's service token — not from a container-side `curl`.
- **`DEBUG_ROUTES=true` is Worker-local** and `/debug/*` is ROOT-only; neither is
  provisioned by Lerma for a client instance, so this is a
  staging/dev/operator surface.
- **Process ids are per DO incarnation.** After an eviction/redeploy the v1
  process map is empty (`/debug/processes` shows nothing, `/debug/logs?id=` 404s)
  even though the gateway is healthy — use `/debug/gateway-logs` (the file tail)
  instead.
- **Nothing here is destructive beyond what it says**: `restart`/`destroy`/
  `reprovision` never touch R2 on v1 except `reprovision`'s own snapshot (which
  advances the recovery point) and `rescue`'s Containers-API image repair.

### Container Image Build Profiles (dev vs production)

The Dockerfile takes a `BUILD_PROFILE` build arg (default `production`):

- **production** (default) installs only `xz-utils` — the slim image
  (~1.36 GB) used by production and by E2E.
- **dev** additionally installs the optional PDF/document tooling
  (`pandoc` + `weasyprint` — a large Python/cairo/pango stack, ~760 MB) that the
  dev-only Fouria Builder `fouria-leads` skill needs
  (`apps/fouria/skills/agent-only/fouria-leads/SKILL.md`; the Builder is injected
  only when `OPENCLAW_DEV_MODE=true`).

The profile is selected per wrangler environment via `image_vars` (Cloudflare's
build-time variables, equivalent to `docker build --build-arg`):

| Environment                            | `BUILD_PROFILE`        | Image                  |
| -------------------------------------- | ---------------------- | ---------------------- |
| `env.staging` (dev/test)               | `dev`                  | with pandoc/weasyprint |
| `env.production`                       | default (`production`) | slim                   |
| default / local Vite dev (local + E2E) | default (`production`) | slim                   |

There is a **single image definition** (`apps/fouria/Dockerfile`); CI
(`test.yml`) builds it with no build args when it pushes `unstable-<sha>`, and
the release CD (`deploy-fouria.yml`) promotes that same image by retagging it, so
released images are the slim production profile. The E2E `basic` (PR) and
`production` (develop) profiles both provision the slim production image (only
instance type/env differ). To build a dev image locally:

```bash
docker build --build-arg BUILD_PROFILE=dev -f apps/fouria/Dockerfile .
```

### Sandbox Recovery (stale + orphaned)

When the platform terminates a container behind a live Sandbox DO (idle
recycle, maintenance, SIGTERM), every RPC into it fails with the SDK's
`HTTP error! status: 500` wrapper — and retrying against the same instance
fails forever, bricking the tenant. `gateway/sandbox-recovery.ts` classifies
these dead-container errors (`isStaleContainerError`) and recovers
(`recoverStaleSandbox`): destroy the instance, reset the startup-probe and
persistence caches. The next RPC auto-provisions a fresh container and the
normal cold-start flow runs (R2 restore → `start-openclaw.sh` → gateway).

A container destroyed **out-of-band** (Cloudflare Containers dashboard "Destroy",
failed operator force-stop) is a second failure class: the DO is alive but points
at a placement that no longer exists, so RPCs fail with the SDK's _retryable_
admission errors (`no container instance available`, `maximum number of running
container instances exceeded`) instead of the 500 signature. Those are legitimate
during a cold start, so they are not treated as stale on their own.
`isContainerUnavailableError` + `trackProvisioningFailure` escalate them to a
recovery once the streak reaches `ORPHANED_INSTANCE_MIN_FAILURES` **and** spans
`ORPHANED_INSTANCE_GRACE_MS` (120s — longer than the SDK's ≈110s provisioning
budget, so a slow-but-healthy cold start is never cancelled mid-provision). The
streak is cleared whenever a sandbox RPC succeeds. `maybeRecoverSandbox` unifies
both classes for callers.

Recovery is wired into three failure points in `gateway/process.ts`:
`findExistingGatewayProcess` (listProcesses failures), `checkGatewayHealth`
(exec failures), and the `ensureGateway` `startShell` path (recover + retry
once). It is cooldown-guarded (`STALE_RECOVERY_COOLDOWN_MS`, 60s) and never
fires for deploy-window `OperationInterruptedError`. Permanent image errors still
require the rescue tool (`rescueContainer`, see OPERATOR-GUIDE §5.1). Logs use the
`[SandboxRecovery]` prefix.

#### Transient transport drops (retry, don't reprovision)

A third failure signature is not a dead container: `Network connection lost.`
(and similar socket/WebSocket transport messages). It appears when the DO↔container
connection drops _while the container is alive_ — most visibly while the Gateway is
applying a config/plugin reload (e.g. adding a model provider in the Control UI),
which restarts the Gateway and briefly tears down the container RPC/WS transports.

`isContainerTransportError` (`gateway/sandbox-recovery.ts`) classifies these, and
`isStaleContainerError` explicitly **excludes** them (even when the SDK wraps the
message in its `sandbox.<op> error` prefix) so a transient drop never destroys a
live instance. `ensureGateway` instead retries with a short backoff: the port
probe (`isGatewayPortOpenResilient`, up to `TRANSPORT_RETRY_ATTEMPTS`) and the
`startShell` spawn (once). Before this, a reload that dropped the transport mid
probe fell straight through to `start-openclaw.sh` → `Failed to start process` →
a 502 to the user, even though the Gateway came back seconds later. The retry
keeps the request alive across the reload window.

A Durable Object **code update** (a normal Worker/DO redeploy, or a Lerma
redeploy of the tenant script) resets the control session between the DO and its
container even though the container itself is not replaced. The cold start then
loses the transport on the **supervised port wait**: the atomic
`start-openclaw.sh start` process (config setup + gateway under one lock) keeps
running, but `waitForPort()` on the stale handle rejected and previously aborted
the cold start — killing a gateway that would have completed (observed as
`Network connection lost.`). The atomic `start` path has **no separate
setup-process wait**, so only the gateway port wait reattaches via
`getProcess(id)` and resumes (`waitForPortResilient`,
`gateway/sandbox-exec.ts`), bounded by the shared `STARTUP_TIMEOUT_MS` deadline.
The generic `waitForExitResilient` helper remains in `sandbox-exec.ts` for the
legacy/boot-smoke supervised waits. Only a non-transport error, an
un-reattachable process, or an exhausted deadline still fails the cold start.

The `openclaw doctor` diagnostics run through the same transport-resilient path:
`runShellResilient`, which retries the `exec` launch (`startShellResilient`) and
reattaches the wait (`waitForExitResilient`). Previously a transient
`The operation was aborted` drop on the doctor `startShell` surfaced as a hard
`500 { ok: false, error: "The operation was aborted" }`; it now reattaches to the
process and returns the structured diagnostic instead. `GET /api/doctor` still
runs inline this way; `POST /api/admin/config/fix` and `POST /debug/sqlite-integrity`
now **enqueue a durable maintenance job** on the ActivityScheduler DO
(`activity/maintenance-jobs.ts` + `activity/maintenance-executor.ts`) and return
`202 { accepted, jobId }`, because a large integrity check or doctor repair can
exceed a Worker request's lifetime (the old 30s/90s shell timeouts surfaced as
"timeout in under 2 minutes"). The DO alarm runs the command under a 14-minute
lease with bounded retries (`runMaintenanceOperation`, still `runShellResilient`
internally) and stores the bounded result on the job record, polled via
`GET /debug/sqlite-integrity/jobs/:id` / `GET /api/admin/config/fix/jobs/:id`.
Both endpoints fall back to an inline run when the `ACTIVITY` binding is absent
(local dev / tests). The doctor repair runs `openclaw doctor --fix --non-interactive`
(the supported command for index-only `integrity_check` corruption — missing or
wrong-count index entries — which verifies the table data, preserves a backup,
and `REINDEX`es) and **quiesces the gateway first**: `runMaintenanceOperation`
opens the cross-isolate maintenance window (`beginMaintenance(..., 'doctor-repair')`),
kills the gateway so the repair can take exclusive state ownership, then closes
the window so the next request re-launches the gateway.

**Restore → repair ordering (`restore-config-pending.json`).** A restore replaces
the container's writable layer but launches nothing, so a config repair applied
around a restore is silently overwritten by the next re-apply of the recovery
point (observed on `tenant-tarjetero-dev`: a config fix was lost on the next
gateway restore). Every successful restore (`restoreIfNeeded` and
`restoreBackupById`) now writes a durable `restore-config-pending.json` marker.
The durable boot job **and** the durable restore job launch the gateway on the
restored state through the shared `launchRestoredState` helper
(`src/post-restore.ts`); the admin sync/async-fallback restore route ends there
too, so every restore path has the same ordering guarantee. The helper calls
`ensureGateway({ waitForReady: true })` and then
`captureRestoredConfigVersion` — writing `configs/<ts>.json` and
`stable/config.json`, so the restored config becomes the **newest and stable**
config version and the dashboard's diff/history reflects what is actually live.
The marker is cleared only after that capture. A gateway-launch failure never
re-runs the destructive restore: restore jobs stay `succeeded` with
`detail: "gateway-launch-pending"` and the sync route returns
`gatewayReady: false`; the pending marker drives the next boot / maintenance
tick. A `doctor-repair` job checks,
before running: an active restore job, an open maintenance window / in-flight
restore (`getPersistenceBusyReason`), or a pending restored-config marker. In any
of those cases it **defers** (`MaintenanceJobDeferredError`): the job keeps its
attempts and queue position (60s re-check) instead of failing its three attempts
while the restore runs, and executes only after the gateway is up on the restored
state.

**Automatic repair when the launch is blocked by a corrupt state database.**
A warm R2 restore can bring back a damaged `state/openclaw.sqlite` (or an agent
`openclaw-agent.sqlite`). OpenClaw's schema preflight reads `PRAGMA
user_version` before anything else, so the Gateway dies within seconds with
`Error: database disk image is malformed` (`readSqliteUserVersion` →
`inspectStateDatabaseSchema`) and exits — no amount of relaunching helps. Two
things made that a multi-hour loop where the loading page sat on "Starting
OpenClaw Gateway":

1. **The crash is now detected immediately.** The readiness wait races a third
   signal, `watchGatewayEarlyExit`: the atomic cold start `exec`s the Gateway, so
   a **non-zero** exit of the supervised process before the port opens is a
   terminal launch failure. Previously the Worker waited out the whole shared
   `STARTUP_TIMEOUT_MS` budget (≈5 min) before even reading the diagnostics. A
   zero exit is deliberately ignored — that is the "lost the startup-lock race to
   a healthy owner" path, decided by the existing port/lock adoption check — and
   transport/plumbing errors from `waitForExit` (a DO code update resets the
   control session) park the watcher instead of failing a healthy boot.
2. **The documented repair is queued.** When the launch diagnostics match a
   corruption signature (`gateway/state-db.ts`:
   `database disk image is malformed`, `file is not a database`, `SQLITE_CORRUPT`,
   …), `ensureGateway` enqueues the durable `doctor-repair` maintenance job and
   throws `GatewayStateDatabaseRepairError` (retryable). The job carries
   `reason: 'gateway-launch-state-database'`, which makes it (a) skip the
   post-restore **gateway** gate in `preflight` — waiting for a gateway the
   corruption prevents is a permanent deferral; the restore-safety checks stay —
   and (b) **re-queue the durable boot** when the repair succeeds, since the
   launch itself is what failed and no request would retry it. Re-requesting is
   deduped (`MAINTENANCE_JOB_BUSY` → keep the active job), so repeated failed
   launches cannot pile up repairs. Without the `ACTIVITY` binding (local dev)
   the launch reports `no-scheduler` and the operator runs **Repair Environment**.

The loading page follows it: `/api/startup/ws` pushes `repair` (the DO's active
maintenance job) and the page renders "Repairing the OpenClaw state database
(openclaw doctor --fix, attempt N)…", suppressing the slow-step notice — the
container's own `startup-progress.json` still names the launch step the repair is
recovering from, which is exactly the stale "Starting OpenClaw Gateway (311s)"
message.

Job status is pushed over a **WebSocket** — `GET /api/admin/config/fix/jobs/:id/ws`(and the `/debug/sqlite-integrity/jobs/:id/ws` variant) — instead of the
dashboard polling the JSON endpoints above every 2s (one request + Worker log
line per tick for the whole multi-minute repair). The streamer
(`routes/api/maintenance-ws.ts`) reads the durable record from the
`ActivityScheduler` DO, sends a frame only when it changes, and closes `1000`
once the job is terminal; the dashboard's `subscribeMaintenanceJob` falls back to
a slow HTTP poll only while the socket is down. While a job is active the
streamer also calls the DO's `rearm()` **self-heal**, so a job whose alarm was
lost is scheduled again immediately.

The shared alarm is written through an **optimistic-concurrency guard**
(`armWithGenerationGuard`): the `ActivityScheduler` owns one alarm, so three job
enqueues, the warm-path schedule refresh, and the alarm handler's own trailing
re-arm can interleave at `await` points. A stale arm that read "no pending job"
could execute `setAlarm(later)`/`deleteAlarm()` _after_ a just-enqueued job's
`setAlarm(now)` and strand the job until the later time — observed as a
`doctor-repair` job enqueued at 12:57:30 that only began at 13:02:11 (exactly the
previous boot job's 14-minute lease). The guard bumps a stored generation and
abandons a superseded arm's write, so the newest arm (fresh read) owns the
alarm.

While the control session reconnects, the loading page's `GET /api/startup/ws`
stream treats a transport-drop snapshot as transient: it re-sends the last known
state (or a synthetic `starting` view) and backs off exponentially
(`STARTUP_WS_POLL_INTERVAL_MS` → `STARTUP_WS_MAX_BACKOFF_MS`, 30s) instead of
re-probing — and re-logging the same error — every 5s. A successful probe resets
the cadence.

The manual lever is `POST /debug/reprovision` (DEBUG_ROUTES, ROOT): it
checkpoints the live container first and **fails closed** — unless it just
committed a snapshot or a restorable backup already exists it refuses with
`409` rather than silently destroying tenant state; `{"force":true}` is the
explicit override for an already-unrecoverable container. The checkpoint is only
attempted when `isRuntimeActive()` reports a live runtime (so an idle container
is never woken) and is time-bounded (`REPROVISION_CHECKPOINT_TIMEOUT_MS`); the
restorability check HEADs the newest catalog entries' `meta.json` rather than
trusting the entry count. On success it destroys the instance, clears the
startup/persistence caches, and resets the orphan streak without ever contacting
the gateway — so it works when the container is entirely gone. Compare
`POST /debug/restart-openclaw` (gateway-aware, checkpointed) and
`POST /debug/destroy-container` (teardown only, used by E2E).

### Activity-Driven Wake (`ActivityScheduler`)

Automations ("jobs") run **inside the Gateway**, so the Gateway must be running
for a schedule to fire. Historically a Worker cron trigger woke every tenant on
every tick (expensive and since removed). `src/activity/` replaces it with a
per-tenant activity clock:

- **`ActivityScheduler` Durable Object** (SQLite-backed, binding `ACTIVITY`) —
  stores a mirror of the tenant's automation schedule and owns a single alarm.
  Deliberately a **separate** DO from the SDK `Sandbox` DO, which already uses its
  own alarm for `sleepAfter`.
- **Mirror** — warm paths (`GET /api/health`, only when `readiness.ok`) read
  `openclaw automations list --json` and push the earliest `nextRunAtMs` into the
  DO. The DO throttles this to once per 5 min, and it never wakes a sleeping
  container.
- **Mirror bootstrap** — the first mirror must not wait for a dashboard poll: the
  durable boot job mirrors the schedule right after `launchRestoredState`
  (`activity.schedule.mirrored`). A converted tenant that never opens the
  dashboard therefore still has a wake plan before it sleeps.
- **Alarm** (`activity/alarm.ts`) — fires `CRON_WAKE_AHEAD_MINUTES` (default 10)
  before the next job. On **legacy** it calls `ensureGateway({ waitForReady: false })`
  (fast; the schedule read retries until the gateway serves). On **v1** it
  enqueues the **durable boot job** (deduped) and returns `{ ready: false }`
  instead of blocking the alarm turn on a restore + launch; the boot job's launch
  step mirrors the schedule when the gateway is ready, and a 60s fallback check
  covers a lost mirror. No CLI read runs against a booting container, and the
  read is gated on the non-waking `isRuntimeActive()` read. No enabled jobs →
  alarm cleared → the container sleeps.
  A due-but-pending job is woken once per occurrence: `wokenFor` survives mirror
  writes and resets only when the gateway reports a **different** next run, so a
  wake-ahead cannot turn into a 60s wake/CLI loop while the job is pending. A
  woken occurrence that does not advance backs off 5m → 30m; a failed wake/read
  keeps the schedule and backs off (bounded), it never deletes the only trigger
  a sleeping tenant has. Every activity re-arm goes through the shared combined
  alarm, so it cannot postpone a boot/maintenance/backup job.
- **Non-waking diagnostics** — the alarm's memory watch checks
  `isRuntimeActive()` before its `/proc` exec. On v1 that exec would start a
  stopped instance (the disk is gone), so a skipped scheduled snapshot or a
  failed wake can never warm a sleeping tenant by accident.

Net: healing/wakeup is independent of user traffic, gated on real job activity,
and free when idle. Traffic may cold-start the container; jobs get a pre-warmed,
healed one. Requires the `ACTIVITY` binding in `wrangler.jsonc` and in the lerma
provisioner (`tenant_provisioner.ex` `resource_bindings`) + the
`new_sqlite_classes` migration (`dispatch_namespace.ex` `@do_sqlite_classes`).

### Change-Aware Backups

Scheduled snapshots are gated on whether tenant state changed. Detection is **on
by default**; the watch loop is opt-in:

| Flag                                                | Effect                                                                                                                                                                                                                                                                                |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BACKUP_CHANGE_DETECTION` (`true` default)          | **Phase 1 (cost).** The scheduled snapshot path (`scheduleBackupIfNeeded`) runs a container-side scan first. Unchanged state ⇒ no squashfs, no legacy staging, no lease, no upload. Only an explicit `false`/`0`/`off` disables it.                                                   |
| `BACKUP_CHANGE_WATCH=true` + `BACKUP_WATCH` binding | **Phase 2 (RPO).** The per-tenant `BackupWatch` DO checks every `BACKUP_CHANGE_CHECK_MINUTES` (2) and requests a durable snapshot from `ActivityScheduler` once a change has been quiet for `BACKUP_CHANGE_DEBOUNCE_MS` (60 s) and the floor (`BACKUP_INTERVAL_MINUTES`) has elapsed. |

Detection is default-on because a snapshot is not a cheap probe: it runs
`mksquashfs` over the whole tree, uploads the archive, and re-streams it through
the Worker for F4E1 encryption. Measured on `fouria-backup-tarjetero-dev`, two
snapshots 37 minutes apart differed by **0.08 %** (348 KB of 428 MB) — every tick
that re-uploaded it paid for the full archive. Only an explicit opt-out turns the
gate off, so a tenant that omits the var still gets the saving.

The scan itself costs one `find`, so the trade is deliberately lopsided: an
unnecessary scan is negligible, an unnecessary snapshot is not.

**Eviction compensates.** Because a quiet tenant now takes no scheduled
snapshot, its newest archive legitimately ages past the idle window — which would
make `TenantSandbox.onActivityExpired()` re-archive the whole tenant once per
sleep cycle. The hook therefore runs the same scan and treats `unchanged` as
coverage, stopping inline (see [Backup before
deprovisioning](#backup-before-deprovisioning)).

**Scan (`src/change-detection.ts`).** Runs `find` over the real state roots (`/home/openclaw`, `/root/openclaw/.openclaw`, `/root/clawd`; the home directory is symlink-only for state, and `find` does not follow symlinks). It prunes the transient legacy stage and `.npm`. It compares against the baseline marker `/tmp/.fouria-last-backup` and prints `unchanged`, `changed`, or `active` (changed within the debounce window). Only entries _inside_ the roots count (`-mindepth 1`), so a root's own mtime, which moves when the legacy stage is created, cannot read as a change. Known limitation: a pure deletion with no accompanying write is not detected until the next write in the same tree, because `find -newer` sees mtimes, not removals. Every ambiguous outcome is `changed`: a missing baseline, no roots, a non-zero exit, a timeout, or a thrown error.

**Baseline.** `/tmp/.fouria-pending-backup` is stamped before a snapshot captures state and promoted to the baseline only after the catalog commits. A failed snapshot leaves the baseline untouched. This runs on every snapshot, whether or not a flag is set, so enabling a flag later has a valid baseline.

**A restore forgets the baseline.** Restored files keep their archived (older) mtimes, so a `-newer <baseline>` scan would call the replaced tree `unchanged` and skip the first post-restore snapshot. Both successful restore paths (`restoreIfNeeded`, `restoreBackupById`) call `resetChangeBaseline`, which removes both markers so the next gate takes a snapshot unconditionally.

### Archive size and restorable TTL

Two `createBackup` options in `src/persistence.ts` decide how large an archive is
and whether it stays restorable. Both matter more than they look: the archive is
uploaded by the container, re-read and re-encrypted by the Worker, downloaded
again on restore, and stored `BACKUP_MAX_VERSIONS` times, so every byte is paid
for `maxVersions + 2` times.

- **`SNAPSHOT_EXCLUDES`** — passed to `mksquashfs` as wildcard patterns (the SDK
  normalizes `**` away and warns). Excludes `.npm`, `*.log`, `*.tmp`, `.cache`,
  and `workspace/*node_modules*`. It deliberately does **not** exclude plugin
  `extensions/<id>/node_modules` (a restore has no reinstall step of its own —
  `start-openclaw.sh` only installs when the extension _directory_ is missing, so
  stripping the deps leaves a plugin that loads and then fails at import), SQLite
  databases, or session transcripts.
  **Opt-in (`BACKUP_ARCHIVE_EXCLUDES`, default `false`).** This is a fidelity
  trade, not a pure win: the archive is the recovery tier, and an archive restored
  without those paths no longer matches the filesystem a Tier 1 workspace snapshot
  restores (or the pre-eviction tree). The default keeps the archive a complete
  image; enable it per instance once you know the tenant regenerates those paths.
- **`SNAPSHOT_ARCHIVE_TTL_SECONDS`** (10 years) — the SDK **enforces** the TTL at
  restore time (`doRestoreBackup` refuses an archive past `createdAt + ttl` with
  `BackupExpiredError`), so it is a restorability gate, not a storage-lifecycle
  hint. The previous 7-day value silently made a retained recovery point
  unrestorable — `cleanupOldBackups` prunes by _count_, so an idle or frozen
  tenant could keep `maxVersions` archives that no boot could apply, and
  `isUnrecoverableArchiveError` then dropped the stable pointer and failed the
  boot closed. The durable `repair` step heals existing tenants:
  `extendArchiveTtl` rewrites `meta.json` for every archive below the minimum,
  preserving `dir`/`createdAt`/`encrypted`/`encryptionVersion`, and reports the
  ids as `ttlExtended` / `ttlFailed` from `POST /api/admin/storage/reconcile`.
  **The restore path heals its own archive** (`ensureArchiveRestorable`, called
  right after `restore-target.json` is published in both `restoreIfNeeded` and
  `restoreBackupById`): the `repair` step only runs inside a snapshot job, so a
  tenant that cannot boot could never reach the repair that would make it
  bootable — the wedge observed on `staging-client-dev`, where all three
  catalogued archives (7-day TTL) expired on 2026-10-29 and every boot then
  failed in ~250 ms on the expiry check after its container was replaced. The
  SDK's expiry message (`Backup <id> has expired (created: …, TTL: …s)`) is now
  also classified as an unrecoverable archive, so a metadata rewrite that cannot
  take effect drops the dead entry instead of retrying it forever.

Compression is still the SDK default (`lz4`, chosen for speed, not ratio); moving
to `zstd` trades container CPU for bytes and is tracked separately.

**Watch (`src/backup-watch/`).** `watch.ts` is the pure state machine. `durable-object.ts` is the adapter, and `arm.ts` holds gating and policy with no DO imports, so warm paths can call it without an import cycle.

- Never wakes a container: liveness is the non-waking `isRuntimeActive()`. A not-running tenant is disarmed and not scanned.
- **Keepalive guard.** Every scan is an `exec` that renews the container's sleep timer. A tenant unchanged for `BACKUP_CHANGE_IDLE_DISARM_MS` (10 min, below the staging 15 min sleep window) is disarmed, and re-arming never resets the idle clock. The watch can therefore delay a quiet container's sleep by at most that window.
- **No starvation.** An `active` tenant defers snapshots for at most `BACKUP_CHANGE_MAX_DEFER_MS` (10 min), so constant writes still get snapshotted.
- **No hot-path work.** Snapshots are never taken from the watch directly. It enqueues the existing `ActivityScheduler` durable `snapshot` job, which keeps that job's gateway start, lease, retention, and restore-pending guards.
- **Gated before probing.** Each tick first asks the scheduler for `getSnapshotGate()` — one DO storage read, no container RPC — and re-arms **without** any probe or scan when a snapshot job owns the durable slot or the floor has not elapsed since the newest _committed_ snapshot (whichever trigger produced it: dispatch cron, operator, or this watch). A gate read failure is fail-open (logged, then the normal tick). Previously a busy tick still ran a liveness probe plus a `find` scan every two minutes (~1,440 container operations per tenant per day) to discover there was nothing to do (durable-agents audit finding F7). A changed tenant inside the floor is no longer scanned, so `dirtySince` starts at the first scan after the floor; `maxDeferMs` still bounds starvation.
- Arming happens on a running tenant only: the dispatch `backup` control action (`/control`) and the warm `refreshActivitySchedule` path (gateway healthy).

**Wiring.** `wrangler.jsonc` declares the `BackupWatch` class (migration `v8`) and the `BACKUP_WATCH` binding in every environment. In production the tenant's bindings come from lerma, which binds `BACKUP_WATCH` only when the `backup_change_watch` provisioning flag is on (see `apps/lerma`). Both flags are plain-text vars that lerma always sets to `true` or `false`, so turning a flag off explicitly reverts it on redeploy.

### Persistence policy telemetry (Phase 0)

Before changing the backup policy (see `docs/plans/persistent-workspace-phase0.md`)
the current behaviour is measured, not guessed. Three Analytics Engine events are
the input; they are **observability only** and never gate a snapshot, restore, or
stop:

| Event                  | Emitted from                                                  | `double1`     | `double2`         | `double3` |
| ---------------------- | ------------------------------------------------------------- | ------------- | ----------------- | --------- |
| `persistence-snapshot` | `finalizeSnapshot` (success), `ActivityScheduler` (fail/skip) | durationMs    | archiveDurationMs | bytes     |
| `persistence-restore`  | `restoreIfNeeded` (recovery point), `restoreBackupById`       | durationMs    | bytes             | —         |
| `backup-change-window` | `BackupWatch` tick (`runBackupWatchTick` observation)         | dirtyWindowMs | scanMs            | —         |

`bytes` is the captured workspace size from the archive's `meta.json` (one small
metadata read at `archive`), so "the artifact is large" and "the pipeline is slow"
are separable. `trigger` (`scheduled` / `manual` / `deprovision`) and the skip
reason (`interval-not-elapsed`, `unchanged`, `not-running`, `backup-lock-busy`)
are blobs, so a quiet series has a visible denominator. `backup-change-window`
records the continuous dirty time the watch accumulated (`dirtySince`), which is
the RPO distribution the checkpoint cadence will be chosen from.

Queries and the phase gates are in `docs/plans/persistent-workspace-phase0.md`.

### Workspace snapshots on eviction (Phase 1, flag-gated)

`WORKSPACE_SNAPSHOT=true` makes `TenantSandbox.onActivityExpired()` capture a
platform filesystem snapshot immediately before eviction and commit the handle at
`workspace/snapshot.json` in `BACKUP_BUCKET` (one head handle; image-pinned;
30-day TTL). A new `workspace-snapshot` Analytics Engine row records
`durationMs`, `bytes`, `trigger=evict` and the outcome.

Capture is **observability-grade in Phase 1**: boot still uses the archive, so a
capture failure (or an unsupported runtime) never blocks the archive/stop path —
`unsupported` and `failed` are warnings, not errors.

**The fallback must restore the same tree.** Phase 2 may boot from the handle and
fall back to the archive, so the two tiers have to hold the same content. The
archive is a complete image today because `SNAPSHOT_EXCLUDES` is behind
`BACKUP_ARCHIVE_EXCLUDES` (**off by default**); an instance that opts into the
excludes makes an archive restore diverge from a snapshot restore by exactly
those paths. Do not enable the excludes on an instance whose snapshot boot must
be able to miss into the archive without regeneration.

**Runtime prerequisite.** Cloudflare only supports snapshots on a container
application with `scheduling_policy: "durable_object"`. The tenant container is
on the `default` policy, and that cannot be changed in place: the policy is
immutable, a new Durable Object class/namespace is required, and the legacy
`Container` class is unsupported under it. Tier 1 therefore depends on the
Sandbox 1.0 workload adapter, tracked in
[#380](https://github.com/0xCAB0/fouria/issues/380) (S2–S4), not on a config
change. `SandboxV1` already runs on `durable_object` and implements the same
`snapshotWorkspace()` method, so the capability is real where the runtime allows
it. Keep the flag off until an instance is opted into v1; see
`docs/plans/persistent-workspace-phase1.md`.

### Backup before deprovisioning

A container is never deprovisioned without a committed snapshot. The durable
backup-job queue (`src/activity/backup-jobs.ts`) gains a `deprovision` operation
(`mode: 'sleep' | 'freeze'`); the `ActivityScheduler` alarm runs it as
`snapshot → stopIfIdle()` (sleep) or `snapshot → destroy()` (freeze), and a
failed snapshot retries rather than tearing the container down (fail closed).

**Reuse a snapshot that already covers the state.** `onActivityExpired()` stops
inline instead of enqueueing a snapshot when the container has already been idle
for a whole window and an existing archive already captures the current
filesystem — a second archive is pure cost. Coverage is proved two ways, cheapest
first:

1. **Age.** `hasRecentSnapshot()` (a pure catalog read in `src/persistence.ts`)
   answers whether the newest committed archive is newer than the instance's idle
   timeout (`resolveSleepAfterMs()`: staging 15m, dev 2h, production 6h) plus
   `IDLE_SNAPSHOT_REUSE_SLACK_MS` (5m, so the archive finalized just before the
   previous stop attempt still counts).
2. **No change since the last commit.** With `BACKUP_CHANGE_DETECTION` on — the
   default — a quiet tenant takes **no scheduled snapshot at all**, so its newest
   archive legitimately ages past the idle window and the age test alone would
   re-archive it on every sleep cycle. When the gate is enabled, the hook runs the
   same `scanTenantChanges()` the scheduled path uses and treats `unchanged` as
   coverage: the baseline is promoted only after a catalog commit, so
   `unchanged` means that archive already holds this state. `changed`/`active`,
   a missing baseline, or detection being off fall through to the archive.

A read failure is treated as "no recent snapshot" (fail closed). This also breaks
the loop where the deprovision snapshot's own RPCs renewed the activity window,
`stopIfIdle()` refused, and the next expiry snapshotted again (observed on
`tenant-tarjetero-dev`: three 422 MB archives per idle expiry while the container
never stopped).

**Deprovision archives stamp the schedule.** `finalizeSnapshot` receives
`stampScheduled: true` for `deprovision` jobs as well as `trigger: 'scheduled'`
ones, so a committed deprovision archive resets `lastScheduledBackup` and the
next cron tick does not archive the same state again.

**Stop only on inactivity (stop-time re-check).** The snapshot can take minutes
on a large tenant, so the container may be used again between enqueue and stop.
The sleep branch therefore does **not** call `sandbox.stop()` directly: it calls
`TenantSandbox.stopIfIdle()`, an RPC on the Sandbox DO (the activity authority).
That re-reads the DO's own idle window — renewed by every request, including a
proxy request for an in-flight chat — and the container process list, and stops
only when the window has genuinely elapsed and no transient work is running. If
the tenant became active during the snapshot the job returns `skipped`
(`detail: active:<reason>`) and leaves the container running; the next idle
expiry enqueues a new job. Refusals: `activity` (window renewed),
`transient-activity`, `probe-failed` (unknown is **not** idle),
`billable-production` (never auto-sleeps), `not-running`.

- **Idle sleep.** `TenantSandbox.onActivityExpired()` stops inline when an
  existing snapshot covers the state (a recent archive, or `unchanged` from the
  change scan when detection is on), and otherwise enqueues a `deprovision`
  (sleep) job, so the Sandbox DO leaving memory never loses an in-flight
  multi-minute snapshot. With no `ACTIVITY` binding it falls back to the
  previous stop-now path.
- **Freeze.** `POST /control` freeze enqueues a `deprovision` (freeze) job and
  acknowledges immediately (`202 { ok, accepted: true, frozen: false, jobId }`),
  failing closed at enqueue (`409` when a durable job is already active). The
  caller does not hold a multi-minute request open: it polls the tenant's
  **`GET /api/health/backup/:jobId`** until the job reaches a terminal status.
  A snapshot failure surfaces as a `failed` job and the container stays running,
  so lerma never marks a still-running instance frozen.
- **Platform stops** (an inactivity timeout firing after the DO left memory, a
  container crash, or main-process exit) have no Worker hook; the periodic +
  change-watch snapshots bound that residual RPO.

The pollable endpoint (`GET /api/health/backup/:jobId`) carries no in-Worker
authentication — the same posture as `/api/health` — and is reachable by the
lerma control plane's health-monitor service token through the **health Access
tunnel** (the `*.fouria.io/api/health` path application in
`apps/fouria/iac/access.tf`). Cloudflare Access path inheritance covers the
`/backup/*` sub-path, and it is also listed explicitly so the tunnel allowlist
documents the endpoint.

### Health & Status Endpoints

`GET /api/health` is the **unified** health endpoint: it returns the startup/readiness/liveness
probes plus the canonical `status` (`running|starting|stopped`), `processId` and `processType`.
It requires no authentication inside the Worker, but at the Cloudflare edge it is
Access-protected by path-scoped applications (`*.fouria.io/api/health`) that carry
no role requirement: they admit any authenticated user (browser) and the lerma
health-monitor service token (headless).
`GET /api/status`, `GET /api/admin/gateway/status/polling` and `GET /api/admin/gateway/status/ws` are thin
aliases over the same snapshot. Browser clients use the sibling paths outside `/api/health/*` —
`GET /api/health-ws` (WebSocket stream every ~2s), `GET /api/health-snapshot` (HTTP snapshot), and
`GET /api/health-components` (Cloudflare component states) — while `/api/health`, `/api/health/ws`,
and `/api/health/components` remain mounted for headless service-token consumers (lerma). The
browser-facing sibling paths exist because `/api/health` and `/api/health/*` are covered by the
path-scoped `FourIA Health - <slug>` Access app whose immutable AUD differs from the host app's: the
browser's host-app session is invalid there, so Access answers with a 302 login redirect to
`*.cloudflareaccess.com` — which WebSocket handshakes cannot follow and browser `fetch()` calls
reject on cross-origin CORS.

By default `/api/health` **warms** the container (lerma keep-alive). `?passive=1` probes **without** warming
— the ROOT `/_admin/diagnostics` surface uses it so an operator can debug an instance whose container is
down. The admin dashboard subscribes to the health WebSocket (with an HTTP polling fallback) and never
blocks the UI on gateway status.

`GET /healthz` is a Kubernetes-style **liveness** probe: one non-waking
`isRuntimeActive()` read, always `200 { ok, ts, status }` (`status` = `active` |
`idle`), never warming the container or probing the gateway. It exists because
`/healthz` otherwise fell through to the gateway proxy (`app.all('*')`), which
warmed the container and waited for the port — ~108 s on a cold instance, so the
platform killed the request before it answered. Readiness stays on `/api/health`.

A single `/api/health` request issues **one** gateway `/health` `curl` exec, shared by the
readiness and liveness probes. They previously each ran their own, so every poll spawned a
**pair** of identical `curl` processes (visible in `/debug/processes`) — a duplicate on the
frequent lerma probe. The process scan and the `/health` probe are now both fetched once, in
parallel, and reused across the snapshot.

#### Gateway memory (`gateway: { pid, rssBytes, threads }`)

The gateway is one Node process in a 4 GiB / no-swap container whose RSS is dominated by
per-`worker_thread` V8 heaps that an in-process restart never reclaims
(`docs/plans/sandbox-memory-usage.md`). A dev tenant "rebooted" once and the only way to answer
"was it memory?" was to SSH in and read `/proc`, `dmesg` and the gateway log by hand.

`src/gateway/memory.ts` makes that a first-class signal:

| Piece                            | Behaviour                                                                                                                                                                                                                                                                                              |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `buildGatewayResourceScript()`   | One shell exec: reads the cgroup memory limit (`mem:<bytes>`; v2 `memory.max`, v1 fallback) and `pgrep -x 'openclaw-gatewa'` → `gw:<pid>:<rssKb>:<threads>` per live gateway server (same exact `comm` as the soft reload, so it never measures the `openclaw gateway run` launcher; zombies skipped). |
| `readGatewayMemory()`            | Runs it (`gateway.memory-read`, bounded) and returns `{ resources, containerLimitBytes }`; `readGatewayResources()` is the resources-only view. Never throws: a failed read is `null`s, not a broken health probe.                                                                                     |
| `resolveMemoryRestartBytes()`    | Derives the watchdog threshold: 75% of `containerLimitBytes`, or `null` below 3 GiB (too small for a healthy gateway; the gate is below the 4 GiB `standard-1` class to tolerate runtime accounting). Unknown limits fall back to 3 GiB (the 4 GiB `standard-1` value).                                |
| `sampleGatewayResources()`       | Throttles the read to `GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS` (60s), so the frequent health poll issues at most **one** extra exec per interval and only a `fresh` sample writes a metrics row. A cached `null` is served inside the window too, so a broken probe is not retried every poll.            |
| `maybeRestartGatewayForMemory()` | The dev-gated watchdog (below).                                                                                                                                                                                                                                                                        |

`attachGatewayResources()` in `routes/public.ts` attaches the sample to the snapshot as
`gateway` and writes a `gateway-resources` Analytics Engine row (`double1` = RSS bytes,
`double2` = threads, blob1 = `gateway-resources`, blob2 = customer, trailing
`platform`/`environment` blobs like every other cost row). It is applied by **both** `/api/health`
(and `/api/health-snapshot`) and the `/api/health-ws` stream, and **only when the snapshot is
`running`** — a sleeping/stopped container is never woken for a diagnostic.

**Dev-gated memory watchdog.** When the sample reaches the **instance-aware threshold** —
`resolveMemoryRestartBytes()`: 75% of the container's cgroup memory limit (3 GiB on a 4 GiB
`standard-1`, 4.5 GiB on a 6 GiB `standard-2`, 6 GiB on an 8 GiB `standard-3`, 9 GiB on a 12 GiB
`standard-4`) — on a **dev** instance (`isDevAgentsEnabled`: `-dev` hostname,
`OPENCLAW_DEV_MODE`, or local `DEV_MODE`), the gateway is replaced — `killGateway` +
`resetStartupCache` + `ensureGateway({ skipRestore: true, bypassMaintenance: true })`. That is the
only mechanism that actually returns RSS to baseline.

The limit is read from the container on every sample (one extra local read in the same exec), so an
instance resized by an **instance-type rollout** picks up the new threshold without a Worker
deploy. Instances smaller than 3 GiB (i.e. `basic`/`lite`) are skipped (`instance-too-small`): a
restart cannot make an undersized box viable, and a 3 GiB fixed threshold on an 8 GiB
`standard-3` would be only 37% of the limit and restart a healthy gateway.

Design constraints worth keeping:

- **The threshold guards the container limit, not OpenClaw's warning.** OpenClaw's own
  `rss_threshold` is 1.5 GiB and fires permanently on a healthy-but-large dev gateway
  (~2.1 GiB observed, ~2.6 GiB peak). Using it as the action threshold would restart the gateway
  in a loop.
- **Scale with the instance, never hardcode.** A fixed byte threshold means a different headroom
  on every instance type; the ratio keeps ~25% of the limit free for the kernel and the next
  allocation burst.
- **No soft-reload step.** A measured in-process `SIGUSR2` restart does **not** reclaim RSS
  (1.14 → 1.38 → 1.18 GiB), so reloading first would only add a failed attempt.
- **`skipRestore`.** A kill leaves the filesystem untouched; re-entering the R2 restore barrier
  would deadlock on the backup lock for no benefit.
- **Cooldown claimed before acting.** `GATEWAY_MEMORY_WATCHDOG_COOLDOWN_MS` (30 min) is recorded
  _before_ the replacement, so two concurrent health polls cannot start two restarts.
- **Detached.** The watchdog is dispatched through `executionCtx.waitUntil` (or left detached when
  there is no context), so a multi-minute replacement never delays or fails a health response.

Every decision returns a stable machine reason (`not-dev`, `no-resources`, `instance-too-small`,
`below-limit`, `cooldown`, `rss-above-limit`), and an actual restart logs a structured
`gateway.memory.pressure` event with `pid`, `rssBytes`, `rssMib`, `threads`, the resolved
`limitBytes`/`limitMib`, and the measured `containerLimitBytes`/`containerLimitMib`.

**Two-stage policy and continuous sampling (2026-11-03).** The watchdog above only runs when
something calls `attachGatewayResources`, and **nothing polls `/api/health` while the gateway is
healthy** — the Control UI talks to the gateway directly. During the `tenant-tarjetero-dev` outage
that blind spot was total: `gateway-resources` had no rows for ~17 h, so the crash had no memory
evidence at all.

- `checkGatewayMemoryHealth()` (`gateway/memory.ts`) is now the single sampler used by both
  `/api/health*` and the durable `ActivityScheduler.alarm()` (`runMemoryWatch()`), so a tenant is
  observed whether or not a browser is polling. The throttle and the `gateway-resources` row are
  unchanged.
- Every **fresh** sample also writes `gateway-memory-sample` (`blob1`, `blob2` = customer,
  `blob3` = `running | absent`, `double1` = RSS, `double2` = threads, `double3` = container
  limit). The row exists even when no gateway process is running, so a gap in the series means
  "nobody sampled" instead of "nothing happened".
- `decideMemoryAction()` is the pure policy behind both callers:
  **≥ 60 %** of the cgroup limit (`resolveMemoryCheckpointBytes`, `GATEWAY_MEMORY_CHECKPOINT_RATIO`)
  → `checkpoint`; **≥ 75 %** (`resolveMemoryRestartBytes`) → `restart`, escalating to
  `checkpoint-and-restart` when the newest archive is older than
  `GATEWAY_MEMORY_LOSS_BUDGET_MS` (15 min). `maybeRestartGatewayForMemory` delegates to the same
  function (mapping its reasons back to the vocabulary above), so the thresholds cannot drift
  between the request path and the durable alarm.
- The durable alarm performs the checkpoint by enqueuing a snapshot job with
  `trigger: 'memory-pressure'` **before** restarting, so the work is on disk regardless of what
  the kernel does next. The checkpoint has its own cooldown
  (`EMERGENCY_SNAPSHOT_COOLDOWN_MS`, 10 min) and is surfaced as a
  `gateway.memory.policy` event.
- The request path may still only restart: it cannot own a durable job, and blocking a health
  response on a 450 MB snapshot is worse than restarting.
- `healthWsHandler` now passes its `executionCtx` to `attachGatewayResources`. Without it the
  detached watchdog promise was cancelled as soon as the frame was sent, so the watchdog was a
  no-op on the one path a browser actually keeps open.
- **Sampling never wakes the container, and it yields to every durable job.** `sandbox.exec`
  auto-starts a stopped container, so an unbounded `/proc` read from the alarm raced
  `container.start()` during a local restore (`gateway.memory-read failed (launch-failed)` /
  `Container failed to start`, and a workerd `jsgInternalError` that aborted the start).
  `checkGatewayMemoryHealth` now takes the non-waking `isRuntimeActive()` read first (bounded by
  `PASSIVE_PROBE_TIMEOUT_MS`, like the passive health probe) and returns
  `{ action: 'skipped', reason: 'runtime-inactive' }` with a `fresh: false` sample when the
  runtime is off — so a sleeping tenant is never billed for a diagnostic and never woken by an
  alarm tick. `runMemoryWatch` additionally returns before sampling while a **boot**, **backup**,
  or **maintenance** job is active (queued/running/retrying): those own the container, and the
  watchdog is observability, never a participant in a boot.

Design and remaining work: [`docs/plans/graceful-shutdown-and-emergency-checkpoints.md`](/llms/platform/reference/plans/graceful-shutdown-and-emergency-checkpoints/index.md).

**A destructive restore is never triggered by an unreadable container.** `probeContainerState()`
returns `present | absent | unknown`, and `restoreIfNeeded` only restores on `absent`. On `unknown`
it throws the retryable `ContainerStateUnknownError` and writes **no** `restore-needed` latch, so
the durable boot job retries instead of replacing the filesystem of a container it cannot read.
This is what turned the 2026-11-03 wedged container (dead control channel) into a destructive
450 MB restore that timed out and pinned every later boot behind
`RestoreInFlightError (restore-outcome-unknown)`.

#### Cloudflare component states

The dedicated `GET /api/health/components` endpoint returns a quick view of the
Worker-bound platform pieces rendered by the ROOT diagnostics horizontal
**Cloudflare Components** widget. `GET /api/health` also carries the same
(redacted) `components` entry — but **only on the non-passive path**. Lerma's
HealthMonitor polls `/api/health?passive=1` frequently and never reads
`components`, so the passive probe skips the component probe entirely (no
`isRuntimeActive` read, no R2 HEAD) to keep the frequent poll slim:

| Component      | Probe                                                                                  |
| -------------- | -------------------------------------------------------------------------------------- |
| Container      | Non-waking `sandbox.isRuntimeActive()` (`up` / `sleeping` / `unknown`)                 |
| Durable Object | `Sandbox` + `ACTIVITY` bindings present and the control channel answered               |
| R2             | Bindings-shaped `BACKUP_BUCKET` plus a HEAD of `backup-handle.json`                    |
| AI Gateway     | Present **only** when all three gateway credentials are configured (`up` = configured) |

Every probe is **non-waking** and individually bounded (3s), so the endpoint is
safe on a sleeping instance and can never hang the health response; on failure
the `components` entry is omitted instead of blocking. The AI Gateway is
inclusion-gated on credentials and reported as `up` ("configured") — the probe
deliberately does not issue an authenticated Cloudflare API call on every poll,
and a missing account-scoped token must not read as a false outage.

Each component also carries binding-derived `facts` (a `{ label, value }[]`
shown in the widget's hover panel) and a Cloudflare dashboard `logsUrl` (a
per-product logs/metrics deep link opened on click). Both are derived from
Worker bindings/env only — **never** an authenticated Cloudflare API call — so
the surface stays non-waking and free (`src/gateway/cloudflare-links.ts` builds
the links: Container → Workers Logs/observability, Durable Object → DO
metrics/logs, R2 → the `BACKUP_BUCKET_NAME` bucket page, AI Gateway → the
gateway page). When the account id is not bound the links fall back to the
dashboard's documented `?to=/:account/…` form; a component without a resolvable
link renders as a plain (non-clickable) card. The unified `GET /api/health`
snapshot **redacts** `facts`/`logsUrl` (`redactComponentDetails`) — it is a
broadly-consumed public probe — and serves only the state summary; the enriched
view is exclusive to `GET /api/health/components`.

On phones the widget collapses into a compact **status widget** that shares a
row with the gateway and persistence-lock status widgets at the top of the
diagnostics page (not floated or overlaid): a single pill carrying the Cloudflare
icon, a worst-case rollup status dot and an `up/total` summary. Tapping it
expands the same component strip as a compact panel. To build that parallel row,
`AdminLayout` omits the topbar gateway + persistence indicators on the
diagnostics page on phones (`App.tsx`) and `DiagnosticDashboard` renders
`GatewayStatusLed` + `PersistenceLockIndicator` beside the Cloudflare widget. The
gateway pill gains an identifying icon and the lock an
`Idle`/`Backing up`/`Restoring` label (its color still encodes the state); both
use compact phone sizing so the three widgets stay on one line on normal phones
and wrap gracefully on very narrow ones. The desktop layout is unchanged — the
topbar keeps its indicators and the Cloudflare card renders full-width.

The ROOT `/_admin/diagnostics` page renders its operational tools as separate
clickable **Fouria DevOps** widget cards (backup connectivity + configuration
editor + paired devices). Each card's title row toggles its body while the
action buttons stay live, and the cards load their data asynchronously and
render their chrome immediately — there is no blocking loading animation for the
data that populates them. On phones each option collapses into a vertical
**icon + name** menu row (collapsed by default; tap to open) with the option's
actions revealed inside the opened body. The Configuration Editor reflows to the
viewport: action buttons stack full-width, the section tabs scroll horizontally,
long JSON wraps, and the raw textarea shrinks. The Debug Tools grid drops to a
single column so nothing forces sideways scrolling, and the admin bottom nav
uses flexible items (with safe-area insets) so every nav entry fits. The
floating dev role switcher (`DevUserSelectorCompact`) is lifted above the bottom
nav on phones so it neither covers the last nav item nor overflows the viewport.

The gateway status indicator lives in the admin **topbar** beside the
persistence-lock indicator and the notification bell (`GatewayStatusLed`, mounted
in `App.tsx`'s `AdminLayout`). Its color-coded LED reflects the gateway status
(`running`/`starting`/`sleeping`/`stopped`/`unknown`); clicking it reveals the
complete status value (status, process, process type, the startup/readiness/
liveness probes, restore state and last update) and a **reboot icon** that
performs the same soft-reload/restart as the removed Gateway Controls banner. The
reboot action is hidden behind an `Admin` badge when `/debug/*` is disabled,
matching the previous gating.

#### Live gateway detection

On the Worker-driven cold start the gateway is a **supervised `sandbox.exec`
process** (`/bin/bash -lc 'exec openclaw gateway run …'`), so it _is_ visible to
`listProcesses()` for the life of the container. The legacy `full` mode (manual
runs, `test:gateway:boot`) still launches it as a detached background child of
`start-openclaw.sh` (`… &`), and a container provisioned by an older image may
still be serving a detached gateway during a rolling deploy — for those, after
the script exits the gateway binary is **invisible to `listProcesses()`**.
To avoid false `no_process` reports on log/status surfaces, those endpoints
(`GET /api/admin/logs`, `GET /debug/logs`, `GET /debug/stop-gateway`,
`POST /api/admin/gateway/restart`, `GET /api/admin/audit-logs`) call
`findExistingGatewayProcess(sandbox, { probeLive: true })`, which falls back to
a bounded `/health` probe and — when the gateway responds — returns a synthetic
`gateway-live` process whose logs are read from `/tmp/gateway-stdout.log`. The
passive health snapshot path deliberately does **not** probe so a down container
fails fast.

### Log Levels

| Level   | When to use                   |
| ------- | ----------------------------- |
| `debug` | Local development             |
| `info`  | Production default            |
| `warn`  | High-traffic production       |
| `error` | Security-sensitive production |

---

## TypeScript & Tooling

### Type Generation

Worker types are auto-generated:

```bash
bun run types   # wrangler types → worker-configuration.d.ts
```

**Never manually edit `worker-configuration.d.ts`** — it's regenerated from `wrangler.jsonc`.

### Testing

```bash
bun run test             # Unit tests (Vitest)
bun run test:coverage    # With coverage report
bun run test:client      # SPA component/page tests (jsdom)
bun run test:ui          # Playwright UI suite (Vite dev server + mocked API)
bun run test:ui:headed   # Same, with a visible browser
bun run test:e2e         # E2E tests (cctr + Playwright)
bun run test:e2e:cli     # CLI-only E2E (cctr)
```

The Playwright UI suite (`test/ui/`, config `playwright.config.ts`) drives the
real SPA served by `bun run dev` with per-test `/api/*` mocks — no Worker,
container, or Cloudflare credentials. It covers the app shell, role-based
navigation and redirects, every admin page's render, and the notification bell.
See [docs/TESTING.md](/llms/platform/reference/testing/index.md#ui-tests-playwright-local) for the
full description and the Nix `PLAYWRIGHT_BROWSERS_PATH` notes.

### Formatting & Linting

```bash
bun run format          # oxfmt (auto-format)
bun run format:check    # Check formatting
bun run lint            # oxlint
bun run typecheck       # tsc --noEmit
```

---

## Common Tasks

### Adding a New API Endpoint

1. Create route handler in `src/routes/api/<domain>.ts`
2. Register in `src/routes/api.ts`
3. Add client method in `src/client/api.ts`
4. Write tests (unit + integration)
5. If new dependencies: update `package.json`

### Adding a New Environment Variable

1. Add to `OpenClawEnv` interface in `src/types.ts`
2. If passed to container: add to `buildEnvVars()` in `src/gateway/env.ts`
3. Update `.dev.vars.example`
4. Document in [../../docs/ENV-VARIABLES.md](../../docs/ENV-VARIABLES.md)
