F FourIA staging 2026.10.29 GitHub ↗

FourIA — Worker Dashboard Architecture

On this page

Deep-dive into the apps/fouria component, the core Tenant Worker of FourIA.

See ../docs/DEVELOPMENT.md for project-level overview and ../docs/ENV-VARIABLES.md for environment variable reference.


Architecture Overview

Browser
   │
   ▼
┌─────────────────────────────────────────┐
│       Cloudflare Worker (index.ts)       │
│                                          │
│  Middleware: Logging → Auth → RBAC       │
│                                          │
│  Routes:                                 │
│    /api/*     → Hono API routes          │
│    /_admin/*  → React SPA (Vite build)   │
│    /debug/*   → Debug endpoints (ROOT)   │
│    /*         → Proxy to OpenClaw GW     │
│                                          │
│  Activation: Worker restore → onboard    │
│              → patch → launch gateway    │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│   Cloudflare Sandbox Container           │
│                                          │
│  start-openclaw.sh:                      │
│    1. openclaw onboard (auth setup)      │
│    2. Enable plugins (unified billing)   │
│    3. moltlazy patch (config injection)  │
│    4. Start Cloudflare Tunnel (optional) │
│    5. background gateway + health poll   │
│                                          │
│  ┌────────────────────────────────────┐  │
│  │     OpenClaw Gateway               │  │
│  │  - Control UI on port 18789        │  │
│  │  - WebSocket RPC protocol          │  │
│  │  - Agent runtime                   │  │
│  └────────────────────────────────────┘  │
└─────────────────────────────────────────┘

Frontend Architecture (Vite + React SPA)

The dashboard is a React 19 SPA built with Vite 6 and served from /_admin/.

Build & Dev Flow

bun run dev           → Vite dev server on :8787 (vite.ui.config.ts)
    │                     Hot-reload for frontend changes
    │                     Proxies /api/*, /debug/* to :8788 (worker)
    │
bun run dev:worker    → Vite dev server on :8788 (vite.config.ts)
    │                     Runs the actual Worker in workerd via the
    │                     Cloudflare Vite plugin; Worker changes hot-reload
    │
bun start             → bun run build:docker && vite dev
                          Full local stack (build + Worker dev server)

Key insight: both dev servers are Vite. bun run dev only handles the admin SPA; the Worker (routes, auth, gateway, DOs, containers) runs under the Cloudflare Vite plugin via bun run dev:worker, the same plugin used by vite build. The plugin has no wrangler dev flags: local E2E vars are injected through its config hook and .dev.vars is loaded from this directory.

Vite Configuration

// vite.config.ts — Worker dev server (bun run dev:worker)
base: '/_admin/'              // All assets served under /_admin/
server: {
  port: 8788,
  allowedHosts,               // DEV_ALLOWED_HOSTS (tunnel hostnames)
}

// vite.ui.config.ts — admin SPA dev server (bun run dev, :8787)
server: {
  port: 8787,
  proxy: {
    '/api':   'http://localhost:8788',
    '/debug': 'http://localhost:8788',
    '/cdp':   'http://localhost:8788',
  },
}

Exposing the Worker through a tunnel (e.g. Telegram webhooks): Vite blocks unknown Host headers, so pass the public hostname explicitly — DEV_ALLOWED_HOSTS=foo.trycloudflare.com bun run dev:worker (comma-separated; a leading dot allows a whole domain). Point the tunnel at :8788, not the SPA at :8787 (it does not proxy /telegram-webhook).

Client Module Structure

src/client/
├── App.tsx              # Root component with React Router
├── main.tsx             # Entry point — mounts React app
├── api.ts               # Typed API client → /api/admin/*
├── gateway-gate.ts      # Health circuit breaker (park/resolve gateway queries)
├── GatewayHealthContext.tsx  # App-wide passive health subscription + gate wiring
├── pages/               # Route-level pages
│   ├── HomePage.tsx       # Dashboard overview
│   ├── AgentsPage.tsx     # Agent management
│   ├── ChannelsPage.tsx   # Connections/integrations
│   ├── KnowledgePage.tsx  # Knowledge base/wiki
│   ├── AccountPage.tsx    # User/team management
│   ├── IntegrationsPage.tsx   # Integration config
│   ├── SecretsPage.tsx    # Vault management
│   ├── BackupsPage.tsx    # Backup management
│   ├── DiagnosticDashboard.tsx  # Costs + diagnostics
│   ├── AuditLogsPage.tsx  # Gateway ledgers: audit activity + background tasks
│   └── AutomationEditorPage.tsx # Automation workflows
├── components/          # Shared UI components
└── hooks/               # Custom React hooks

Frontend ↔ Backend Communication

The frontend (src/client/api.ts) calls /api/admin/* endpoints. These are served by the Worker’s Hono routes (src/routes/api.ts), which:

  1. Verify RBAC (requireWriteRole() for mutations)
  2. Execute OpenClaw CLI commands inside the Sandbox container
  3. Parse CLI output and return JSON

Client circuit breakers (health-gated gateway queries)

Two client-side circuit breakers keep the dashboard responsive when the OpenClaw gateway is not ready (issue #371):

  • #1 — gateway queries are asynchronous. Pages paint immediately and never block first paint on a gateway response; mutations are fire-and-notify through NotificationProvider. api.ts routes every gateway-touching call through gatewayApiRequest.
  • #2 — gateway queries require the passive health stream to be UP. The single app-wide GatewayHealthProvider (src/client/GatewayHealthContext.tsx) subscribes once to /api/health-ws?passive=1 and drives a module-singleton GatewayGate (src/client/gateway-gate.ts). UP is defined as snapshot.readiness.ok === true; starting/stopped/sleeping are not UP (sleeping is healthy-idle but the gateway is not running). Before dispatching, apiRequest awaits whenGatewayUp(signal); while down the query is parked and the GatewayGateNotice indicator shows, then the queue flushes automatically on the next UP snapshot. A transient (503/transport) failure is retried once after the next UP.

The gateway-touching boundary is the GATEWAY_TOUCHING_PREFIXES manifest in api.ts (/agents, /models, /channels, /config, /devices, /integrations, /secrets, /knowledge, /audit-logs, /dreaming, /gateway/restart). Diagnostics (/debug/*, /logs, /gateway-api, /gateway/status/polling, /gateway-token, /version) and worker-local surfaces (/storage, /costs, /billing, /users, /vault, /releases, /fouria-docs, /whoami) are deliberately not gated, so an operator can still inspect a down gateway and non-gateway features keep working.

Diagnostics (DiagnosticDashboard). The ROOT dashboard is explicitly skip-not-park: useGatewayRunning() (GatewayHealthContext.tsx) reports snapshot.readiness.ok ignoring the ROOT bypass, and the page only dispatches its container reads (/config/raw, /config) while it is true; they re-run automatically when the gateway transitions back to running. The debug-routes flag is Worker-local and still loads. Every call is dispatched after paint (scheduleAfterPaint → requestIdleCallback, setTimeout(0) fallback) and the widgets render their waiting state first, so nothing is blocked on a container response: the Configuration Editor renders its tabs/editor with a non-blocking “Loading configuration…” / “Waiting for the gateway to start…” notice (gatewayReady prop on ConfigSection) and the device list shows a waiting state until Refresh. Gateway-only actions (device list/approve, config save, channel probe) are disabled/guarded while down; doctor --fix repair is NOT one of them: it is the recovery action for a broken/down gateway (the backend stops the gateway itself), so a dedicated Repair Environment control sits outside the Configuration Editor (always rendered for ADMIN/ROOT, disabled only while a repair is in flight) and the in-editor repair button stays enabled while the gateway is down. The Debug Tools that call the container (logs, processes, version, sqlite integrity) and the recovery actions (restart/rescue/destroy) stay available on purpose — they are how an operator inspects and revives a down gateway — while the ones that require a running gateway (gateway-api, stop-gateway) are skipped with an inline explanation.

ROOT bypass. ROOT operators are never gated: App.tsx passes bypass={whoami?.role === 'ROOT'} to the provider, which flips the gate’s bypass (GatewayGate.setBypass). This preserves the “ROOT can always reach a tenant” rule and keeps the operator’s non-passive device-pair request (GET /api/admin/devices/ws → listDevices → ensureGateway({ waitForReady: false })) able to wake a sleeping container. The notification bell is passive for every non-ROOT role (subscribeDeviceNotifications(..., { passive: true }) → ?passive=1), so a regular dashboard load never wakes a sleeping tenant; only ROOT’s stream is non-passive. Server-side, a passive /api/admin/devices/ws first does the non-waking readRuntimeActive() read and, when the runtime is not live, emits an empty snapshot and skips the gateway event stream. Once the ROOT wake brings the gateway UP, the shared health stream flips to up and every parked query (for all roles) flushes. The GatewayGateNotice is suppressed while bypassed, since ROOT actions are never queued.

The gate is fail-open when no provider is mounted (non-SPA callers, unit tests): the module default is up. The provider installs the shared gate during its first render, before any child page effect runs. It is a module singleton rather than a useState initializer because React StrictMode double-invokes initializers in dev — two gates would leave api.ts parked on a different instance than the provider drives.


Backend Architecture (Hono Worker)

Request Pipeline

Incoming Request
    │
    ├─► Logging middleware (all routes)
    ├─► Sandbox init (getSandbox() into context)
    │
    ├─► Public routes (no auth):
    │     /sandbox-health
    │
    ├─► CF Access Auth middleware:
    │     Extract JWT → validate → set accessUser
    │
    ├─► RBAC resolution:
    │     resolveCustomerRole → set userRole, customerSlug
    │
    ├─► Route dispatch:
    │     /api/*     → API routes (auth + RBAC)
    │     /_admin/*  → Admin UI (SPA assets)
    │     /debug/*   → Debug routes (ROOT only)
    │     /*         → Catch-all proxy to gateway
    │
    └─► Gateway proxy:
          HTTP proxy → containerFetch(port 18789)
          WS proxy   → WebSocket relay with token injection

API Route Structure

// src/routes/api.ts — route aggregation
api.route('/admin', adminApi)
  ├── /whoami, /version, /logs, /gateway-api, /debug-status, /gateway-token  (misc)
  ├── /devices/*      → devicesRoutes
  ├── /storage/*      → storageRoutes
  ├── /releases/*     → releasesRoutes
  ├── /gateway/*      → gatewayRoutes
  ├── /config/*       → configRoutes
  ├── /cron/*         → cronRoutes
  ├── /dreaming/*     → dreamingRoutes
  ├── /users/*        → usersRoutes
  ├── /integrations/* → integrationsRoutes
  ├── /knowledge/*    → knowledgeRoutes
  ├── /secrets/*      → secretsRoutes
  ├── /agents/*       → agentsRoutes
  ├── /models/*       → modelsRoutes
  ├── /channels/*     → channelsRoutes
  ├── /costs/*        → costsRoutes
  ├── /vault/*        → vaultRoutes
  ├── /billing/*      → billingRoutes (plan, checkout, portal, invoices, freeze-status, freeze, unfreeze)
  └── /audit-logs/*   → auditLogsRoutes (audit activity ledger + task ledger)

// Write gate: POST/PUT/DELETE/PATCH require ADMIN or ROOT
// Read gate: GET/HEAD available to all authenticated roles

Build identity. GET /api/admin/version and GET /debug/version report the FourIA Worker’s own fouria_version (component CalVer from apps/fouria/package.json) and fouria_commit (short git SHA) alongside the container’s OpenClaw/Node versions. Both are injected into the bundle by vite.config.ts (__FOURIA_VERSION__ / __FOURIA_COMMIT__, consumed in src/build-info.ts; GITHUB_SHA wins in CI, git rev-parse --short HEAD locally). The DiagnosticDashboard Version Info tool surfaces them for operators.

Cloudflare error and trace correlation

The Worker emits structured worker.request.started and worker.request.completed records with the method, path, status, duration, and Cloudflare Ray ID (cfRay). Unhandled exceptions are logged as worker.request.failed with the exception name/message/stack and the same request fields. Cloudflare also attaches its invocation $metadata.requestId and $metadata.traceId to each Worker log, so the invocation’s request, logs, and trace can be opened together in Workers Observability. Error records pass an Error object to console.error as well, so Cloudflare populates its exception metadata (including $metadata.error) alongside the indexed event fields.

Sandbox command failures are emitted as sandbox.process.failed. These records include a stable operation label and phase, the Sandbox processId when a process was launched, and available exit-code/timeout/duration fields. The processId is the join key to the processId field on the Containers dataset’s process.complete event; inspect exitCode and timedOut there rather than trusting the generic event message, which can say success when the process itself exited non-zero. Raw shell commands, stdout, and stderr are deliberately omitted because they can contain tenant config or credentials.

Persistence failures use stable events such as persistence.backup.failed, persistence.config-version.failed, persistence.restore.failed, and storage.restore.failed. Config-version and restore errors include backupId when an archive exists; use it to join to the backup.create / backup.restore container events. A config-version capture failure is best-effort: it identifies the archive whose supplemental config snapshot was not captured, but does not mean the archive itself failed to be created.

Backup and restore work also emits paired persistence.phase records. Each record has an operationId, operation (backup or restore), phase, and status (started, completed, or failed); terminal records include durationMs, and phases after an archive handle exists include backupId. source names the route or job, cfRayId correlates request-triggered work with worker.request.*, and jobId correlates work running later in the ActivityScheduler. Use operationId to group the phases; if a started phase has no terminal record, the Worker may have been interrupted during that phase. Useful phases include sandbox.createBackup, archive.encrypt, catalog.commit, sandbox.restoreBackup, and legacy-state.materialize.

For a failure investigation, filter the cloudflare-workers dataset to event = "sandbox.process.failed", record processId and the Worker $metadata.requestId/$metadata.traceId, then search the containers dataset for the same processId (and the tenant’s $container.applicationId). For a request-level failure, filter to event = "worker.request.failed" or use cfRay to pivot to the matching edge request. For backup/restore CPU-limit investigations, filter to event = "persistence.phase", find the last started event for the relevant cfRayId or jobId, then group by its operationId to see which phases completed before it.

Worker observability export destinations

Account-level Cloudflare Observability → Destinations export this Worker’s logs and traces to Lerma over OTLP/HTTP JSON (issue #337). The destination names are referenced in the observability block:

// staging
"observability": {
  "logs":   { "destinations": ["lerma-worker-logs"],   "head_sampling_rate": 0.1, "persist": false },
  "traces": { "destinations": ["lerma-worker-traces"], "head_sampling_rate": 0.1, "persist": false }
}
// production
"observability": {
  "logs":   { "destinations": ["lerma-worker-logs-prod"],   "head_sampling_rate": 0.1, "persist": false },
  "traces": { "destinations": ["lerma-worker-traces-prod"], "head_sampling_rate": 0.1, "persist": false }
}

The -prod destinations point at https://lerma.paso4.io/api/ingest/otel/v1/*; the un-suffixed pair points at https://lerma-dev.paso4.io/....

  • apps/fouria/wrangler.jsonc carries the matching names in each env block (staging and production, direct deploys).
  • WFP-deployed tenant workers get the names from Lerma’s Lerma.Cloudflare.Observability.worker_metadata/0 (WORKER_OBSERVABILITY_LOGS_DESTINATIONS / WORKER_OBSERVABILITY_TRACES_DESTINATIONS). The CD sets the staging names for staging and the -prod names for production, so a redeploy inherits them.
  • Lerma is the destination of record: persist: false stops the duplicate Cloudflare-side copy, and the head sampling rate (10% at rollout) bounds export volume. Lerma drops message text, stacks, URLs/queries, user agents, request/response bodies and DB query text at ingestion; only allowlisted metadata is stored.
  • Cloudflare gzips each OTLP/HTTP JSON batch (Content-Encoding: gzip with Content-Type: application/json). LermaWeb.Plugs.OtlpBodyReader runs before Plug.Parsers on the ingest routes and decompresses it; without that step the normal JSON parser returns 400 and the destination retries forever.
  • The export only feeds diagnostics + the Lerma instance Worker telemetry widget. It never replaces HealthMonitor (still the source of instances.status) or billing, and it is not a probing path — no container is woken to produce it.

Ops: create the two destinations under Observability → Destinations with the custom header Authorization: Bearer <OTEL_INGEST_TOKEN>, set the token and the destination names on Lerma (staging first), then redeploy the tenants. Delivery status shows Last: n minutes ago; the uptime numbers come from GET /api/analytics/v1/uptime/<script_name>?window=24h.

Gateway start lifecycle + stuck-start watchdog

A cold start is one shared promise (ensureGateway) spanning the restore barrier → the supervised start-openclaw.sh start launch → the port wait. A wrapper that never execs the gateway, or a container control-channel drop, can leave that promise “in progress” for many minutes while every request logs Gateway start already in progress … waiting... and waits on it.

The start is now tracked per sandbox key with a phase (probe → restore → launch → port-wait) and a per-phase timer. post-start runs detached after readiness (see below), so it is deliberately not part of the awaited start promise:

  • gateway.start.begin / gateway.start.phase structured events record the attempt, phase, elapsed time, and (after launch) the process id; the “already in progress” log carries phase, age, and attempt.
  • isGatewayStartStuck flags a phase that overruns its budget + GATEWAY_START_STUCK_GRACE_MS (the restore phase gets GATEWAY_RESTORE_PHASE_BUDGET_MS because the legacy stage may take 10 min; all other phases use STARTUP_TIMEOUT_MS).
  • When the watchdog trips, recoverStuckGatewayStart emits a gateway.start.stuck structured event, kills the stuck supervised process (killGateway), clears the startup/readiness caches, and returns so the next request starts clean. After GATEWAY_START_STUCK_MAX_ATTEMPTS consecutive resets it reprovisions the container (recoverStaleSandbox).
  • Separately, findExistingGatewayProcess uses classifyGatewayStart to detect a start-openclaw.sh start wrapper older than the cold-start budget with the port closed and kills it immediately (source startup-wrapper-age) instead of waiting out another full readiness budget.
  • A non-owner isolate that merely observes a running wrapper waits to the same deadline the owner’s watchdog uses — STARTUP_WRAPPER_STUCK_BUDGET_MS (STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS) measured from the wrapper’s start — and re-probes /health before any kill. A fixed STARTUP_TIMEOUT_MS measured from observation expired while the wrapper had already execed the gateway and was still binding, killing it mid-boot, releasing the startup lock, and making the waiting second wrapper re-run the entire setup (observed on staging-client-dev: a ~14-minute cold boot). The kill now only fires once the wrapper has genuinely overrun the owner’s own budget with the port still closed.

The port-wait also completes on a positive /health probe. The SDK’s waitForPort drives a container-side port-watch + log-readiness subscription that can hang indefinitely on some boots even after the gateway is serving. Waiting on it alone let the shared start promise overrun its budget, so the watchdog force-killed a healthy gateway and reprovisioned the container (observed on tenant-client-dev: gateway.start.stuck phase=port-wait after 368 s while HTTP proxying returned 200). The wait races waitForPortResilient against the same positive /health probe the public proxy uses. If that waiter reaches its deadline and parks, the watchdog makes one fresh bounded health check before resetting: a late-positive response emits gateway.start.late-ready and is adopted without killing the gateway. A failed health RPC is treated as unknown, not closed, and defers the reset until a later request can probe again; only a confirmed closed port follows the normal stuck-start reset path.

Process count is not a health signal. listProcesses() returns every tracked process for the container’s lifetime — including exited/error entries the Worker never garbage-collects — so the raw length only grows. The log line now reads listProcesses returned <total> processes (<running> running) and every health decision (classifyGatewayStart, the gateway matcher) considers only running entries. The SDK exposes no completed-process cleanup, so the list is append-only; never gate health on the total.

Gateway protocol ledgers (/api/admin/audit-logs)

The AuditLogsPage reads the two append-only ledgers from the Gateway protocol (docs.openclaw.ai/gateway/protocol/ledgers):

  • GET /api/admin/audit-logs — audit activity ledger (audit.activity.list, operator.read), newest first. Supports agentId, sessionKey, runId, kind, status, direction, channel, after, before, limit, and cursor; returns { events, nextCursor? }. The bundled admin-http-rpc plugin allowlists tasks.* but not audit.*, so this read goes over the Gateway WebSocket RPC using OpenClaw’s reference client (@openclaw/gateway-client/browser + @openclaw/gateway-protocol) bridged through sandbox.wsConnect, authenticating with the shared gateway token as an operator (see src/gateway/gateway-rpc.ts). It falls back to openclaw audit --json when the WS path is unavailable (older Gateway); the CLI fallback gets the resolved OPENCLAW_GATEWAY_TOKEN injected because a fresh sandbox.exec shell does not inherit the gateway process env.
  • GET /api/admin/audit-logs/tasks — task ledger (tasks.list, operator.read), newest first. Supports status (repeatable), agentId, sessionKey, limit, cursor, and sortBy; returns { tasks, nextCursor? }. RPC-first over the admin-http-rpc plugin, with an openclaw tasks list --json CLI fallback (same token injection).

Both surfaces are ADMIN-only (absent from DIAGNOSTICS_PATHS) and return an empty page with an error string when the ledger is unavailable, so the dashboard degrades gracefully instead of failing the request.

Why a WebSocket client. admin-http-rpc only exposes an allowlisted method subset. Methods outside it (notably audit.*, plugins.*, secrets.*, agents.files.*) require the wire-v4 Gateway WebSocket handshake (connect.challenge → connect → hello-ok). src/gateway/gateway-rpc.ts adapts sandbox.wsConnect to the reference client’s GatewayProtocolSocket; the client version sent in the connect frame is injected at build time from catalog.openclaw (__OPENCLAW_VERSION__).

RPC-first agent board + integrations

Eligible dashboard routes try Admin HTTP RPC first, the official Gateway WebSocket RPC second, and keep the CLI as the final fallback for older Gateways (helpers return null/false on transport or method failure):

  • Agent board (/api/admin/agents) — list reads agents.list and channel bindings in config.get over Admin HTTP RPC. If either result is unavailable, the Gateway WebSocket client fetches the missing read(s) over one connection; the CLI remains the final fallback. Create → agents.create; delete → agents.delete; identity (name/emoji/avatar) → agents.update; model (+ fallbacks) → config.patch on agents.entries.<id>.model. Explicit bind/unbind operations have no matching RPC method and stay on the CLI; persona read/write are filesystem operations.
  • Devices (/api/admin/devices) — list, approve, reject, and approve-all retain Admin HTTP RPC as the first path, use Gateway WebSocket methods (device.pair.list, .approve, .reject) if HTTP RPC is unavailable, and retain the CLI as the compatibility fallback.
  • Models (/api/admin/models) and channel probe (/api/admin/channels/probe) — models.list and channels.status use Admin HTTP RPC first, then Gateway WebSocket RPC, then the existing CLI path.
  • Integrations (/api/admin/integrations) — the whole endpoint is WebSocket-free. GET / reads the allowlisted channels.status + agents.list over the admin HTTP RPC, and resolves plugin enabled-state from config.get (plugins.entries.<id>.enabled) merged with the static PLUGIN_FIELDS catalog — no plugins.list socket. When no channel accounts are configured the supported-channel catalog is synthesized locally from CHANNEL_FIELDS instead of shelling out to openclaw channels list --all. Plugin/channel toggles and non-secret configure fields → config.patch; secrets reload → the openclaw secrets reload CLI. secrets.apply and agent bindings have no 1:1 RPC method and stay on the CLI. CLI fallbacks remain for a degraded HTTP RPC transport. The admin RPC fetcher also cancels a late containerFetch body when its timeout wins, preventing the Sandbox DO’s “ReadableStream received over RPC disconnected prematurely” warning and the churn that dropped later sockets. The same cleanup is shared by the public proxy and the gateway-api/diagnostics probes via src/utils/container-rpc.ts (containerFetchWithTimeout cancels a late body, wsConnectWithTimeout closes a late socket). Reading a running gateway’s logs must not open its live SDK stream (it never settles and is later disconnected): GatewayProcess.getLogs() tails /tmp/gateway-stdout.log, and /debug/processes?logs=true / /api/admin/logs?id= refuse a live stream for a still-running process. The catch-all proxy logs only an allowlisted header subset (safeHeaderLog) — never the forwarded map, which carries CF Access credentials and cookies.
  • Config / secrets / agent resolution — config set/unset and dreaming settings use config.patch over Admin HTTP RPC, then Gateway WebSocket RPC, then the CLI fallback. POST /api/admin/secrets/reload uses secrets.reload (WebSocket); resolveAgentFlag (shared by the models/knowledge CLI calls) resolves the roster with agents.list over Admin HTTP RPC, then WebSocket RPC, then CLI.

secrets apply has no --yes. The current CLI accepts only --from <plan.json>, --dry-run, --allow-exec, and --json; the plan itself is the confirmation. Both integrations and dreaming apply without --yes.

Agent Studio (/_admin/agents)

The Agent Garage is an all-in-one, RPC-only workbench (roster + Overview, Capabilities, MCP, Memory, Automations, Connections, Health). The old character-builder page was replaced; Channels/Integrations/Automation remain the platform-wide configuration surfaces.

The studio never shells out to openclaw and never runs cat/find/stat:

SurfaceGateway methodTransport
Agent entries, bindings, channels, MCP, plugins, skills declarationsconfig.getAdmin HTTP RPC (WS fallback)
Roster / identity / workspaceagents.listAdmin HTTP RPC (WS fallback)
Bootstrap (AGENTS.md, SOUL.md, …)agents.files.list / .get / .setWS
Durable notes + dreaming filesagents.workspace.listWS
Tool catalogtools.catalogWS
Effective surface for a sessiontools.effective (sessionKey required)WS
Skills, workshop, trust, required binsskills.status, skills.workshop.list, skills.securityVerdicts, skills.binsWS
Hookshooks.statusWS
Plugins + runtime stateplugins.listWS
Sessions + store/WAL sizessessions.list, sessions.storage.statusWS
Jobs + run historycron.list, cron.status, cron.runsAdmin HTTP RPC (WS fallback)
Memory providermemory.statusWS
Node/process/disksystem.infoWS
Update / backupsupdate.status, backup.statusHTTP / WS
Activity (future)audit.activity.listWS

Endpoints (all ADMIN-only, mounted at /api/admin/agent-studio): GET / (instance + roster), GET /instance, GET /:agentId (?sessionKey= selects the effective session), GET|PUT /:agentId/files/:name, GET /:agentId/automations, POST /:agentId/automations/:jobId/:action, PUT|DELETE /:agentId/bindings, POST /plugins/:pluginId/:action.

The dedicated automations read uses the same job normalization as the agent detail: schedule is always display text, never the Gateway’s structured { kind, everyMs, anchorMs } object. This prevents React error #31 when opening the Automations pane. The shared gateway status indicator queries the ROOT-only /api/admin/debug-status endpoint only for ROOT users; ADMIN users retain the health indicator without an unauthorized debug-status request.

Health lists plugins in pages of 10 with Previous/Next controls and a total count. Pagination is client-side (no additional RPC reads), resets when switching agents, and hides its controls when the list fits on one page.

Capabilities uses independent 10-item pages for Allowed, Denied, Subagents, Skills, Available tools, Effective surface, and discovery Notices. Tool pages count individual tools across groups while preserving their group headings. Filtering applies before pagination and resets the skills/catalog/effective pages; changing agents resets all capability pages. No extra RPC reads are made.

Reads are non-waking. The route checks isRuntimeActive() first and returns { available: false, reason: "sleeping" } before opening a socket, so a sleeping tenant is never started just to paint the dashboard. Every endpoint opens one withGatewayRpcClient connection for its WS methods and guards each call independently, so one unsupported method only removes its pane (e.g. skills: { known: false }) instead of failing the page. Core reads use the admin HTTP RPC when available and fall back to config.get/agents.list over WS.

Writes: bootstrap files use agents.files.set with the hash from agents.files.get (409 agent_file_conflict on a stale hash); bindings are a config.patch on the top-level bindings array (no openclaw agents bind); automation run/enable/disable use cron.run/cron.update; plugin enable/disable/refresh use plugins.setEnabled/plugins.refresh with operator.admin.

RPC gaps (deliberately rendered as unknown, never faked): there is no MCP probe/reload method, so MCP health is the last-known tools.effective inventory plus its discovery notices; there is no binary-presence method, so the Health pane lists skills.bins required binaries and per-skill missing requirements rather than claiming presence.

Tests: src/agents/studio.test.ts (pure normalizers/builders), src/routes/api/agent-studio.test.ts (WS/HTTP method map, per-call degradation, sleeping short-circuit, secret redaction, file conflict, bindings via config.patch), src/client/pages/__tests__/role-isolation.test.tsx (BASE_USER/ADMIN), and test/ui/agent-studio.spec.ts (roster, tabs, file editor, sleeping state).

Realtime device-pairing notifications (/api/admin/devices/ws)

The dashboard NotificationBell subscribes to GET /api/admin/devices/ws:

  • On connect it pushes a { type: "device.pair.snapshot", pending, paired } frame (best-effort, non-blocking), then relays Gateway device.pair.requested / device.pair.resolved / device.pair.changed events as { type: "device.pair.event", event, payload }.
  • The Gateway subscription uses openGatewayEventStream (the official @openclaw/gateway-client browser client over sandbox.wsConnect), which reconnects with backoff until the browser socket closes.
  • Worker-hosted notification sockets rotate after 10 minutes with close code 1012; this is intentionally below the observed Cloudflare load-shed lifetime for long-lived Worker invocations. The dashboard reconnects and receives a fresh snapshot, releasing the old Worker invocation and Gateway subscription.
  • Admins approve/reject directly from the bell: POST /api/admin/devices/:requestId/approve / .../reject (RPC-first device.pair.approve / device.pair.reject, CLI fallback). The client helper is subscribeDeviceNotifications in src/client/api.ts.

Dashboard notification centre. NotificationBell also renders application notifications (backup created/failed, backup restored/failed) from the NotificationProvider context (src/client/NotificationContext.tsx) that wraps the admin layout. Long-running admin actions push their result there instead of blocking on a modal, so the outcome still reaches the admin if they navigate away before the request settles. The bell badge counts unread app notifications plus pending device-pairing requests; opening the panel marks app notifications read and each can be dismissed.

Persistence-lock indicator. PersistenceLockIndicator (rendered beside the bell) shows a closed red lock while a dashboard-initiated backup or restore is in flight, and an open green lock otherwise. The state comes from PersistenceLockContext (src/client/PersistenceLockContext.tsx): the BackupHistoryPanel raises it (begin('backup' | 'restore')) when it fires an async snapshot/restore and drops it (end()) when its completion check settles (success, failure, or timeout), so the indicator is synchronous with the operation lifecycle — there is no polling and no lease read. A counter keeps the lock held if two operations overlap. The widget is ADMIN/ROOT only (a BASE_USER cannot trigger a backup).

OTel Ingest (sandbox bridge http://otel.internal, shared-secret auth)

There is no public /api/otel route. The container’s OpenTelemetry exporter targets http://otel.internal/v1/{traces,metrics}, a virtual host resolved by the Sandbox SDK inside this Worker (TenantSandbox.outboundByHost, alongside docs.r2.internal / releases.r2.internal). The traffic never leaves the sandbox ↔ Worker control plane, so Cloudflare Access never interposes a login redirect and no path has to be published.

The per-tenant OTEL_SHARED_SECRET is still verified (constant time) by the handler: the bridge is only resolvable for this tenant’s sandbox, but a leaked handler must not become an open sink. buildEnvVars always sets the two endpoints and the bearer header; WORKER_URL is irrelevant to the exporter now.

  • POST /api/otel/v1/metrics — OTLP/HTTP JSON metrics (cost/tokens). AI cost rows keep openclaw.provider + openclaw.model so lerma can attribute the spend (unified-billing gateway-routed vs. direct provider / OTel-only).
  • POST /api/otel/v1/traces — OTLP/HTTP JSON traces ({resourceSpans:[...]}). Only openclaw.message.processed, openclaw.harness.run, openclaw.tool.execution spans are approved; others are counted as ignored. Each approved span is classified (src/billing-classification.ts) into a semantic billing action (telegram_query, signal_analysis, email_draft, …) — an explicit openclaw.billing.action_type marker wins, span attributes are the fallback. Approved-but-unmapped spans are accepted and counted as nonBillable. Stable action_id = lowercase(traceId) + lowercase(spanId) (no separator); action_key dedupes a business unit and trace_id drives precedence (email_draft/review_analysis suppress telegram_query). Tenant is required per span (customer.slug et al.; one batch = one tenant). application/x-protobuf is rejected with 415 (use http/json). See docs/contracts/otel-trace-action-id.md. Normalized traces are held in-memory for downstream reconcile; Analytics Engine stays source of truth.
  • One identity authority for both paths. /v1/metrics and /v1/traces resolve the caller through the same resolveCostIdentity(env) (src/sandbox-tenant.ts): the explicit trusted CUSTOMER_SLUG (or the WORKER_NAME it mirrors) first, then the routing resolver so a legacy worker that only carries a routable WORKER_URL is still attributed to its own tenant instead of the default sentinel. A billable instance whose identity cannot be resolved fails closed with 503 TENANT_UNRESOLVED on both paths, and a metric that names a foreign customer through any of customer.slug / customer_id / customer is rejected with 403. The cost summary read side uses the same resolver, so the dashboard reads exactly the identity the write path emitted. Emission guard: isBillableEventsEnabled(env) (src/dev-mode.ts) decides whether an instance may mint billable actions. Billable production always emits and cannot be opted out, and the staging platform always emits in every environment so the billing pipeline runs with production fidelity before release (its rows land in the staging Analytics Engine dataset). A dev environment is non-emitting by default — it accepts spans (200) and the cron drain drops buffered traces — and emits only when BILLABLE_EVENTS=true opts it in (e.g. to exercise billing end-to-end from a dev instance). The override is emission-only — getCostContext still labels rows with the real platform/environment (staging/dev stay non-billable cost axes), and shouldAutoSleep is unchanged, so an opted-in dev instance keeps its idle timeout and staging keeps its 15m one.

Browser Rendering (private http://cdp.internal bridge)

The agent browser is served by Cloudflare Browser Rendering through the Worker’s BROWSER binding. It is deliberately not a public /cdp dependency for the container: on a tenant hostname the host Cloudflare Access application answers /cdp with a 302 login redirect, and adding a path-scoped bypass is not acceptable. The public /cdp shim remains for direct/host-side CDP clients and the cloud E2E worker (workers.dev), not for the tenant.

The private path has three parts:

  1. Container proxy (apps/fouria/cdp-proxy.mjs, started by start-openclaw.sh in launch_gateway_process). It listens on 127.0.0.1:9223, serves the small CDP HTTP surface (/json/version, /json/list, /json/new, /json/close/:id) and a WebSocket command channel, and forwards each command to http://cdp.internal/command. It is detached with setsid and closes fd 9 so it never holds the gateway startup lock.
  2. Worker bridge (src/sandbox-cdp-bridge.ts, registered in TenantSandbox.outboundByHost). POST /command runs the CDP command through the tenant’s Sandbox Durable Object (cdpCommand), DELETE /session closes the browser, and GET /health is a liveness probe. The Sandbox SDK resolves cdp.internal inside the Worker, so the hop never reaches the internet and Cloudflare Access never sees it.
  3. Session ownership (src/cdp-session.ts + TenantSandbox). The browser is launched lazily from this.env.BROWSER and kept in the DO across the proxy’s per-action WebSockets, serialized by a command queue. The Browser Rendering credential never enters the container. The CDP handlers are the same handleCDPMethod used by /cdp, so the two transports cannot drift.

Moltlazy writes the matching remote-CDP profile (fill-only, user-editable): browser.enabled=true, browser.defaultProfile="cloudflare", browser.profiles.cloudflare with cdpUrl="http://127.0.0.1:9223" and attachOnly=true (a loopback cdpUrl without attachOnly is treated as a local managed Chrome), and tools.alsoAllow: ["browser"] (the coding profile excludes the browser tool). A restored boot re-applies --only gateway,browser so the profile self-heals from an older snapshot.

Known limit: OpenClaw’s Playwright-backed snapshot/evaluate paths expect CDP session multiplexing (Target.setDiscoverTargets/setAutoAttach, sessionId routing) that the raw shim does not implement yet; raw open, tabs, navigation and screenshots work. The E2E covers registration, open, and cross-call tabs.

Cost-tracking dimensions & development expense

Every Analytics Engine cost-tracking row (ai-query, container-active, r2-operation, cron-trigger, ws-session, action-completed, action-step, health-check) carries two trailing blobs — platform (production | staging) and environment (production | dev) — resolved by getCostContext(env) (src/metrics.ts → resolveInstanceDimensions in src/dev-mode.ts). The canonical customer key is the base client slug (CUSTOMER_SLUG, else tenant-{slug}[-dev] with the prefix/suffix stripped), so worker-side rows join against the client.slug lerma reconciles. health-check rows (emitted from non-passive GET /api/health) feed lerma’s uptime/downtime statistics. Dev/staging rows are non-billable development expense — infra + AI cost of building for a client, tracked in AE and pulled asynchronously by lerma, never charged to the client’s credit pool.


Software Provisioning (declared, re-applied every reprovision)

Software that agent skills install at runtime (a plugin, an apt/npm package, a downloaded binary) lives only in the container filesystem, so it is lost when the image is upgraded or the container is destroyed on idle-sleep. The softwareProvisioning feature flag makes that software declarative:

StepWhere
Declarationcustomers/<slug>/software.json in BACKUP_BUCKET (PUT /api/admin/software/manifest), { id, check, install, verify?, timeoutSec? } per package
MaterializationmaterializeSoftwareManifest() → /home/openclaw/.openclaw/software.json (inside the backup root, so a warm restore also carries it)
Reconcilemoltlazy software provision (packages/moltlazy/src/software/): check → skip, else install → optional verify; per-package timeout, non-fatal, exit 0
Boot triggerensureGatewayImpl dispatches the CLI detached (void spawnSoftwareProvision(...)) just before start-openclaw.sh start, so it runs in parallel and never blocks readiness
On-demand triggerPOST /api/admin/software/provision (ADMIN/ROOT) runs the same CLI to completion, bounded by SOFTWARE_PROVISION_RUN_TIMEOUT_MS (5 min)

It applies on every provisioning cycle — cold start, restored boot, idle wake, image upgrade, and /debug/restart-openclaw — because it lives in the Worker’s ensureGatewayImpl rather than in the (skippable) setup body of start-openclaw.sh. A soft reload (SIGUSR2) does not re-run it; nothing is lost there.

Outputs: /tmp/software-report.json (structured; read by GET /api/admin/software/report) and /tmp/software-progress.log (ticks; read by /api/startup-progress as softwareLog). The CLI truncates the progress log at the start of every run because /tmp survives an in-place restart, so otherwise ticks from earlier boots would be interleaved.

Operational notes:

  • Both GET endpoints return enabled so the dashboard can distinguish “flag off”, “instance asleep”, “not run yet” and “nothing declared”; the homepage widget’s Run provisioning button (ADMIN/ROOT) calls the on-demand endpoint and renders the fresh report.
  • A failed package stays failed in the report and does not abort the rest of the manifest or the boot. A report is only rewritten when the CLI actually runs, so a stale report can exist inside one container after the flag is switched off — the enabled: false marker is what the UI keys on.
  • The manifest is arbitrary shell executed in the tenant container. The upload and on-demand routes are ADMIN/ROOT-only; there is no command allowlist by design (the container is single-tenant).

Container Startup Flow (start-openclaw.sh)

start-openclaw.sh takes a phase argument ($1):

ModeBehaviour
startProduction cold-start (single entry point): acquire the container-wide startup lock first, then run the full setup body, then become the single OpenClaw owner — execing the gateway while still holding the lock.
setupStandalone/smoke only: run everything up to (not including) the gateway launch, then exit.
launchStandalone/smoke only: acquire the container-wide launch lock and become the single OpenClaw owner (or wait for the current owner to become healthy), then exec the gateway.
post-startAfter the gateway is ready — dev automation + boot-log reset, then exit. Dispatched detached (fire-and-forget) by the Worker so it never extends the shared start promise.
full (default)Manual/back-compat + boot smoke: run setup, then launch + poll the gateway in the background, then exit.

The Worker’s ensureGateway drives production cold-start as a single /usr/local/bin/start-openclaw.sh start call followed by post-start. Unlike the standalone phases, start acquires the container-wide startup lock before running setup and holds it through the Gateway exec, so setup and launch are serialized under the same owner — the Worker no longer invokes setup and launch separately. The gateway is not a shell background job: the start owner execs openclaw gateway run while holding the lock. The lock is held for the Gateway process lifetime via the inherited fd (released exactly when the gateway exits), so the supervised process is the gateway — tracked by listProcesses/getProcess for the life of the container and killable via its handle. A concurrent invocation that loses the lock polls /health and exits 0 without starting a duplicate owner. The serial test/e2e/ac_gateway_startup_concurrency.txt suite checks that exactly one owner exists and that neither the migration-input-change error nor the SQLite offline-maintenance error appears. The standalone setup/launch modes (used only by smoke/manual runs) keep the older split behaviour — a setup flock that serializes the slow setup body and is released before a separate launch — which is why ownership there is enforced solely by the dedicated launch lock. full keeps the legacy background-launch behaviour for manual runs and test:gateway:boot.

Restored containers skip the setup body (zero-data-loss). The R2 restore happens in the Worker (restoreIfNeeded → sandbox.restoreBackup()), before start-openclaw.sh runs — the script never performs the restore. When the launch runs against a container that already holds tenant state, ensureGateway passes OPENCLAW_RESTORED=true in the launch env (see hasRestoredTenantState), and the script skips the entire setup body — onboard → plugin installs → moltlazy patch → config validate — emitting those phases as skipped and launching the gateway directly from the restored filesystem. The restored state is therefore authoritative: the cold-start budget is spent launching it, not re-mutating it (which could diverge from or clobber the snapshot, and which wastes startup time). A brand-new tenant (no state, no backup) still runs the full setup body and onboards normally; the image ships no openclaw.json, so “state present” is exactly “already initialized/restored”. Each boot rewrites a durable receipt at /tmp/openclaw-restored-setup-skipped (written on the skip path, removed on the setup path) so an operator can confirm which path ran; the local backup E2E (test/e2e-backup/zzz_backup_restore_skip_setup.txt, opt-in via E2E_BACKUP_RESTORE=1) asserts the receipt and that the restored setup state is left byte-for-byte intact.

OPENCLAW_RESTORED is decided by hasRestoredTenantState(), which combines the in-memory restore status (getRestoreStatus() — hadBackup && restored, set by restoreIfNeeded) with the container-state probe (containerHasState — the restored openclaw.json, legacy /root or v1 /home layout). The in-memory result alone is not sufficient, because the launching isolate is frequently not the one that restored:

  • the durable boot job is two steps (restore, then launch in a later alarm), so a DO eviction in between loses _restoreState;
  • any request that reaches the proxy while the restored container has not been launched yet (the loading page polls, the browser keeps its tab open, an API call arrives) calls ensureGateway from its own isolate, where restoreIfNeeded only runs the cheap request-scoped check and never records a restore status;
  • clearPersistenceCache() (freeze / resume / stale-container recovery) resets it too.

Without the durable fallback those launches passed no OPENCLAW_RESTORED, so start-openclaw.sh re-ran the whole setup body — re-onboarding and re-installing plugins over the just-restored snapshot (observed on restored staging tenants after a rollout/reprovision). The restore decision is deliberately not keyed on the durable R2 restore-needed latch, which is a separate cross-isolate retry/fence (written before sandbox.restoreBackup(), cleared only after restore

  • legacy-state materialization succeed): its absence is normal while idle and after a successful restore, and the restore itself is triggered by the container-state probe plus the committed recovery point (stable-backup.json). The loading page’s “Restoring…” spinner observes the in-memory restore status the same way, so it appears the moment a cold restore is in flight regardless of whether the marker was present in R2.

One targeted exception: the restored boot re-applies only the gateway section. The Worker-critical gateway config (trustedProxies, auth, controlUi.allowedOrigins) must self-heal even on a restored boot: a warm-restored openclaw.json can carry a stale gateway.trustedProxies (e.g. the pre-2026-09 default ['127.0.0.1','::1']) that omits the Cloudflare Sandbox bridge range (10.1.0.x) the Worker proxy connects from, so OpenClaw rejects every proxied request with proxy_attribution_required (symptom: the tenant serves the gateway’s JSON error {"error":{"type":"proxy_attribution_required"}} on the root HTML/WS proxy). The skip path therefore runs moltlazy patch --only gateway --file "$CONFIG_FILE" (bounded, non-fatal): it heals trustedProxies (drop catch-alls, union the required ranges — see healTrustedProxies) and re-asserts auth/origins, but never regenerates the user-editable agents/session/tools/logging sections nor re-seeds workspaces, so the snapshot’s user data is untouched (the backup E2E fingerprints moltlazy-session.json + a sentinel, both unaffected).

The skip is guarded on the restored config actually existing. A “restored” boot can carry partial container state but no materialized openclaw.json — an interrupted restore (a container-image rollout changes the runtime mid-restore), or restoreIfNeeded’s “container already holds tenant state; skipping duplicate restore” short-circuit, which skips the legacy-state materialization yet still reports restored=true and therefore sets OPENCLAW_RESTORED=true. Skipping the setup body then launches OpenClaw with no config at all (observed on tenant-tarjetero-dev after the v2026.10.13 rollout). start-openclaw.sh therefore only honours OPENCLAW_RESTORED=true when $CONFIG_FILE exists; otherwise it logs a warning and falls through to the setup body, which onboards/repairs the config instead of launching a configless gateway.

The cold boot is a durable ActivityScheduler DO job, not a request waitUntil(). Cloudflare cancels a request’s background (waitUntil) work a short time after the response, so a cold boot driven from a request was killed mid-restore and re-attempted forever: [Gateway] Gateway start already in progress … (phase=restore, age=686365ms) with start-openclaw.sh never run, 141 waitUntil() tasks did not complete warnings, 14× re-decrypt of the same archive, and 64 probe processes hammering the Sandbox DO (observed on tenant-client-dev). The boot is now a durable job (src/activity/boot-job.ts) on the ActivityScheduler DO: POST /api/warmup, non-passive GET /api/health and /api/startup/ws only request it (requestBoot, deduped) and return; the alarm performs restoreIfNeeded → ensureGateway and retries on a bounded lease independent of any request. When the ACTIVITY binding is absent (local dev / unit tests) the routes fall back to the request-scoped waitUntil boot.

A request never owns a destructive restore. Even with the durable boot job, every ensureGateway call on a cold path re-asserts the restore barrier. A request that would actually need an archive restore now defers it: restoreIfNeeded(..., { restoreOwner: 'request' }) (the default for ensureGateway) performs only the cheap non-destructive checks — the restore-needed latch and containerHasState() — returns immediately when the container already holds tenant state, and otherwise throws RestoreDeferredToBootJobError. ensureGateway catches it, calls requestDurableBoot(), and throws GatewayBootPendingError, so the proxy/route answers 503 (or the loading page) while the DO alarm performs the restore + launch. Durable callers (the ActivityScheduler boot/wake jobs and the queue consumer) pass restoreOwner: 'durable' and run it inline. The proxy no longer calls restoreIfNeeded itself. Motivation: on tenant-tarjetero-dev a 350 MB encrypted archive was re-decrypted and sandbox.restoreBackup aborted at ~30.1 s (backup.restore error … The operation was aborted) on every cold request, because the restore ran under a request execution context that Cloudflare cancels shortly after the response — a reboot/restore loop while the tenant was actively used.

Three supporting bounds keep a stalled boot from hiding behind the long restore phase: the SDK sandbox.restoreBackup RPC is bounded by RESTORE_BACKUP_TIMEOUT_MS (fail fast on a hung control channel); the restore reports sub-phase progress (onProgress) so isGatewayStartStuck judges the restore phase on GATEWAY_RESTORE_NO_PROGRESS_MS of no progress rather than the full 11-min budget; and the persistence withLease releases the shared R2 lease when its enclosing request aborts (PersistenceDiagnosticContext.signal), so a cancelled operation cannot leave restore-needed + a held backup-lock re-triggering the loop.

Snapshot lease renewal and admin snapshot mode. The shared R2 backup-lock TTL is STEP_DEADLINE_MS (12 minutes), with a heartbeat every quarter-TTL (3 minutes). Every acquire/CAS write is bounded by LOCK_WRITE_TIMEOUT_MS (10 s) and a timeout is reported as persistence.lock.write-timeout + a lost acquire: a conditional put that never settles would otherwise hold the whole durable step (6 min of silence, then job-step.interrupted) instead of failing the acquire. This was observed in local dev, where a takeover of an expired lease wrote the object server-side while the promise never resolved — the operator saw only Restore — Waiting for the backup lock for the entire step. Snapshot work explicitly renews before/after legacy staging, sandbox.createBackup, and encryption, and before final config capture; restore renews around decrypt and archive application. Renewals are serialized on one promise chain, awaited before the release tombstone, and best-effort (a failed renewal is logged but does not abort the snapshot). Structured events report renewed, late (at least half a TTL), and failed renewals. A 12-minute TTL trades a longer stale-holder recovery window for avoiding false lease expiry during a legitimate long step; lock waiters can still hit their bounded acquisition timeout.

POST /api/admin/storage/sync uses the durable backup job by default when the ACTIVITY binding is configured (202 with jobId, 409 if another backup/restore owns the slot). ?inline=1 explicitly requests synchronous completion; ?async=1 keeps the accepted-then-background behavior. Environments without ACTIVITY retain the inline fallback. E2E scenarios that need an immediate backup id must use ?inline=1.

Never call output()/runShell on the live gateway handle. The gateway never exits, so the SDK’s buffered output() only opens an RPC log stream, waits for the local timeout, and disconnects it, surfacing workerd’s ReadableStream received over RPC disconnected prematurely. Read diagnostics from the /tmp/gateway-stdout.log / /tmp/startup.log tails (readLogTail) instead.

The setup/post-start phases must stay child-free. The Worker waits on them with waitForExit, which — despite its name — only settles once the supervised terminal exits, which never happens while the script leaves a live background child. setup is awaited on the critical path; post-start is dispatched detached but still waited with a bounded waitForExit (a lingering child would keep the post-start process alive until the GATEWAY_POST_START_TIMEOUT_MS kill). So neither phase may start a tail -f log mirror: the mirror is started by the native launch (buildGatewayRunScript). A lingering child in setup stalls cold start for the full STARTUP_TIMEOUT_MS and makes the Worker re-run setup on retry (observed: ~11 min to first gateway launch).

post-start never blocks readiness. Once the port wait succeeds and markGatewayReady runs, the gateway is serving; ensureGatewayImpl dispatches start-openclaw.sh post-start with void runGatewayPostStart(...) instead of awaiting it. Dev instances run openclaw automations list/create there (30–60s), and awaiting it previously held the shared start promise so every request logged Gateway start already in progress … waiting... while /health already returned 200. startShell dispatches the sandbox.exec synchronously before its first await, so the work still runs server-side even if the isolate is evicted; runGatewayPostStart swallows its own errors (best-effort).

The entrypoint runs sequentially:

1. Fast-path check: Is gateway already running? → exit 0 (skipped for `post-start`)

2. Serialize concurrent startup/setup/full invocations (exclusive `flock` on
   /tmp/openclaw-setup.lock, ~900s wait). The Worker's in-process
   `gatewayStartPromises` guard is per-isolate, so bursty cold-start polling can
   land several runs in different isolates at once; without the lock they
   all run the slow setup body concurrently, contend on OpenClaw's native
   config/state locks, and stall cold start for many minutes (observed: ~18 min
   to gateway-ready in run 36692775633). Only one body runs; a waiter re-checks
   the gateway fast-path after acquiring and exits 0 once the winner is up.
   In production, `start` takes this lock **before** the setup body and holds it
   through the Gateway `exec`; `post-start` is outside the lock (it runs after
   the gateway is reachable). The standalone `launch` mode has its own dedicated
   launch lock and must not double-lock. `full` releases fd 9 before
   backgrounding the gateway so the child does not inherit the setup lock.

3. Worker-side lifecycle coordination
   - Restore, snapshot, restart, and destroy are coordinated by the Worker.
   - OpenClaw's native lock protects gateway ownership; the startup `flock`
     above serializes onboarding/config/plugin filesystem mutations **and** the
     launch across concurrent Worker isolates — `start` holds it from before
     setup until the gateway is `exec`'d, so a single owner covers both. The
     standalone `setup`/`launch` smoke modes still split the lock (setup lock
     released before the separate launch lock), so within that path single-owner
     launch is enforced by the dedicated launch lock alone.

4. Pre-backup optimization:
   - Move logs to /tmp (excluded from squashfs snapshots)
   - Clean stale backup files

5. Onboard (if needed):
    - Detect Cloudflare AI Gateway auth from env vars
    - Ensure the `cloudflare-ai-gateway` provider plugin is present before
      onboard. It is **pre-installed into the image at build time** (see
      Dockerfile), so this is normally a no-op detection; the install is a
      resilience fallback. Non-interactive onboarding cannot approve plugin
      capabilities, so without the plugin its plugin phase fails and burns the
      60s onboard cap on every cold start. With it present, the phase is a no-op
      and onboarding completes cleanly.
    - Run: openclaw onboard --non-interactive --accept-risk

6. Enable plugins (single working `config patch`):
   - All three plugins are **pre-installed into the image at build time**
     (Dockerfile, unpinned → latest ClawHub release). The installs below run
     only as a fallback when the extension directory is missing (e.g. a warm R2
     restore replaced state). Detection (`plugin_extension_present`) checks that
     directory — the artifact the gateway actually loads — and deliberately does
     **not** read `openclaw plugins list --json`: the registry refresh hangs for
     its full `CLI_TIMEOUT` when ClawHub/npm are slow, and a hung/partial list
     reads as "not installed" and triggers a needless network install (~57s
     ai-gateway + ~74s otel on a full cold boot on `staging-client-dev`). See
     `docs/plans/cold-start-latency.md`.
   - **Build/boot guard:** every ClawHub install is best-effort. A ClawHub
     outage, a transient network error, or a plugin that requires a newer plugin
     API than the pinned OpenClaw runtime (observed: `@openclaw/cloudflare-ai-gateway-provider`
     resolving to 2026.9.9 against the 2026.9.8 runtime) logs a `WARNING` and
     continues — the Docker build no longer fails, and the boot proceeds. The
     enablement patch below only enables a plugin whose extension directory is
     present, so a failed install cannot invalidate the whole patch.
   - cloudflare-unified-billing (when installed) — the Paso4 plugin, served from
     https://clawhub.ai/paso4/plugins/cloudflare-unified-billing. Entry opts
     into `hooks.allowConversationAccess=true` so the
     `agent_end`/`message_received` hooks run (OpenClaw 2026.8.2 blocks
     non-bundled conversation hooks without it)
   - cloudflare-ai-gateway (when installed)
   - admin-http-rpc (always) — the Worker's RPC-first admin read path
   - diagnostics-otel (enabled only when `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`
     is set and not in E2E mode; otherwise explicitly disabled). Exporter
     protocol is always `http/json`. When `WORKER_URL` is set (and
     not in E2E mode) `buildEnvVars()` also derives
     `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT={WORKER_URL}/api/otel/v1/traces`.
     `diagnostics.otel.traces` is opt-in via `OTEL_TRACES_ENABLED=true`
     (default: disabled); the config shape is unit-tested in
     `src/gateway/diagnostics-otel.ts` (`buildDiagnosticsOtelConfig`).
   - The patch must NOT carry `plugins.bundledDiscovery` (removed in OpenClaw
     2026.9.x — including it fails schema validation and drops the whole patch)
     nor `plugins.allow` (an allowlist disables every bundled provider not
     listed). Memory search is disabled via `memory.search.enabled=false`.
   - **memory-core dreaming cadence (dev):** the same patch slows the dreaming
     sweep to weekly (`plugins.entries.memory-core.config.dreaming.frequency =
     "0 3 * * 1"`, default `0 3 * * *`) when `OPENCLAW_DEV_MODE=true`. The full
     sweep is a heavy isolated agent turn (~6 min, `toolsAllow: ["*"]`) that the
     cron replays on every wake (`cron: running missed jobs after restart`) — a
     major RSS spike source on the 1-vCPU dev sandbox (observed 1.6 → 2.08 GiB
     during a replayed sweep). Production keeps the OpenClaw default cadence.
   - **Version drift:** the plugin-API mismatch above means a build on the
     pinned runtime ships **without** the Cloudflare provider plugins until
     OpenClaw is bumped (see the openclaw-migration skill) or the specs are
     pinned. The guard keeps the image and boot healthy in the meantime; it does
     not restore the missing provider by itself.

7. Patch config via moltlazy:
   - Run: node /app/moltlazy/dist/cli.js patch
   - Generates per-section includes (moltlazy-session/tools/logging.json) injected
     via per-section $include; gateway/agents written inline (no root $include)
   - References the gateway token as an env SecretRef (kept in the container environment)
   - Run: node /app/moltlazy/dist/cli.js validate
   - **Validation is FATAL**: an invalid config would prevent the gateway from
     booting, so the script aborts (exit 1) instead of proceeding.

8. Harden file permissions:
   - chmod 600 moltlazy-*.json, chmod 700 .openclaw

9. Start gateway:
    - `start` mode (production): after setup, `exec` the gateway under the same
      startup lock the phase acquired at step 2 — the Worker no longer calls
      `launch` separately (its stdout log mirror / `post-start` run as usual)
    - `setup` mode: smoke/manual-only standalone path — `mark_step gateway`,
      then exit 0 without launching the Gateway. The production Worker uses the
      atomic `start-openclaw.sh start`, which holds the startup lock and `exec`s
      the Gateway, followed by `post-start`
    - `full` mode: background `openclaw gateway run --port 18789 --verbose
      --allow-unconfigured --bind lan`, poll `/health`, then exit while the
      gateway child remains detached

Startup Progress (/api/startup-progress)

The loading page shows a transparent boot stepper + live log while the gateway starts. start-openclaw.sh declares its boot phases in a single STEPS array (single source of truth) and emits progress to two /tmp files on each phase (kept in /tmp so they stay out of squashfs backups and R2):

FileContent
/tmp/startup-progress.json{ steps:[{id,label}], currentStep, currentStepId, totalSteps, skipped, ts }
/tmp/startup.logClean, secret-free, chronological boot detail (verbosity-gated)

mark_step <id> [skipped] resolves the label/index from STEPS, writes the progress JSON atomically (mktemp + mv), and echoes the transition to the container stdout. log_boot <level> <msg> writes clean boot detail to /tmp/startup.log. Step transitions are intentionally not written to the log — the loading page’s stepper shows step progress, so the log stays clean (no duplicated step/checked info). Adding a phase = one STEPS row + one mark_step call — the loading page renders the list dynamically, so no other code changes.

GET /api/startup-progress (public, no warm-up) reads both files and returns { available, steps, currentStep, currentStepId, totalSteps, skipped, ts, log }. It is bounded by STARTUP_PROGRESS_TIMEOUT_MS and never calls restoreIfNeeded/ensureGateway, so it adds no load to the health path.

The loading page consumes this data over the GET /api/startup/ws WebSocket (not by polling). Each push is a combined { ...HealthSnapshot, progress, boot, jobLog, restoreProgress?, recovery?, repair? } payload — the health snapshot (which drives readiness/redirect) plus the boot steps + log, the durable boot job, and the formatted job-status log. On socket open the endpoint registers one ensureGateway warm-up with waitUntil; every subsequent health snapshot is passive/read-only. It does not re-enter restoreIfNeeded on every push. This distinction matters when restore is blocked on an R2 backup-lock: the former per-push warm-up queued a new 120-second restore-lock waiter every 5–30 seconds, which could starve gateway startup and the pre-replacement checkpoint while the UI kept its socket open.

The loading page shows the boot log panel while the gateway is starting up — both while start-openclaw.sh is running its setup body (processType === 'script') and while the gateway binary is binding its port (processType === 'gateway'). The processType === 'gateway' state no longer hides the log: start-openclaw.sh start execs the gateway and a start_boot_log_stream mirror tails the gateway’s own stdout (/tmp/gateway-stdout.log) into the curated boot log, so the log keeps filling with real launch progress (SQLite migration, plugin load, port bind) instead of going blank for the minutes the gateway takes to bind — the exact “Launching the gateway — no progress” window that previously looked stuck. The mirror is killed (stop_boot_log_stream) and the buffer cleared once the gateway is ready.

The terminal never depends on /tmp/startup.log alone. That file does not exist during the durable boot job’s restore phase (no start-openclaw.sh process yet) and can stay quiet while the gateway binds its port, so the panel used to sit blank next to a slow-step notice like Starting OpenClaw Gateway (188s). The Worker now formats the durable state it already reads into a jobLog string (activity/job-status-log.ts, pure and unit-tested):

LineSource
[HH:MM:SS] Boot queued / running / retrying / failed — …ActivityScheduler.getBootJob() (step, attempt, elapsed, next attempt, reason)
[HH:MM:SS] Restore — <phase label> (<percent>%)getRestoreProgress() (idle annotation and terminal failure included)
[HH:MM:SS] Repair running — openclaw doctor --fix, attempt N/3getActiveMaintenanceJob()

/api/startup/ws pushes jobLog on every frame, and the non-passive not-yet-ready /api/health / /api/health-snapshot carries the same fields for the page’s HTTP polling fallback. loading.html renders the container’s own log when it has content and otherwise the formatted job status, labelling the terminal source (fouria@gateway: /tmp/startup.log vs boot job status). The panel is therefore visible for the whole restore phase instead of being replaced by the brand logo, and the brand logo only appears when there is nothing at all to show (no process and no durable job state).

While a cold restore is in flight, the page also shows a restoring spinner beside the status line (title “Restoring FourIA” / status “Restoring data from backup…”). It is driven by the health snapshot’s in-memory restoreStatus (hadBackup && !restored) — the same signal the startup.ok probe uses to report “Backup found but not restored” — or the durable recovery.restoring marker (below), and is cleared the moment the restore commits (or the start fails/redirects). It is intentionally not keyed on the R2 restore-needed latch, so the spinner shows for a restore that would have run anyway (a fresh container with a committed recovery point).

The page also names the backup + date being loaded, so a cold restore is legible instead of a generic spinner. /api/health (non-passive, not-yet-ready) and /api/startup/ws attach a durable recovery object read from R2 (readRecoveryState): { restoring: { id, created } | null, committed: { id, created } | null }. restoring is the in-flight restore-target.json written by restoreIfNeeded/restoreBackupById just before sandbox.restoreBackup and cleared on success; committed is the stable-backup.json recovery point a cold boot restores. It is read from R2 (not the per-isolate restoreStatus) because the durable boot/restore runs in the ActivityScheduler DO, whose in-memory state is invisible to the request isolate serving the page. Lerma’s frequent passive poll never pays the two R2 reads; they are bounded by RECOVERY_STATE_TIMEOUT_MS and fail soft to null.

During a restore the loading UI is a progress bar synchronized with the DO’s own phase events (not a client-side timer). restoreIfNeeded / restoreBackupById call their onProgress(phase) hook on every persistence.phase; the DO wires that to recordRestoreProgress, which maps the phase to a label + a monotonic fraction (describeRestoreProgress, in activity/restore-progress.ts) and exposes it via getRestoreProgress(). /api/startup/ws pushes it as restoreProgress: { phase, label, fraction }; loading.html renders the bar and hides the stepper while a restore is in flight, then switches back to the container’s startup-progress.json stepper once the gateway launch begins. A request-scoped restore with no DO feed shows an indeterminate bar. /api/admin/boot-state also returns restoreProgress for operators.

The label stays honest when a restore is interrupted. A failed attempt is marked before the durable job rethrows (recordRestoreFailure), so the bar reads Restore attempt failed during "<phase>" — retrying instead of the last phase. A kill that skips that catch (e.g. a Worker CPU-limit kill) leaves the last durable phase in place, so getRestoreProgress() appends — no progress for N min once the record is idle for 2 minutes. The fraction never moves, so a stale bar is labelled rather than presented as live.

A terminal failure must not be presented as a retry. — retrying is only true while the durable job still has attempts left. When the boot job (or a manual restore job) exhausts them, the DO records describeRestoreTerminalFailure (recordRestoreTerminalFailure) with terminal: true: the label becomes Restore failed — <reason> (or Gateway launch failed — … for the launch step), describeRestoreProgressIdle stops appending an idle note (a finished failure is not stalled), and the next boot.start clears the flag. readRecoveryState likewise stops reporting a restore-target.json whose outcome is failed as restoring, and loading.html gives a failed boot precedence over the leftover progress bar and status line (Boot failed after N attempts: <reason>. Refresh the page to retry.). Without this, a tenant whose recovery point could not be applied sat on Restoring the backup — no progress for 10 min forever: the three failed attempts took ~70 s, and the page then rendered the stale record (with the restoring spinner, because any restore-target.json counted as restoring) instead of the terminal failure the job had already recorded.

Archive decryption (decryptStream / encryptStream in crypto/backup-encryption.ts) assembles each 4 MiB chunk from R2 reads with a byte queue that copies each byte once. The earlier concat-per-read buffer recopied the pending chunk on every ~64 KB read, which exhausted the Worker CPU budget on large recovery archives (Worker exceeded CPU time limit during Decrypting backup …). Measured in isolation on a 96 MiB archive: decrypt 1507 ms → 140 ms at 64 KB reads, and 5614 ms → 137 ms at 16 KB reads.

The R2 binding delivers the archive in 4 KiB reads (local workerd: 64 MiB arrives as 16,385 reads), so the old per-read concat was the dominant cost. Each restore logs [crypto] Decrypt read profile with the read count, bytes, largest and average read, and chunk count, so the read pattern can be checked on a live tenant.

The loading page uses a single push channel — GET /api/startup/ws — and does not poll while the WebSocket is open. Flow:

  1. Handshake — the first WS push reports whether the gateway is already running; if so the page redirects immediately, otherwise the stream keeps warming and pushing step updates.
  2. Boot phase — while boot steps are in progress, the sandbox pushes the health snapshot + progress (steps + log) every STARTUP_WS_POLL_INTERVAL_MS (~5s). No client polling. If a transient DO↔container transport drop makes the probe fail, the stream re-sends the last known state and backs off exponentially up to STARTUP_WS_MAX_BACKOFF_MS (30s) until the control session reconnects, so the page keeps waiting and the same error is not surfaced/re-logged every 5s.
  3. Fallback — if the WebSocket drops (e.g. local dev server, a network blip), the page falls back to slow HTTP polling (/api/health every 5s) indefinitely. The page has no wall-clock budget or retry cap of its own: the cold boot is driven out-of-band by the ActivityScheduler DO, which owns retries and terminal failure. A run of failed polls does not hard-fail the page (“Unable to reach server”): a container-image / Worker deploy briefly resets the DO↔container control session, so the page keeps retrying and shows “Reconnecting — the instance may be updating…”.

The page only declares failure on a permanent container error (degraded: true in the /api/health snapshot — deleted image, missing app, where a rescue is required), surfaced immediately via the container error, or on the durable boot job’s terminal boot.status === 'failed' state. Every other state — still booting, gateway reachable but warming up, restore in progress — keeps waiting, because a cold start legitimately takes 3–5+ minutes (onboard + gateway warmup; the provider plugins are pre-installed in the image, so plugin installs only run on the fallback path). Mixed probe states (e.g. startup.ok lagging behind readiness.ok while a backup restore is pending) are not treated as failures, and there is no client-side timeout that interrupts a long-but-progressing boot.

The loading page treats the canonical gateway status (status === 'running', driven by the readiness probe) as “ready” rather than trusting startup.ok alone, because startup.ok is gated on backup-restore state that can lag a healthy, already-serving gateway.

A slow boot step is surfaced as an alert. Once the current boot step has run longer than SLOW_STEP_MS (120s), the loading page shows a non-fatal notice naming the step and its elapsed time (e.g. “Still working on “Installing plugins” (185s)…“), driven by the script’s own progress.ts. It is not a failure — cold starts legitimately take minutes — and it is hidden on ready/error. It exists so an operator can tell a slow-but-progressing boot from a stuck one without opening logs or the Workers Observability console. (See docs/plans/cold-start-latency.md.)

The Worker’s root HTML route (app.all('*') in src/index.ts) uses the same ready predicate (snapshot.readiness.ok, i.e. status === 'running') — it must not additionally require startup.ok. Requiring both made the two disagree whenever readiness.ok was true but the startup probe lagged (a restore still settling, a stale cached startup result, or a start-openclaw.sh start wrapper not yet replaced by the gateway binary): the loading page redirected to / while the root route re-served the loading page — an infinite reload loop where OpenClaw never appeared to boot (observed on tenant-client-dev). Keep the two predicates in lockstep.

The gateway WebSocket proxy gates on the same readiness predicate. The catch-all WS path in src/index.ts must call isGatewayReadyForProxy() before sandbox.wsConnect(). wsConnect (and containerFetch) wait for their target port inside the Sandbox DO for containerTimeouts.portReadyTimeoutMS (CONTAINER_PORT_TIMEOUT_MS, 90s). During a container-image rollout the container is healthy — Cloudflare reports it active and the container logs are clean — but the gateway has not bound 18789 yet, so that one RPC parks for the full 90s and every other RPC (health, components, other proxies) queues behind it, surfacing as “Unable to reach server” on the loading page. When the gateway is not ready the WS path returns a fast, retriable 503 + Retry-After: 5 (and fires ensureGateway({ waitForReady: true }) via waitUntil so the cold start still progresses) instead of calling wsConnect. The HTML and non-HTML HTTP proxy paths already gate the same way.

A "Timed out waiting for runtime port" error — the DO’s port wait expiring because the container is up but the service has not begun listening — is classified by isRuntimePortTimeoutError() (src/gateway/sandbox-recovery.ts) as transient: the WS path returns a retriable 503, never a 502, and it must not trigger killGateway/stale-container recovery, which would discard a healthy container mid-rollout. It is deliberately excluded from isStaleContainerError.

Likewise, Network connection lost, WebSocket connection failed, and the Worker’s bounded Operation timed out after … from sandbox.wsConnect() are transient Sandbox DO WebSocket-transport failures, not proof that the gateway process died. The proxy returns 503 with Retry-After: 5 and leaves the container and R2 state untouched. This matters when the admin HTTP RPC health/status path still works: it uses containerFetch, a separate transport from the browser’s WebSocket connection.

Security invariant: the boot log contains curated log_boot lines — the script’s raw stdout (which can contain the AI Gateway API key as an openclaw onboard --cloudflare-ai-gateway-api-key … argument) must never be written to /tmp/startup.log. In addition, once the gateway launches, a start_boot_log_stream mirror tails the gateway’s own --verbose stdout (/tmp/gateway-stdout.log) into the boot log so the loading page can follow launch progress; this is the same output already surfaced to ADMIN/ROOT via /debug/gateway-logs and /api/admin/logs, and it is stopped (stop_boot_log_stream) and the buffer truncated at readiness. The gateway’s startup output is a distinct stream from the script’s onboard stdout (which is where the API key appears as a CLI argument and is still never captured).

Verbosity is controlled by STARTUP_LOG_VERBOSITY (steps | detailed (default) | verbose), passed through from the Worker via buildEnvVars().

moltlazy patch — Config Injection

moltlazy patch is the mandatory config system. It runs every startup and generates:

Config SectionTypeContent
gateway.*MandatoryPort, mode, bind, auth
session.*MandatorySession scope, DM settings
tools.toolSearchMandatoryTool search configuration
logging.*MandatoryLog levels, format, destinations
agents.defaultsAlways-onDefault model, system prompt
agents.entriesAlways-onPremade agents keyed by id (dev-only fouria-builder, fouria-qa)
skills.entriesAlways-onBuilt-in skills
memory.*Feature-gatedMemory backends (knowledge graph)
channels.*Env-gatedChannel tokens (Telegram, Discord, etc.)
plugins.*Feature-gatedPlugin management config

The moltlazy-owned sections (session/tools/logging) are generated into per-section include files and injected into openclaw.json via per-section $include directives — not a root $include (root includes fail closed for Control UI saves / config.apply). They are regenerated from scratch every startup — never edit manually. gateway.* and agents.* are written inline into openclaw.json; the gateway token is an env SecretRef (resolved from the container environment, which start-openclaw.sh keeps set).

Memory-pressure limits. Two moltlazy-owned defaults keep the session store from growing into the sandbox’s memory/disk: agents.defaults.compaction (maxActiveTranscriptBytes: '8mb', keepRecentTokens: 12000, midTurnPrecheck.enabled: true) and session.maintenance (maxDiskBytes: '512mb', highWaterBytes: '384mb', maxEntries: 500, pruneAfter: '14d', resetArchiveRetention: '14d', coldStorage: { enabled: true, afterDays: 7 }). OpenClaw’s own defaults (20 MB transcript, 10 GB per-agent disk budget, 5000 entries, 30-day retention) are sized for a developer machine and are too large for a 4–12 GB container. These limits bound growth but cannot reclaim worker-thread heaps that already grew; the hard restart is still the only RSS reset (see docs/plans/sandbox-memory-usage.md).

Admin RPC (admin-http-rpc) — RPC-first read path

The Worker reaches the gateway’s admin control plane over the gateway’s loopback HTTP listener using OpenClaw’s bundled admin-http-rpc plugin (POST /api/v1/admin/rpc). The typed client lives in packages/moltlazy/sdk (createClient, HTTP RpcClient); apps/fouria/src/gateway/admin-rpc.ts supplies the transport — a fetch-shaped adapter over sandbox.containerFetch (global fetch cannot resolve the container’s localhost).

This replaced the CLI-first read path for the hot dashboard routes. CLI invocations (startShell(sandbox, 'openclaw ...')) reap a fresh process and re-establish a WebSocket on every call (~10–15s); the HTTP RPC path is a single container request. Routes attempt RPC first and fall back to the CLI (runCliFallback / the existing parsers) whenever the transport is unavailable or returns an unexpected shape:

  • GET /api/admin/devices → device.pair.list
  • POST /api/admin/devices/:id/approve → device.pair.approve
  • GET /api/admin/models → models.list
  • GET /api/admin/channels/probe → channels.status { probe: true }

Enablement: start-openclaw.sh enables the plugin with a dedicated plugins.entries["admin-http-rpc"].enabled = true config patch. It is kept separate from the batched plugin patch because OpenClaw 2026.9.x rejects that batch (plugins.bundledDiscovery was removed upstream), and setting plugins.allow would disable every bundled provider that is not listed. Enablement applies to a running gateway automatically; no restart.

Security: the plugin exposes a full operator control-plane surface on the gateway listener. It is only reached in-process via sandbox.containerFetch; src/index.ts shadows /api/v1/admin/rpc with a 404 before the public catch-all gateway proxy, so it is never reachable through a tenant hostname. Never remove that guard.

Cold-start guard (health probe). sandbox.containerFetch auto-starts the container and blocks until the requested port is ready. The RPC-first health probe (getRpcHealthSnapshot, src/gateway/health-probes.ts) therefore only issues the RPC once a gateway process is actually detected; while the container is cold, or the production start-openclaw.sh start wrapper is still running setup, it returns null — RPC health is skipped until the local Gateway health/port is open — so /api/health falls back to the exec-based getHealthSnapshot (which never waits on the gateway port). Callers that already hold a process record pass it as { proc } to skip the extra scan — buildHealthSnapshot does this with the findExistingGatewayProcess result it already fetched.

Cold-start boot lifetime. #318 split the boot so the Worker launched the gateway after start-openclaw.sh setup exited. /api/health is bounded to a 5s snapshot, so the request returns long before the (multi-second) setup finishes; workerd then cancels the unregistered ensureGateway continuation and the container-side setup completes with no gateway ever launched (the observed cold-start stall: 68 processes, setup exit 0, every curl /health exit 7). The fix was to keep the boot alive past the response, and the current runtime contract collapses the production path into a single /usr/local/bin/start-openclaw.sh start call (startup lock acquired before setup and held through the Gateway exec) followed by post-start — the Worker no longer invokes setup and launch separately. Any route that triggers a cold boot must still keep it alive past the response with c.executionCtx.waitUntil(ensureGateway(sandbox, env, { waitForReady: true })) — waitForReady: true returns the shared start promise unwrapped (no read budget) and dedupes onto the promise the bounded snapshot already started. /api/health and /api/warmup do this; do not replace it with a bare await ensureGateway(..., { waitForReady: false }).

The SDK’s WebSocket transport (createWsClient) is not used: the current OpenClaw wire protocol (v4) requires a connect handshake and a paired operator.admin device identity, which the shared-token loopback path does not provide. HTTP RPC over the shared token is the least-effort RPC-first transport and is what admin-rpc.ts implements.


RBAC & Access Control

Roles

RolePermissionsAccess Scope
ROOTadmin:platform, admin:customer, api:allAll tenants, all debug endpoints
ADMINadmin:customer, api:allOwn tenant only
BASE_USERapi:ownOwn tenant, read-only

Dashboard Pages by Role

PageROOTADMINBASE_USER
Dashboard HomeYesYesYes
AgentsYesYesRead-only
ConnectionsYesYesRead-only
KnowledgeYesYesRead-only
Account/TeamYesYesRead-only
IntegrationsYesYesNo
Secrets VaultYesYesNo
BackupsYesYesNo
ReleasesNoYesNo
DiagnosticsYesNoNo
Debug (/debug/*)YesNoNo
Audit LogsYesNoNo

ROOT is the maintainers’ intended role. It has full access to debugging tools, container inspection, and platform-level diagnostics at /_admin/diagnostic and /debug/*. ROOT is restricted to diagnostics-only API paths (DIAGNOSTICS_PATHS in src/routes/api.ts) — management pages (Agents, Connections, Backups, etc.) are ADMIN-gated. As a diagnostic exception, ROOT can still verify backup connectivity (Test Backup Connectivity on the Diagnostic Dashboard → POST /api/admin/storage/test-connectivity) without the Backups page.

Workflow commit tree. The tenant’s workflow git history (commit tree, branches, tags, releases) is client data. It is served by GET /api/admin/releases/commits, which is deliberately not in DIAGNOSTICS_PATHS: ROOT is blocked at the diagnostics gate and BASE_USER by requireAdminOrRoot, leaving ADMIN-only access. The tree is produced from the sandbox git repository by the release tooling; until that substrate ships the endpoint returns an empty tree. It is no longer rendered on the Backups page — the ADMIN-gated Releases page (/_admin/releases) is the promotion surface, and no dashboard view renders the commit tree today.

In a harness/dev mode (DEV_MODE or E2E_TEST_MODE), /debug/* additionally permits ADMIN so the automated E2E suite can drive CLI/operator routes from a bare workers.dev worker that has no Cloudflare Access or Dispatch Worker in front of it. Real production (DEV_MODE=false, E2E_TEST_MODE=false) stays ROOT-only per the table above.

Agent Push (dev → production promotion)

A tenant owns two instances: its dev environment builds and validates agents (the dev-only fouria-builder / fouria-qa personas), and its production environment serves them. Agent push promotes the built content from dev to production.

The promoted artifact is the full promotable manifest, owned by @moltlazy (packages/moltlazy/src/promotion/index.ts): openclaw.json, the moltlazy per-section $include files, and the agent workspace/. The dev-only agents and all runtime state — state/, credentials/, agents/, sessions/, secrets/ — are never promoted. Provider/channel credentials carried inside openclaw.json do travel; only the runtime secret stores are excluded.

Promotion is driven entirely from the Worker API (operator calls on the dev and production instances); no promotion command runs inside the container or through OpenClaw. The transport is private R2, never HTTP. The dev instance stages the sanitized bundle in its own bucket (promotion/staged/<id>.tar.gz + promotion/pending.json); the production instance pulls it through a read-only PROMOTION_SOURCE_BUCKET binding to that sibling bucket. Both buckets stay separate — production only reads the prefix — and no tenant route is exposed for the transfer.

Production applies the release as a custom backup: it keeps its own runtime state (sessions, transcripts, the control-plane SQLite) while overwriting config / agents / tools and re-applying the gateway section for its own WORKER_URL, then soft-reloads. The apply happens on boot, after the restore barrier and before the launch (so a restored tenant still loads the deployed release), or when an operator calls POST /api/admin/releases/apply. It runs under the shared persistence lease (withPersistenceLease), so a scheduled snapshot can never capture a half-applied tree; it is skipped while the tenant is parked or while a maintenance window / restore is in flight; and it is idempotent — the container release pointer (which is part of the snapshot) is the applied marker, so a restore cannot silently roll it back and an interrupted apply simply reruns on the next boot.

Routes (src/routes/api/releases.ts, src/services/promotion/r2-source.ts):

RouteInstanceBehavior
GET /api/admin/releases/commitsbothWorkflow commit tree (client data; ADMIN-only, not a ROOT diagnostic).
GET /api/admin/releases/statusbothThe release this instance runs (container pointer, falling back to the applied pointer in R2), plus available when the sibling bucket holds a newer release. Also describes this instance: environment, canPromote (dev), canApply (production) and sourceBinding (whether PROMOTION_SOURCE_BUCKET is provisioned).
GET /api/admin/releases/historybothThe push history this instance can see, newest first. Dev reads its own promotion/history/<id>.json entries; production merges the sibling dev bucket’s staged entries with its own applied/failed entries.
POST /api/admin/releases/promotedevExport a sanitized bundle and stage it in this instance’s own bucket, targeted at its production sibling. Body { message?: string } records the deployer’s commit message; the verified Access identity is stored as the author. Returns 202 { staged: true }. Refuses to run on a production instance (409 NOT_DEV_INSTANCE).
POST /api/admin/releases/applyproductionPull the pending release (R2), apply it under the persistence lease, checkpoint it as a recovery point, and reload the gateway. Idempotent.

Every push is content-hashed (SHA-256 of the bundle, computed with crypto.DigestStream while the bundle is uploaded) and carries the deployer’s optional commit message. The hash identifies the push across both instances, and the history makes the sequence of pushes auditable without exposing either bucket to writes from the other side:

  • the dev instance writes a staged entry (promotion/history/<id>.json) into its own bucket;
  • the production instance reads those entries through the read-only PROMOTION_SOURCE_BUCKET binding and writes its own applied/failed entry into its own bucket;
  • the history endpoint merges the two views by release id, with the local entry authoritative (production’s applied wins over the dev staged record).

There is no POST /api/admin/releases/import and no PROMOTION_PUSH_TOKEN: nothing is pushed across the wire, so there is nothing to authenticate on the production side. The R2 binding is the authority and is scoped to the tenant’s own sibling bucket.

All promotion routes (plus the commit-tree endpoint) are ordinary operator routes behind Cloudflare Access and requireAdminOrRoot(). Two practical consequences:

  • On the production platform any ADMIN user whose JWT resolves to the tenant works; on staging the Dispatch gate is ROOT-only, so an operator session with the Paso4 Root group is required.
  • An Access service token (CF-Access-Client-Id/Secret) gets past the edge but resolves no role (it has no email and no custom.role), so the Worker refuses it. Use a browser session or a forwarded root identity.

Export builds a gzip bundle: the config files + workspace are staged in the container (dev-agent workspaces skipped), openclaw.json is sanitized (dev-only agent entries removed, ambient-owner pointers repaired), then archived and streamed into R2 with sandbox.readFile(…, { encoding: 'none' }). That call returns the raw stream and its authoritative byte size from the same RPC, so the FixedLengthStream used for the R2 upload can never disagree with the bytes it writes (measuring the file separately with wc -c did, and workerd rejected the upload with “Attempt to write too many bytes through a FixedLengthStream”). The production side reads the object, writes it with sandbox.writeFile, extracts over /home/openclaw (state untouched — it is not in the archive), re-sanitizes, runs moltlazy patch --only gateway (restored-boot self-heal), and soft-reloads. src/services/promotion/index.ts keeps its own copy of the exclusion list so the Worker bundle never depends on the moltlazy config module; a unit test pins both lists together.

E2E: the former test/e2e-push/ suite drove the HTTP push and is retired with the transport (two local Workers cannot share a miniflare R2 bucket). The R2 pull is covered by src/services/promotion/r2-source.test.ts (stage → pending → apply → applied pointer, plus the park/busy/idempotency skips) and the route tests in src/routes/api/releases.test.ts.

Releases admin page (/_admin/releases)

ReleasesPage (src/client/pages/ReleasesPage.tsx, ADMIN-only, in the Governance group) is the dashboard face of the routes above. It reads GET /api/admin/releases/status and GET /api/admin/releases/history in parallel and renders the steps that apply to the instance it is running on, so the operator no longer needs curl:

  • Dev instance (canPromote): a Commit message field and a Stage release for production button (POST /promote). The message and the verified operator identity are stored on the hashed push, and the page notes that production applies the result on its next cold boot or on demand.
  • Production instance (canApply): an Apply release now button (POST /apply), the release waiting in the sibling bucket (hash, message, author), and — when sourceBinding is false — an explicit warning that the instance cannot see the staged release and must be upgraded/re-provisioned.
  • Both: the Push history (newest first, with short hash, commit message, author, status and dates), the current release pointer (status badge, source, dates, bundle size, promoted agents and the excluded dev-only agents), what is and is not promoted, and the post-apply verification checklist. A skipped apply is explained from the route’s reason (parked, busy:<phase>, already-applied, nothing-pending, bundle-missing, …).

The page never gates on the gateway: /releases/* are Worker-local routes (absent from GATEWAY_TOUCHING_PREFIXES) and stay usable while the container is asleep. Results — including failures — also surface in the notification bell. UI coverage: src/client/pages/__tests__/ReleasesPage.test.tsx and test/ui/releases.spec.ts (mocked API, no Worker).

Auth Flow

  1. Cloudflare Access intercepts request → user authenticates (email OTP, Google, GitHub)
  2. Access injects Cf-Access-Jwt-Assertion header
  3. Worker verifies JWT against Cloudflare Access JWKS endpoint
  4. resolveCustomerRole() resolves the RBAC role — Paso4 ROOT users (platform Access AUD) are always ROOT; client users get their lerma-assigned role (ADMIN/BASE_USER) forwarded by the Dispatch Worker via X-FourIA-Role
  5. userRole and customerSlug are set in Hono context
  6. Route handlers check role before serving content

Dev-instance role preview (ROOT → ADMIN)

On a dev instance — a -dev hostname, or an explicit OPENCLAW_DEV_MODE override (see isViewAsEnabled in src/dev-mode.ts) — a Paso4 ROOT user may view and perform as a lower role from the dashboard’s “View as” switcher: the X-Mock-Role cookie (POST /api/admin/dev/set-mock) downgrades a resolved ROOT role to ADMIN (or BASE_USER) for that session. This lets support/ops inspect and drive a tenant’s dev dashboard as the client’s ADMIN without the production diagnostics-only restriction.

The preview is a strict downgrade and dev-only:

  • resolveCustomerRole() only honours the cookie when the role it already resolved is ROOT (src/auth/middleware.ts). A non-ROOT user setting the cookie cannot escalate — e.g. ADMIN → ROOT is ignored.
  • POST /api/admin/dev/set-mock returns 404 outside DEV_MODE / E2E_TEST_MODE / dev instances, and 403 for non-ROOT callers on a dev instance.
  • The tenant slug is pinned to the instance’s own tenant; X-Mock-Customer is ignored in preview mode, so the preview can never cross tenants.
  • Production instances are unaffected — canViewAs stays false and ROOT keeps its diagnostics-only scope.

GET /api/admin/whoami reports canViewAs: true for a ROOT user on a dev instance so the SPA renders the switcher (DevUserSelectorCompact); local DEV_MODE keeps the existing full mock-role switcher (devMode).

Billing Freeze Gate (F4)

Durable per-customer billing_frozen flag (src/billing-freeze.ts). Frozen blocks new billable actions only — in-flight actions complete, basic non-billable service continues. This is distinct from tenant suspension (Dispatch/Lerma-level), which denies everything.

  • Durability: persisted per-customer as billing-freeze/<slug>.json in BACKUP_BUCKET (R2 — survives cold starts; never in-memory only). No CustomerRegistry DO exists.
  • Lerma → Fouria: Dispatch Worker propagates billing_frozen inside X-FourIA-Metadata (shared Postgres is the source of truth; Fouria never calls Lerma directly). Effective freeze = stored flag OR header flag (fail closed).
  • Fouria → Lerma: GET /api/admin/billing/freeze-status (ROOT-readable diagnostics path) plus the R2 object itself.
  • Writes: POST /api/admin/billing/freeze | /unfreeze (ADMIN-only; ROOT is blocked by the diagnostics-only gate).
  • freezeGate(env, customer) is implemented in src/billable-actions.ts; it reads the durable per-customer R2 freeze record (no request headers available, fails open on read error); enforcement happens at emit time inside emitBillableActions, which filters billing-frozen customers’ events before writing Analytics Engine and enqueueing to BILLING_QUEUE, so in-flight work completes and basic service continues.

Development Tips

Debugging the Container

# Get container ID from Worker logs
docker ps | grep sandbox

# Enter container shell
docker exec -it <container_id> bash

# Inside container: check openclaw status
openclaw status
openclaw devices list --json
openclaw config get gateway.port

# Check logs
tail -f /tmp/openclaw-logs/*.log

Understanding Boot Logs

During cold start, these log patterns are normal:

[Gateway] listProcesses returned 0 processes
[Gateway] Starting new OpenClaw gateway...
[ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP)

The gateway takes up to 3 minutes to become ready. The Worker retries automatically. If errors persist beyond 3 minutes:

  1. Check docker logs <container_id> for OpenClaw errors
  2. Verify CF AI Gateway config (CLOUDFLARE_AI_GATEWAY_API_KEY + CLOUDFLARE_API_TOKEN + CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID)
  3. Run bun run dev:worker to see full Worker output

Service Integrations (Cloudflare Tunnel)

Cloudflare Tunnel support for tenant containers was removed. The container does not start cloudflared, and no CLOUDFLARE_TUNNEL_TOKEN binding is provisioned. The obsidian-sync skill still runs its sync server on port 18790 locally.

CRON and Scheduled Tasks

The Worker runs one scheduled handler against two crons (see wrangler.jsonc → triggers.crons). The handler branches on controller.cron (CRON_BILLABLE_DRAIN / CRON_OPENCODE_CREDITS in src/config.ts):

CronJob
*/2 * * * *Drain buffered billable OTel traces to BILLING_QUEUE (retry safety net for the sync emit).
0 */8 * * 1-5Enqueue the OpenCode credits check onto TASKS (the queue consumer runs it in the sandbox).

The credits enqueue is deliberately not on the frequent tick — otherwise the OpenCode credits notification would fire every couple of minutes. Staging and production keep only the weekday 8-hourly cron; the frequent drain is a local/E2E aid because the synchronous /api/otel/v1/traces emit already covers the normal path.

Containers sleep per a derived idle timeout (staging 15m, dev 2h — the Sandbox SDK 1.0 setInactivityTimeout() platform cap). The billable production environment does not auto-sleep; it is stopped only by an explicit lerma freeze (POST /control → sandbox.destroy()), so a warmed gateway never incurs a cold-start/downtime window after an idle period. There is no SANDBOX_SLEEP_AFTER binding; resolveSleepAfter() derives the value from the instance’s own WORKER_URL (src/dev-mode.ts), the same source the dev/billable axes use. Waking is driven by the per-tenant ActivityScheduler Durable Object alarm, which fires ahead of the next automation and then runs an interval-gated R2 checkpoint.

The container’s idle-sleep decision is owned by the Worker’s Sandbox Durable Object class, TenantSandbox (src/sandbox-do.ts), exported under the DO binding name Sandbox (wrangler.jsonc → class_name: "Sandbox"). It extends the SDK’s legacy Sandbox and overrides onActivityExpired(): on a non-billable instance the hook ignores the persistent gateway (its lock wrapper, the gateway binary, and the tail -f stdout mirror) and stops the container once only that persistent work remains — matching Sandbox SDK 1.0, where code inside the instance does not keep it alive. Any other running process (an in-flight CLI/backup command) still counts as activity and defers to the SDK gate. On the billable production environment the hook defers to the SDK’s keepAlive gate instead, so the container never stops itself.

When a non-billable instance is due to idle-sleep, the hook first proves that an existing archive already covers the current filesystem: either a committed snapshot inside the idle window (the container has had no activity for the whole window), or — with change detection on, the default — an unchanged result from the same change scan the scheduled path uses (a quiet tenant takes no scheduled snapshot, so its newest archive can age past the window). Either way the hook stops inline through stopIfIdle() without re-archiving. That shortcut exists because the previous unconditional snapshot was itself renewing the DO activity window, so the stop-time re-check refused and the next idle expiry snapshotted again — a loop of 422 MB archives that kept tenant-tarjetero-dev awake. Otherwise (state changed, no archive, no BACKUP_BUCKET, or a probe failure) it enqueues a durable deprovision job (mode sleep) on the ActivityScheduler DO, which snapshots the tenant and only then calls sandbox.stopIfIdle() — fail closed, so a container is never slept without a committed archive. Without the ACTIVITY binding (local/dev) it falls back to stopping inline. See Backup before deprovisioning below.

Lerma’s HealthMonitor probes the passive GET /api/health?passive=1 every few minutes (authenticated at the Cloudflare edge with a service token — see the path-scoped public-health Access applications in apps/fouria/iac/access.tf). Passive probes never run the exec RPCs that would cold-start a stopped container: getHealthSnapshot first does the non-waking isRuntimeActive() read and reports status: "sleeping" (HTTP 200, healthy-idle) for an idle container. Health probes are observability-only and never schedule a backup.

The container runs scheduled tasks via OpenClaw’s cron system.

Task Buffer (Cloudflare Queues)

The Worker is a durable buffer for OpenClaw-side jobs (src/tasks/): it never implements the job logic itself.

  • Producer — scheduled enqueues the recurring OpenCode credits check onto TASKS; POST /api/admin/tasks/opencode-credits-check/enqueue is the manual trigger (ADMIN/ROOT).
  • Consumer — default.queue wakes/heals the sandbox via ensureGateway, runs openclaw automations run opencode-credits-check inside it, then acks (Cloudflare Queues own retries/backoff; max_retries: 5).
  • POST /api/admin/tasks/opencode-credits-check is the manual consumer trigger (runs the automation directly, bypassing the queue). It clamps its total wall-clock to TASK_TOTAL_TIMEOUT_MS (90s) — the gateway wake, exec launch, CLI wait, and log recovery are each clamped to the remaining budget — so a cold/contended boot always answers within the endpoint’s ceiling instead of outliving the caller (the e2e suite’s curl --max-time hit 120s exactly on cold runs). The shared gateway start promise keeps booting in the background.
  • Automation definition — start-openclaw.sh’s ensure_opencode_credits_automation (dev-only) creates the recurring opencode-credits-check cron job; the Worker only runs it. The guard queries openclaw cron list --all --json in addition to the legacy automations list --json, which missed a prior boot’s job and produced two duplicate opencode-credits-check jobs on a dev instance.
  • Heartbeat cadence — start-openclaw.sh’s ensure_dev_heartbeat_cadence (dev-only) reconciles the heartbeat-fouria-builder job on every post-start. OpenClaw 2026.10.5 retired agents.entries.*.heartbeat: a heartbeat is now an ordinary automation job, and config reloads/restarts never rewrite an existing job, so the instance created by the migration kept the default 30-minute cadence even though moltlazy declares heartbeat.every: 12h (observed on tenant-tarjetero-dev: 30-minute heartbeats for 9 days, ~317k tokens each on the main session, until the OpenCode subscription hit 0%). The reconciler reads openclaw automations get <id> --json, compares schedule.everyMs/sessionTarget/payload.lightContext, and only then runs automations edit --every 12h --session isolated --light-context (split so a rejected --light-context cannot lose the cadence fix). It never resets the schedule anchor when the job already matches.

Telegram Webhook Ingress (#364)

Managed tenants switch Telegram to webhook mode so an inbound update wakes a sleeping container (long polling runs inside OpenClaw and cannot wake it). The Worker is the always-on ingress (src/telegram-ingress/):

  • Route — POST /telegram-webhook/:account is mounted before Cloudflare Access (secret-auth, not JWT). It resolves the account’s config from a Worker-readable F4E1 registry, constant-time validates X-Telegram-Bot-Api-Secret-Token, and rate-limits — all before any container wake, so a rejected request never starts or renews the container.
  • Registry — telegram-ingress/<slug>/<account> in BACKUP_BUCKET (F4E1-encrypted, BACKUP_ENCRYPTION_KEY). Holds the per-account webhook secret
    • bot token, readable by the Worker without the container running. Populated by POST /api/admin/telegram-ingress/:account (multi-bot).
  • Hot path — when the gateway is running and ready, the update is forwarded straight to OpenClaw’s webhook route (port 18789) and its x-openclaw-delivery-accepted response is passed back to Telegram.
  • Cold path (“instance is not in hot mode”) — the Worker notifies the sender via sendMessage (“waking up…”), enqueues the update on the TELEGRAM_INGRESS_QUEUE (Cloudflare Queue), acks Telegram (the message is now durable), and kicks the cold start. The queue consumer wakes/heals the gateway, waits for readiness, and forwards — acking only after OpenClaw confirms receipt, so a message is never “popped” until it has arrived (at-least-once, mirroring Telegram’s own delivery). max_retries: 10, retry_delay: 60s covers a multi-minute cold boot.
  • Freeze policy — enforced upstream by the Dispatch Worker: a frozen/operator-stopped instance is never woken by a Telegram update (no user role → always non-ROOT); it gets an explicit 503 and the message stays in Telegram’s retry queue.
  • Dispatch ingress — a dedicated host (hooks.fouria.io / hooks.dev.fouria.io, TELEGRAM_INGRESS_HOST) excluded from the wildcard *.fouria.io Access app. It resolves /v1/<slug>/<account>, applies the freeze gate, and forwards (rewritten to POST /telegram-webhook/:account) with no Access JWT.
  • moltlazy — start-openclaw.sh passes the Worker-injected TELEGRAM_WEBHOOK_MANIFEST to moltlazy, which writes channels.telegram.accounts.<account>.webhookUrl/webhookSecret/webhookPath/legacyWebhook (fill-only, preserving the user’s botToken/dmPolicy).
  • Queue observability — the per-tenant TelegramIngress DO records every ingress event (bounded counters + the last 20): queued (a cold update was enqueued and woke the container), forwarded (hot path), delivered (queue consumer confirmed receipt), failed (a delivery attempt failed and will retry). GET /api/admin/telegram-ingress/status returns the snapshot, and GET /api/admin/telegram-ingress/ws streams it (a frame per change) for the dashboard. The admin shell’s TelegramIngressProvider opens that socket only while the feature is in use, raises a notification-bell entry (“Telegram message queued — waking your instance”, or a delivery-failure alert), and feeds the Telegram Queue widget on the home page (state + queued/delivered/failed
    • last event), which is itself hidden until the ingress is in use.
  • Coalesced status writes — the status object is not rewritten per update (durable-agents audit finding F8). telegram-ingress/coalesce.ts batches events in memory and flushes them on one alarm, TELEGRAM_INGRESS_STATUS_FLUSH_MS (1 s) after the first event of a burst; TELEGRAM_INGRESS_STATUS_MAX_PENDING_EVENTS (25) forces an earlier flush so memory and staleness stay bounded. getStatus — and therefore the status route and its WebSocket — merges the unflushed batch, so an event is visible as soon as it is recorded. A failed flush puts the batch back and the alarm retry persists it; only an eviction inside the window can lose counters, and those are UI state, not billing records.

Backup and Hard-Restart Policy

Backups are snapshots of /home/openclaw, not a live filesystem journal. Agent workspaces are therefore kept under /home/openclaw/workspace; files outside that tree are not protected by the Sandbox snapshot.

The Sandbox SDK’s createBackup returns a handle marked with localBucket: true when local-bucket mode is enabled (persistence.ts). That mode is an explicit BACKUP_LOCAL_BUCKET=true opt-in (falling back to DEV_MODE only when unset, for local development) — it must not be keyed on DEV_MODE alone, because the cloud E2E deploy sets DEV_MODE=true merely to bypass Access and its backups are real remote-R2 snapshots (see ENV-VARIABLES.md). The flag is persisted in backup-handle.json and forwarded to restoreBackup(), so a local backup is always restored through the local binding — never through the remote presigned-URL path, which 404s for archives that exist only in the local bucket. Legacy catalog entries without the flag default to the current local-bucket mode, and a presigned-URL 404 during restore is treated as a gone backup and pruned from the catalog.

Health probes are observability-only: GET /api/health must never write the R2 backup catalog — no scheduleBackupIfNeeded, passive or active. A snapshot is only meaningful for a live instance (a sleeping container has no workload, so its filesystem has no changes to store), and waking one purely to snapshot would restart billing. Snapshots therefore run only against a running runtime:

  • the Dispatch Worker cron sends a backup control action; the tenant asserts sandbox.isRuntimeActive() (a non-waking bridge read) and returns skipped: 'not-running' for a sleeping container instead of waking it. A running instance is acknowledged immediately (202) and the snapshot is enqueued as a durable job on the per-tenant ActivityScheduler DO — a waitUntil task is cancelled ~30s after the response (platform cap) and was observed dropping the snapshot mid-cold-boot, so the cron never blocks on it and the alarm runs it to completion;
  • POST /debug/restart-openclaw quiesces the gateway and checkpoints inside a restart maintenance window (see the hard-restart paragraph below);
  • POST /api/admin/storage/sync is the explicit admin checkpoint. ?async=1 acknowledges with 202 and snapshots in waitUntil; the default (no flag) blocks and returns the result, which the E2E checkpoint relies on;
  • POST /debug/backup/sync and POST /debug/backup/async (ROOT) trigger the blocking and non-blocking checkpoint modes on demand;
  • POST /api/admin/storage/reconcile (ADMIN/ROOT) runs the catalog heal (reconcileBackupCatalog) on demand: adopt orphaned backups/*/meta.json archives, sweep metadata-less uploads, re-encrypt plaintext archives, and enforce BACKUP_MAX_VERSIONS. The durable repair step runs the same work before every scheduled snapshot, and GET /api/admin/storage/backups is a pure read that never heals the catalog.

Scheduled snapshots additionally refuse to run while a restore-needed marker is present, rechecked under the shared lease, so a wake that lands mid-restore cannot snapshot and promote a partial filesystem over the committed recovery point. The marker is the durable, R2-backed restore latch — the scheduled gate does not use a container-state probe, because the legacy exists probe can fail in a deployed container and failing the gate closed would silently disable every scheduled backup.

Soft reload (POST /api/admin/gateway/restart) does not snapshot. When no live gateway is found it starts the gateway through ensureGateway (never arming restore-needed, so live changes survive) and returns 409 during a maintenance window.

Restore trigger policy (when may a backup be applied?)

A restore is destructive: sandbox.restoreBackup (legacy) and DirectoryBackup.restore (v1) replace the writable layer, so every write made after the archive was taken is lost. A gateway reboot is not a restore trigger: a gateway that exited, crashed, OOM’d, or is being soft-reloaded keeps its container filesystem, and the boot must launch on that live state. The policy is enforced in restoreIfNeeded / probeContainerState (src/persistence.ts) for the legacy lane and SandboxV1.restoreTenantStateIfNeeded() (src/sandbox-v1.ts) for v1:

TriggerLegacyv1
Admin POST /api/admin/storage/restore/:id (ADMIN/ROOT)restoreBackupById — stops the gateway, opens a maintenance window, force-promotes the pointSandboxV1.restoreRecord inside the DO
Durable restore job (enqueued only by that route)restoreBackupByIdn/a
POST /debug/restart-openclaw (legacy)quiesced checkpoint → signalRestoreNeeded latchkill + relaunch on the same state, no latch
Container positively reports no tenant state (replaced/killed)probeContainerState() === 'absent' → apply the committed pointtenantStatePresent() false → apply the committed record
Gateway down / restartno restore (probe returns present)no restore (state file present)
Probe inconclusive (transport error / malformed answer / success: false)ContainerStateUnknownError — boot fails closedthe state-probe exec throws or exits non-1 — boot fails closed
Request-owned boot (restoreOwner: 'request')deferred to the durable boot job; never runs a destructive restore under a requestthe DO owns the boot

Two consequences worth knowing when operating a tenant:

  • A legacy container is only read as empty when every state path answers a definite “no”. The probe checks /root/openclaw/.openclaw/openclaw.json, /home/openclaw/.openclaw/openclaw.json, and /home/openclaw/.fouria-restored. A success: false answer (the legacy SDK returns the flag alongside exists) is a failed probe, not a “no”. A runtime that exposes no exists() method is probed through the shell over the same paths instead of being assumed empty; an unavailable shell or unrecognized output is unknown, which fails closed.
  • A v1 container is stopped by its inactivity policy (keepAlive for billable production; finite for dev/staging). Sandbox 1.0 loses an instance’s disk on stop, so a stopped v1 container legitimately boots by restoring the committed DirectoryBackup. The data-loss bound is therefore the SANDBOX_V1_SNAPSHOT_INTERVAL_MS cadence, not the gateway process lifetime.

Adversarial coverage of the full trigger matrix (legacy + v1) lives in src/restore-trigger-adversarial.test.ts and the “a v1 boot restores only an empty container” section of src/sandbox-v1-adversarial.test.ts.

Pre-reprovision checkpoint (legacy). When the stuck-start watchdog has exhausted its resets (GATEWAY_START_STUCK_MAX_ATTEMPTS), it no longer destroys the container unconditionally. It first takes a bounded (GATEWAY_STUCK_RECOVERY_CHECKPOINT_TIMEOUT_MS) snapshot — the gateway is already stopped by then, so the archive is quiesced — and proceeds only if the committed recovery point advanced to that archive. A failed checkpoint, a pending restore over the point, or a recovery-hold blocks the reprovision and logs gateway.start.stuck.reprovision-blocked, leaving the container and its state intact for an operator (POST /debug/reprovision checkpoints and fails closed the same way). Without this guard, two stuck boots on a gateway with a persistent startup failure silently reverted the tenant to the previous snapshot. v1 has no equivalent automatic destroy.

Snapshot promotion is further gated. A snapshot taken while restore-needed is set over a committed recovery point is archived but does not move the pointer, and a pre-replacement checkpoint is refused with 409 (RestorePendingError) before the gateway is touched. A tenant with no recovery point yet may promote its first snapshot. A restore-only restore (setRecoveryPoint: false) writes recovery-hold.json: scheduled snapshots archive without promoting until an explicit snapshot clears the hold.

All three snapshot paths — scheduled, manual sync, and the hard-restart checkpoint — share one tenant-scoped R2 backup-lock lease, so only one isolate ever stages the fixed legacy stage directory or calls sandbox.createBackup at a time. The lease has a 60s TTL, is renewed every 20s while a snapshot is in flight, and is released by compare-and-swap to a tombstone (never by deleting the object, which would race a concurrent takeover). The two entry paths differ only in how they acquire it:

  • Scheduled acquisition is non-blocking — scheduleBackupIfNeeded takes the lease with tryAcquireLock and, if another isolate holds it, skips this tick (returns without waiting).
  • Manual (POST /api/admin/storage/sync) and the hard-restart checkpoint wait for the lease with bounded exponential backoff (up to 120s), then fail with an error instead of silently reporting success.

The lease is held across the legacy stage + sandbox.createBackup consumption and released via the CAS tombstone when the snapshot finishes. The dashboard’s PersistenceLockIndicator mirrors the lock in the UI from the operation lifecycle (PersistenceLockContext) rather than reading the R2 lease: the BackupHistoryPanel raises it when it fires an async backup/restore and drops it when the completion check settles, so there is no polling.

Async admin operations + non-blocking Backups page. A manual snapshot or a restore can wait up to the 120s lease and then transfer a large archive, so holding the admin HTTP request open is a poor UX and a needless failure mode (a restore that lost the lease race surfaced as a bare 500 after ~132s, with the underlying error dropped). Both admin routes therefore accept ?async=1: POST /api/admin/storage/sync?async=1 and POST /api/admin/storage/restore/:id?async=1 return 202 { accepted: true } and run the work in executionCtx.waitUntil. The BackupsPage reads the fast raw config (GET /api/admin/config/raw, a single cat) instead of the slow structured /api/admin/config (which awaits ensureGateway + up to two 30s shell waits) and paints every section independently, so no single request blocks the page. BackupHistoryPanel fires the async endpoints, immediately pushes a “started” notification to the bell, then polls the durable job (GET /api/admin/storage/jobs/:id, getBackupJobStatus) while a snapshot is in flight (falling back to the history diff when the server returns no job id) or /api/health (restore) and pushes success/failure when the operation completes — the outcome reaches the admin via NotificationProvider even after navigating away. Restore failures are now console.error’d for Observability.

Crash-consistency (issue #344). If a Worker isolate is killed by its CPU budget between sandbox.createBackup() uploading the archive and the catalog write, the archive would otherwise be orphaned (present in R2 but absent from backup-handle.json, hence invisible to listBackups/restore). Two guards prevent/repair this: the catalog entry is committed immediately after the archive is created (after encryption, before the config-version capture and retention prune), and the catalog is reconciled explicitly from backups/*/meta.json by reconcileBackupCatalog (the durable repair step, POST /api/admin/storage/reconcile, restoreIfNeeded, scheduleBackupIfNeeded, and the reprovision guard) whenever an archive is missing from the handle. getBackupHistory is a pure read: a dashboard poll never adopts, prunes, sweeps, or writes. The scheduled backup control action also runs under a dedicated CPU budget (BACKUP_CONTROL_CPU_MS in workers/fouria-dispatch) so the orchestration can finish; a killed isolate’s lease is reclaimed via the 60s TTL.

Retention is enforced by the reconcile, not by a catalog read. The two are coupled: an isolate killed after createBackup() leaves an orphan that reconciliation adopts, and if later snapshots then fail before their own prune the catalog and R2 grow past BACKUP_MAX_VERSIONS indefinitely (observed on tarjetero-dev: 7 archives under maxVersions: 3, the catalog written entirely by reconcile, with the scheduled snapshots failing at the legacy stage). reconcileCatalogState (behind reconcileBackupCatalog and reconcileOrphanedBackups) runs cleanupOldBackups after merging, and the durable repair step runs it before the gate step of every scheduled snapshot, so the cap holds however the snapshot path ends. It still performs no write/delete when the catalog is already within the cap, and no read path can trigger it.

The catalog is a compare-and-swap, not a last-writer-wins file. Every backup-handle.json write goes through mutatePersistenceStore(bucket, mutate) (persistence.ts): it reads the handle and its etag, computes the caller’s mutation, and writes with onlyIf: { etagMatches } (etagDoesNotMatch: '*' for a create-only handle). On a conflict it re-reads and re-applies the mutation against the fresh catalog — never a blind merge, because retention and restore mutations legitimately remove entries — up to five times before raising PersistenceStoreConflictError. A mutation returns null to skip the write. Reconciliation, snapshot finalization (catalog.record-outcome), the retention prune and the schedule-timestamp commit all run inside this CAS, so a dashboard poll overlapping a snapshot can no longer drop the newly committed archive (or resurrect a pruned one).

R2 enumeration follows truncated/cursor. listAllObjects(bucket, prefix) backs the backup reconcile sweep, the orphan archive sweep, cleanupOldConfigVersions and listConfigVersions. Previously each site read a single page (1,000 backup objects, 500 config versions) and silently treated it as the whole result. A truncated page without a forward cursor logs and stops rather than looping; page sizes are otherwise unchanged.

Restore robustness. restoreIfNeeded/restoreBackupById persist the restore-needed marker before changing the mounted filesystem or applying an archive. This includes an ordinary cold restore where no marker existed before startup: if the restore or legacy materialization fails after writing partial state, the next invocation retries instead of treating a leftover config file as proof of success. The marker is cleared after restoreBackup and legacy materialization both succeed, or when candidate scanning finds no restorable archive and there is no stable recovery point (the explicit clean-start case). The pre-restore umount is best-effort: || true guards the shell command, but the exec RPC itself can still throw (a cold-start/platform interruption); that is caught and logged instead of aborting the restore before the archive is applied. An empty catalog read also never deletes a pending marker — it can be transient (a reconcile list failure, or an archive not yet visible), and dropping it would silently lose a requested retry.

scheduleBackupIfNeeded refuses to run while the durable restore-needed marker is present (or its status cannot be read), rechecking under the shared lease, so a scheduled wake that races a restore cannot snapshot a partial filesystem and replace the stable recovery point. It deliberately does not gate on a container-state probe: the legacy exists probe is unreliable in a deployed container, and a false “no state” would silently disable scheduled backups. The marker is normally absent while idle or after a successful restore; cold restore selection uses the committed recovery point, not marker presence alone.

When candidate scanning finds no restorable archive and there is no stable recovery point, restore remains a permitted clean start; the temporary retry marker is removed so onboarding and the next scheduled snapshot can proceed. If a stable recovery point exists but cannot be restored, startup fails closed and the marker remains.

Fail-closed restore (no silent empty boot). A committed stable-backup.json is the tenant’s recorded recovery point, so its presence means the tenant was initialized. If restoreIfNeeded finds a recorded recovery point but no restorable archive (an empty catalog with no stable pointer is only a brand-new tenant; a catalog drained of its archives, or a stable pointer whose archive is gone, is not), it now throws TenantStateUnavailableError instead of reporting a clean start. Letting the container boot empty would run onboarding against an empty filesystem and the next snapshot would overwrite the last good recovery point — silent data loss. The cold start aborts (the gateway never launches), restore-needed is preserved, and the instance stays visibly unavailable for an operator to recover. When the catalog is empty but the stable pointer is present, the recorded entry is attempted directly first, so a transient catalog-read failure still restores rather than falsely failing closed.

Restore skip is keyed on container state, not runtime liveness. The cross-isolate cold-boot dedupe previously skipped a restore when isRuntimeActive() returned true (“Runtime already active; skipping duplicate restore”). But a freshly provisioned container — started by the prelaunch port probe, or replaced by an image rollout — is already “active” while its filesystem is empty, so that check let a replaced container skip its restore and come up empty (observed on tenant-client-dev: the skip logged 34× alongside legacy-stage bridge timeouts). restoreIfNeeded now probes the container’s actual state (containerHasState, the restored/onboarded openclaw.json under the legacy /root root or the v1 /home root) instead: absent state means a fresh container that must restore. It is always safe to attempt an idempotent restore; it is never safe to infer state from a running process. A probe failure is treated as “no state” (restore), never as “already restored”. isRuntimeActive() remains the correct read for the non-waking health/sleep paths, where the question really is “is the runtime up?”, not “is the tenant state present?”.

Pre-replacement checkpoint (X-FourIA-Control: checkpoint). The /control route gained a synchronous, verified checkpoint action. The control plane (Lerma) calls it immediately before an image rollout / container replacement, so the state about to be discarded is durably captured first. Unlike backup (interval-gated, 202-accepted, best-effort) it bypasses the interval gate, runs the full snapshot orchestration, and fails closed (non-2xx) unless the committed archive’s meta.json is present and restorable. A sleeping/destroyed instance has no live state and is reported as skipped: not-running rather than woken. The Dispatch Worker forwards checkpoint with the same raised CPU budget as backup (BACKUP_CONTROL_CPU_MS). In Lerma, both upgrade paths (TenantProvisioner’s @upgrade_pipeline :checkpoint step and Instances.do_upgrade) invoke Cloudflare.Containers.checkpoint/1 before the rollout. The step is best-effort: a control-plane failure is logged and the upgrade proceeds, so a checkpoint problem can never brick a valid upgrade.

Control-plane reachability (Cloudflare Access). The Dispatch Worker’s control endpoints (POST /_control/:slug/{freeze,unfreeze,backup,checkpoint}, GET /_invalidate/:slug) sit behind Cloudflare Access on the platform domain (dev.fouria.io staging, dash.fouria.io production). Without a path-scoped Access application those headless calls are redirected to the Access login and return HTML, so the checkpoint is skipped. apps/fouria/iac/access.tf declares public_control_apps with a bypass policy (same posture as the Lerma /api/* CI bypass), so the path is reachable without an Access login; the Worker still enforces its own internal bearer token on top. The bearer is the Worker secret INTERNAL_DISPATCH_INVALIDATE_TOKEN (required, no hardcoded fallback); the Worker compares SHA-256 digests in constant time and answers 403 when the secret is unset or the credential does not match. Lerma must send the same value (deploy-dashboard passes the production value, or STAGING_INTERNAL_DISPATCH_INVALIDATE_TOKEN for staging) — without it the internal calls 403 and the checkpoint cannot run. Roll out the Dispatch Worker before Lerma in each environment, because a Lerma deploy holding the new value is rejected until the Worker has it (and vice versa).

Checkpoint outcome telemetry. A successful tenant checkpoint returns the archive backupId, quiesced (whether killGateway completed), and maintenanceWindow (whether the cross-isolate R2 guard was opened). Dispatch forwards only those checkpoint fields, and Lerma logs them as backup_id/quiesced/maintenance_window; skipped is reported separately when the tenant is not running. Require both quiesced: true and maintenanceWindow: true to claim the guarded quiesce path ran; a successful snapshot by itself only proves archive creation.

The checkpoint takes the R2 lease before quiescing. A live createBackup can race the gateway’s own writes, so after acquiring the shared backup-lock lease the action records a durable, cross-isolate maintenance window (src/maintenance.ts, maintenance/replacement.json in BACKUP_BUCKET) and then stops the gateway (killGateway) before snapshotting. A busy lock fails before maintenance is opened or the gateway is stopped; it must never drain the tenant while waiting for another snapshot. While the window is open, ensureGateway refuses to start the gateway (GatewayMaintenanceError) so a request served by another isolate cannot resurrect it mid-snapshot; the window has an explicit expiry (15 min) so a crashed controller can never wedge a tenant, and a malformed/expired record is discarded (fail open). The checkpoint action always restores service and closes the window in a finally, even when the snapshot fails, so a failed checkpoint never leaves the tenant drained. Its recovery restarts the existing live filesystem with skipRestore: true: the snapshot attempt did not replace the container, and trying to restore while recovering from a lock timeout would enqueue another waiter behind the same lock. The proxy maps maintenance refusal to a retryable 503.

restoreIfNeeded also coalesces concurrent calls within a Worker isolate by R2 bucket. The startup WebSocket allows only one in-flight ensureGateway({waitForReady:true}) warm-up, retrying only after a failed attempt settles and with bounded exponential backoff. Its recurring health snapshots are passive/observation-only, and startup-progress shell reads run only when the startup wrapper is actually visible; it does not start a fresh restore attempt or shell RPC on every poll. Together these prevent a long-lived startup stream from multiplying 120-second waiters behind one in-flight persistence operation or waking an empty container ahead of the restore.

Checkpoint recovery restarts the existing container filesystem with ensureGateway({ bypassMaintenance: true, skipRestore: true }). A checkpoint does not replace or clear that filesystem; if its snapshot cannot acquire the shared backup lock, recovery must not call restoreIfNeeded and enqueue another waiter on the same lock it just timed out waiting for.

Only the snapshot is fail-closed. The maintenance window and the gateway stop are best-effort guards: a failure to open the window or to stop the gateway falls back to the previously-shipped live snapshot instead of failing the replacement (a broken guard must never block an upgrade). The post-snapshot meta.json check is retried briefly before it is treated as a real failure, so a transient R2 visibility gap cannot fail an otherwise-successful snapshot. The checkpoint returns quiesced: true only when the gateway was actually stopped.

This closes most of the planned-replacement loss window, but it is not yet a full drain-to-replacement fence: the gateway is restarted after the checkpoint, so writes between the checkpoint and the rollout are still outside the archive, and the shared control-plane coordination record is not implemented. Durable primary storage for unexpected container loss remains the gated next step in docs/plans/legacy-sandbox-zero-data-loss.md.

Rollout park (X-FourIA-Control: checkpoint-park / resume)

The plain checkpoint restarts the gateway before the rollout, so anything written in between is discarded with the replaced container. The park variant is the zero-loss sequence: it is the same verified snapshot, but the tenant is left drained and write-blocked until the control plane calls resume after the container has been replaced.

  • Ordering. checkpoint-park acquires the R2 lease, opens the maintenance window with parked: true (operation: "rollout"), arms the exec lock, and only then stops the gateway and snapshots. The archive therefore describes a filesystem no writer can touch afterwards.
  • Two enforcement layers (src/tenant-park.ts):
    • Request: tenantParkGuard middleware refuses mutating (POST/PUT/ PATCH/DELETE) /api/admin/* and /debug/* requests with 409 { code: "TENANT_PARKED" } while the durable park record exists. Reads stay available so the dashboard can still show state. This is the authoritative, cross-isolate check (R2 read per mutating request).
    • Exec: startShell calls assertTenantExecAllowed before the launch RPC, so background/spawned work that never passes a route (schedulers, promotion helpers, fire-and-forget tasks) cannot start a process either. The park’s own path runs inside withTenantExecBypass (exception-safe, nestable), which is the only way past the lock. That lock is bounded (PARK_EXEC_LOCK_TTL_MS, 60 s): it cannot be released cross-isolate — resume lands in whichever isolate receives it — so it closes the in-isolate race during the park and then lets the durable record (middleware + ensureGateway refusal) be the only gate. Background exec is additionally gated by ensureGateway, which refuses to run during any maintenance window.
  • The gateway is the other writer, and it is gone: automations and channels run inside the gateway process the park stopped. ensureGateway already refuses to launch during any maintenance window, so a request in another isolate cannot resurrect it.
  • Live agent sessions are switched to read-only first. Before the gateway is stopped, checkpoint-park calls drainLiveSessions (src/gateway/session-drain.ts): it reads sessions.list over the Gateway WebSocket (operator.read + operator.write), patches every session to permissionMode: 'read-only' (exec denied, managed mutation tools omitted), and waits for the turns that were already running to finish — so the next turn picks the mode up instead of being interrupted mid-write. The previous mode per session is recorded durably (maintenance/session-modes.json, never inside the container) and handed back by resume/the post-restore path, because the mode lives on the session row and is therefore part of the snapshot. A session that is already read-only is skipped (nothing to restore), and new sessions are untouched — they inherit the configured policy, so nothing here becomes a default. The drain is bounded (SESSION_DRAIN_TIMEOUT_MS, 120 s): a turn that outlives it is reported as sessionsDrained: false and the park proceeds, since the quiesce aborts it cleanly and every write it already made is inside the snapshot. A long agent turn can therefore never block a release. resume waits (SESSION_MODE_RESTORE_WAIT_MS) for the replacement gateway, applies the map and clears it; if the gateway is not up yet the map stays and launchRestoredState applies it once the restored state is serving (so an operator restore of a parked archive also puts the modes back).
  • Telegram users are told, by the ingress, not the model. While the tenant is parked, POST /telegram-webhook/:account answers the sender directly with TELEGRAM_PARKED_NOTIFY_MESSAGE (Bot API sendMessage, no gateway call, no AI turn), records a parked ingress event and returns 200 without enqueuing or waking — the update could not be delivered during the park anyway. The queue consumer does the same for an update that was already queued: it answers and acks instead of retrying for the whole rollout.
  • resume closes the park, disarms the lock, clears the stale in-memory restore barrier (clearPersistenceCache, as freeze does), and requests the boot. It deliberately does not pass skipRestore: the replacement container is empty and must rehydrate from the parked checkpoint. A resume with nothing parked is a no-op (resumed: false) and never wakes a sleeping tenant.
  • Failure behaviour. An aborted park (snapshot/lock failure) un-arms the lock, closes the window, and restores service exactly like a failed plain checkpoint, so a failed rollout never leaves the tenant drained. The 15-minute window TTL remains the crash backstop: a controller that dies mid-park cannot wedge a tenant.
  • Control surface. POST /control accepts checkpoint-park and resume; every other action (backup, freeze, unfreeze) is refused with 409 TENANT_PARKED while the tenant is parked. The Dispatch Worker allows and forwards both actions (CONTROL_ACTIONS/isControlAction), gives checkpoint-park the inline snapshot CPU budget, and echoes only the whitelisted parked/resumed fields.

Lerma drives the sequence from @upgrade_pipeline: :checkpoint calls Cloudflare.Containers.checkpoint(instance, park: true) and is fail-closed (a checkpoint that cannot be taken aborts the upgrade before the container is replaced; the only exemption is a sleeping instance, which has nothing live to lose), :dispatch_deploy performs the script upload + image rollout, and the :resume step (immediately after) releases the park. Both error paths call Containers.resume/1 best-effort so a failed upgrade cannot leave a tenant parked.

What is still outstanding is the post-rollout readiness gate (waiting for /api/health to report a running gateway and verifying the deployed image before declaring success, plus a fleet-level cohort/abort policy) — see the rollout section above.

The pre-snapshot legacy-stage shell runs the node:sqlite online backup plus a symlink-preserving tar pipe of the legacy roots, with --exclude='*/node_modules/openclaw'. It deliberately preserves symlinks rather than dereferencing them (cp -aL) and excludes the image-provided shared OpenClaw package (extensions/<id>/node_modules/openclaw → the image’s /usr/local/lib/node_modules/openclaw). -L expanded that package into every archive (three full copies), which pushed the stage past its bound and — worse — made the restore materialization a multi-minute copy (observed on staging-client-dev: cp -a … /root/openclaw/.openclaw/ running 8m45s, starving the gateway launch, leaving restore-needed set, and re-running the restore every boot). The restore bridge (materializeLegacyRootsAfterRestore) uses the same tar --exclude pipe, so an old backup’s dereferenced copy is never materialized; a copy-then-rm -rf is equally slow because deleting a squashfs-lower tree requires copying each file up first. restoreIfNeeded clears the restore-needed marker immediately after a successful materialization (before the analytics bookkeeping) so an isolate eviction cannot leave a stale marker. On a large tenant the stage can exceed the old 300s bound (observed killing the tarjetero stage at exactly 300s — issue #344 follow-up). The bound is LEGACY_STAGE_TIMEOUT_MS (default 600000), the idempotent stage is retried once on a transient timeout/transport failure, and a failure logs exit/timedOut/stderr plus a structured process event with the Sandbox processId for cross-dataset correlation. The stage wait is transport-resilient (runShellResilient → waitForExitResilient): a Sandbox DO↔container drop mid-stage reattaches by process id and resumes within the stage deadline, so a mid-stage DO reset no longer ends the request. The Dispatch backup CPU budget (BACKUP_CONTROL_CPU_MS) is aligned to the same 10 minute bound.

The scheduled backup control action acknowledges immediately (202 { ok: true, accepted: true, jobId }) and enqueues the snapshot as a durable job on the per-tenant ActivityScheduler DO, so neither the dispatch cron nor an operator request is bounded by it. The DO alarm runs the job past the request and retries under a bounded lease — the previous waitUntil approach was cancelled ~30s after the response and could drop a snapshot that was still cold-booting. Combined with the crash-consistent catalog commit and orphan reconciliation (#344) the archive stays durable even if the isolate is replaced mid-snapshot. POST /debug/backup/async and POST /debug/backup/sync (DEBUG_ROUTES, ROOT) expose the two modes on demand.

Config-version snapshots. Alongside the archive, each snapshot captures /home/openclaw/.openclaw/openclaw.json into R2 under configs/<timestamp>.json (30-day retention) for the Backups page’s Config Version History / diff. The history is rendered with client-side pagination (10 versions per page, newest first) so a long-lived tenant does not paint thousands of rows. The capture normalizes sandbox.readFile() across SDK generations (readSandboxFileText in persistence.ts): the legacy SDK returns { content, … }, SDK 1.0 returns a Response, and older helpers return a bare string/Blob. Calling String() on the structured result collapsed it to "[object Object]", which was stored verbatim and then rejected by the diff with "[object Object]" is not valid JSON. A JSON.parse write-time guard now refuses non-JSON config versions, and the diff viewer (DiffTool) shows a clear “corrupted” message if a pre-existing bad snapshot is opened.

POST /api/admin/storage/test-connectivity (ADMIN/ROOT) is a read-only diagnostic that verifies the R2 backup bucket binding is reachable by HEADing the backup catalog handle — a 404 (no handle yet) still counts as connected. It is whitelisted for ROOT through the diagnostics gate (DIAGNOSTICS_PATHS in src/routes/api.ts) so platform operators can test backup connectivity from the Diagnostic Dashboard without explicit access to the ADMIN-gated Backups page.

Hard restart (POST /debug/restart-openclaw) takes the shared lease, then opens a restart maintenance window and kills the gateway before cutting the archive, so the checkpoint is consistent. It sets restore-needed before closing the window, so no request can relaunch the gateway on pre-restart state in between. If the checkpoint fails after the stop, the gateway is restarted in place on the untouched filesystem and no restore is armed. It refuses with 409 while a restore is in flight or a maintenance window is open, and while a restore is pending over the recovery point. After a successful checkpoint, the next activation restores the committed snapshot.

Restore outcome. sandbox.restoreBackup is bounded by RESTORE_BACKUP_TIMEOUT_MS. Because withTimeout only stops waiting, a timeout is recorded in restore-target.json as outcome: 'unknown', not as a failure. While that record is younger than GATEWAY_RESTORE_PHASE_BUDGET_MS, getRestoreInFlightReason blocks every restore (RestoreInFlightError, 409 on the admin route). A definitive failure is recorded as outcome: 'failed' and does not block. The in-lease checks (restore entry, restoreIfNeeded, restoreBackupById) re-read this under the lease, and restoreIfNeeded also re-checks the maintenance window after it waits for the lease.

Destructive guard. getPersistenceBusyReason is the single busy check for freeze (/control), POST /debug/destroy-container, and POST /debug/reprovision. It is busy while any maintenance window is open or a restore is in flight. Reprovision also re-checks after its checkpoint. A snapshot-only lease is not busy: an abandoned snapshot leaves no half-applied filesystem behind, whereas an abandoned restore does.

flowchart TD
  DC["Dispatch Worker cron<br/>backup control action"] --> RC{"runtime active?<br/>isRuntimeActive()"}
  RC -- no --> SKIP["skip (never wake)"]
  RC -- yes --> S["scheduleBackupIfNeeded"]
  S --> G{"credentials + interval + lock"}
  G -- no --> N["no-op"]
  G -- yes --> L{"backup-lock free?<br/>(non-blocking)"}
  L -- busy --> SKIP["scheduler skips this tick"]
  L -- free --> HOLD
  MAN["POST /api/admin/storage/sync"] --> W["acquire lease<br/>wait ≤120s w/ backoff"]
  HR["POST /debug/restart-openclaw"] --> CP["checkpoint first"]
  CP --> W
  W -- busy > 120s --> ERR["error (no snapshot)"]
  W -- acquired --> HOLD["lease held over legacy stage<br/>+ sandbox.createBackup"]
  HOLD --> A["Sandbox snapshot<br/>/home/openclaw"]
  A --> E["optional F4E1 encryption"]
  E --> P["commit backup-handle.json"]
  P --> REL["release lease (CAS tombstone)"]
  CP --> K["graceful kill + restore marker"]
  K --> R["restore committed snapshot<br/>on next activation"]
  RP["POST /debug/reprovision"] --> RCP["checkpoint first<br/>(or require existing backup)"]
  RCP --> RD["destroy instance<br/>409 unless restorable backup or force"]
  SR["POST /api/admin/gateway/restart"] --> SOFT["SIGUSR2 in-process reload<br/>no snapshot"]

Do not perform repeated hard restarts without verifying a successful checkpoint. The checkpoint protects only files inside /home/openclaw and only after the backup catalog has been committed.

Durable async backup/restore jobs + stable recovery point

POST /api/admin/storage/sync?async=1 and POST /api/admin/storage/restore/:id?async=1 no longer rely on request waitUntil() (Cloudflare only guarantees ~30s of background time after a response). Instead the route commits a job record to the per-tenant ActivityScheduler Durable Object (src/activity/backup-jobs.ts) and returns 202 { accepted: true, jobId }. The DO alarm performs the work and retries with a bounded lease (BACKUP_JOB_LEASE_MS) so a multi-minute snapshot survives isolate eviction and Worker redeploys. GET /api/admin/storage/jobs/:id reads the persisted status. The ActivityScheduler owns one alarm, so the DO merges the next automation wake and the next backup job and arms the earliest (scheduleCombinedAlarm); a queued job is never dropped when the activity schedule clears the alarm.

Every durable job runs one bounded step per alarm invocation. A Durable Object alarm handler has a hard 15-minute wall limit, while a snapshot is stage (≤10 min) + createBackup + config capture + prune — and the alarm also runs the boot, maintenance and activity queues. A single-invocation job could therefore be killed by the platform, stay running until its lease expired, and restart the whole operation from zero (the orphaned-archive class of incident). src/activity/job-steps.ts persists a cursor: { step, attempts, startedAt } on the job record; each alarm runs at most one step, commits the cursor, and re-arms one second later so the other queues get a fair turn.

JobSteps
snapshot (manual)repair → prepare → archive → finalize
snapshot (scheduled)repair → gate → prepare → archive → finalize
deprovisionrepair → prepare → archive → finalize → stop
restorerestore → launch
bootrestore → launch
maintenancepreflight → execute
  • repair brings the catalog back to a consistent state before any snapshot work: it sweeps archive objects that no catalog entry can use (the SDK writes backups/<id>/meta.json only after a successful upload, so an upload that lost its metadata write is invisible to both the reconciler and the retention prune — it would sit in the bucket forever, billed and unrestorable) and it encrypts the archives that are not encrypted at rest.
  • The lease covers one step, not the whole operation (STEP_LEASE_MS = 6 min). A Durable Object alarm has a hard 15-minute wall limit, so the previous 14-minute lease meant a step killed at minute 3 stalled the job for another 11 minutes before it was retried. Alarm invocations are serialized per object, so a healthy step is never duplicated.
  • A step recovered after its lease expired had an unknown outcome: the job says so (detail: 'step-interrupted', or 'step-deadline-exceeded' past the 12-minute soft deadline) and emits a job-step.interrupted event instead of silently replaying. Every step emits a job-step.completed / job-step.failed event with its duration — the measurement the step budget is derived from (max(configured, 2 x the last snapshot duration) for the cadence, 12 min for the per-step deadline).
  • prepare/archive/finalize are the phased snapshot (prepareSnapshot/archiveSnapshot/finalizeSnapshot in src/persistence.ts). The catalog entry is still committed in archive, immediately after encryption, so a crash before finalize leaves a restorable archive (and finalize re-adopts it if the catalog write was lost).
  • Each step takes the shared R2 lease on its own (withSnapshotLease); the stage is idempotent and archive re-checks the restore-needed latch, so a hand-off between steps cannot capture a half-restored container. A scheduled snapshot refuses to archive behind a pending restore; a manual/deprovision checkpoint deliberately still archives (without promoting), which is the documented recovery case.
  • The retry budget is per step (isStepExhausted) as well as per job, so a permanently broken step fails the job instead of consuming attempts that belong to healthy steps. A completed step is never re-run: a boot that is interrupted during launch resumes at launch, not at the destructive restore.
  • The legacy stage runs with maxAttempts: 1 inside a durable step (the step cursor owns the retry), so its worst case stays at the 10-minute LEGACY_STAGE_TIMEOUT_MS instead of 20 minutes.
  • The activity alarm now enqueues a scheduled snapshot instead of running scheduleBackupIfNeeded inline, so a 10-minute stage no longer shares an invocation with the automation wake.

Archives that are not encrypted at rest are visible and repaired. While a master key is configured, an archive whose meta.json does not carry encrypted: true (a legacy plaintext archive, or one whose upload finished while the encryption step never ran) is catalogued with unencrypted: true and queued in pendingEncryption. It is never dropped (it stays restorable) and never promoted to the recovery point; the durable repair step encrypts it in place and clears the flag. The dashboard shows a “not encrypted” badge for such an archive.

Scheduled cadence is a floor, not a promise. BACKUP_INTERVAL_MINUTES (default 5) is the minimum; resolveEffectiveIntervalMs stretches the effective interval to at least 2× the last observed end-to-end snapshot duration (recorded as lastSnapshotDurationMs in the catalog). Without that, a tenant whose snapshot takes 10 minutes would start a new one the moment the previous committed: continuous R2 lease contention, continuous container work, and a starved automation wake. A fast tenant is unaffected.

createSnapshot promotes a stable recovery point only after the SDK archive, optional F4E1 encryption, and the catalog commit have all succeeded (promoteStableBackup → stable-backup.json). A cold restore prefers that pointer and never restores a newer archive whose promotion was interrupted; retention (cleanupOldBackups) always keeps the stable entry, so a just-uploaded-but-unpromoted archive cannot evict the last committed recovery point. The pointer is also monotonic (an older or replayed archive never moves it backwards) and fails closed: a malformed stable-backup.json aborts restore and retention rather than falling forward to an unpromoted archive. Unit-tested in src/persistence.test.ts (the “stable recovery point promotion” block). The latest validated config is also mirrored to a fixed-key hot copy (stable/config.json), kept out of the version-history listing and 30-day prune.

A targeted restore (POST /api/admin/storage/restore/:id) force-promotes the restored archive (the exception to the forward-only pointer) so the next cold boot restores the same archive. ?setCurrent=0 (sync or ?async=1) threads { setRecoveryPoint: false } through restoreBackupById (and the durable restore job) to apply an archive without moving the recovery point — the next fresh container then reverts to the unchanged pointer. A restore also opens a cross-isolate maintenance/replacement.json window (operation restore) and stops the gateway right after taking the shared restore lease, closing the window in a finally; restoreIfNeeded and ensureGateway both refuse to run under it, so a refetch cannot restart the gateway and remount the latest recovery point mid-restore.

This makes the async workflow durable and the recovery point monotonic. It does not turn interval snapshots into zero-RPO replication: writes acknowledged after the last committed snapshot are outside the guarantee. Closing that gap requires a write-through storage path and is tracked in issue #377.

Sandbox → R2 Connectivity (outboundByHost)

Sandbox code can reach Worker bindings through a virtual hostname without any SDK/client library or credentials inside the container. @cloudflare/sandbox (via @cloudflare/containers) exposes the canonical per-host outbound handler:

// src/index.ts
Sandbox.outboundByHost = {
  [RELEASES_R2_HOST]: (request, env, ctx) => handleReleasesR2Request(request, env),
};
// catch-all: internet passthrough
Sandbox.outbound = async (request) => fetch(request);

The handler runs in the Workers runtime with full env bindings, so the sandbox never sees R2 credentials. Unmatched hosts fall through to the catch-all Sandbox.outbound.

Release bridge (releases.r2.internal). src/sandbox-r2-bridge.ts maps plain HTTP from the sandbox onto the tenant’s BACKUP_BUCKET under a fixed releases/ prefix (the bucket is per-tenant, so the prefix is already tenant-isolated):

MethodPathEffect
GET/JSON list of release objects
GET/<key>Stream an object (200) or 404
HEAD/<key>Size/etag metadata (200) or 404
PUT/<key>Upload an object (201)
DELETE/<key>Remove an object (204) or 404

Keys are sanitized against traversal before touching R2; uploads use a FixedLengthStream (bounded memory) when content-length is advertised.

Docs bridge (docs.r2.internal). src/sandbox-docs-bridge.ts exposes the tenant’s client documentation (customers/<slug>/docs/) for on-demand fetch by the Fouria Builder agent / sandbox tooling, reusing R2ClientDocsStore sanitization. The tenant slug is resolved via the shared resolveTenantSlug (src/sandbox-tenant.ts): an explicit customer-<slug> container id wins, otherwise it is derived from the per-tenant Worker env (CUSTOMER_SLUG, else WORKER_NAME / the always-provisioned WORKER_URL, with the tenant- / staging- prefixes and -dev suffix stripped), since production provisions the sandbox under the fixed id openclaw. GET / lists, GET/HEAD /<name> reads, PUT /<name> writes, DELETE /<name> removes. This lets the sandbox read docs without R2 credentials and without waiting for Worker-side materialization.

Software bridge (software.r2.internal). src/sandbox-software-bridge.ts exposes the tenant’s declared-software manifest (customers/<slug>/software.json — the same key the admin upload and the boot-time materialization use) so the dev-only Fouria Builder can read and declare packages without the Access-protected admin API (from the container, /api/admin/* answers 302 to the Access login page). GET / returns a discovery summary (present, path, containerPath, writeEnabled); GET/HEAD /manifest reads; PUT /manifest validates (the same validateSoftwareManifestInput as the route, with a 256 KiB cap) and replaces; DELETE /manifest removes. The slug comes from the shared resolveTenantSlug, so a sandboxed agent can only reach its own manifest. Writes are dev-only (isDevAgentsEnabled): a manifest entry is arbitrary shell executed inside the container at provisioning time, so production stays read-only through the bridge and operators declare software from the dashboard / admin API. The dev-only software-manifest workspace skill (packages/moltlazy/workspace/skills/software-manifest.ts) documents the bridge and is seeded for the Fouria Builder; because workspace skills outrank Skill Workshop skills, it shadows any Workshop copy of the same name.

Leads bridge (leads.r2.internal). src/sandbox-leads-bridge.ts exposes the global paso4-leads proposal bucket (the fouria-leads skill) over LEADS_BUCKET. The bucket folder layer is the tenant slug, and the bridge resolves the caller’s slug via the shared resolveTenantSlug in src/sandbox-tenant.ts (explicit customer-<slug> container id, else the per-tenant Worker env CUSTOMER_SLUG / WORKER_NAME / WORKER_URL). Every path the agent sends is relative to its own <slug>/ root — the handler prepends the slug, so a sandboxed agent cannot list, read, or write another client’s proposal. The only bucket-root object a tenant may read is the shared, proposal-agnostic build-pdf.sh (SHARED_LEADS_READ_KEYS); writes/deletes of it resolve under the tenant’s own folder. Because the bucket is global, the bridge refuses the bare fixed sandbox id (openclaw) as a fallback namespace when no env identity resolves: a worker that lost its identity env must not share openclaw/ with another misconfigured worker. Routes: GET / (or GET /?prefix=<p>) lists relative keys, GET/HEAD /<key> reads, PUT /<key> writes, DELETE /<key> removes. Keys reuse the shared sanitizeLeadsKey / sanitizeLeadsPrefix rules from leads/operations.ts. This is how the sandbox reaches proposal docs without a public route — the legacy wss://<worker>/api/leads/ws proxy (src/routes/leads.ts) is intercepted by Cloudflare Access on tenant hostnames (a per-tenant Access app shadows the *.fouria.io/<path> apps) and is now used only by the host-side E2E fixture (test/e2e/o_leads_proxy.txt, which authenticates with the CDP_SECRET persisted by server/start).

Binding requirement. LEADS_BUCKET must be provisioned as an r2_bucket binding (lerma instances.ex instance_bindings/2, config LEADS_R2_BUCKET, default paso4-leads). A plain-text var holding the bucket name makes env.LEADS_BUCKET a string, so bucket.list() throws inside the outboundByHost handler and the container sees an opaque ECONNRESET. All the R2 bridges validate the binding with isR2Bucket (src/r2-binding.ts) and wrap their handler body in a try/catch, returning a structured JSON 503/500 instead of resetting the socket.

R2 S3 credentials are deprecated for the container. buildEnvVars no longer forwards R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY into the sandbox: the bridges run in the Workers runtime with the BACKUP_BUCKET / LEADS_BUCKET bindings, so the container never sees credentials. The S3 credential remains a Worker binding for the Sandbox SDK presigned-URL backup path only.

SDK 1.0 (preview). The canonical static outboundByHost API is unchanged in the 1.0 preview, so these bridges run as-is on @cloudflare/sandbox@0.13.0-next; see the Sandbox SDK 1.0 section below.

Client Docs (dev-only admin surface)

Client documentation is managed from the Admin Client Docs page (/_admin/docs, ADMIN-only, dev instances only) backed by src/routes/api/fouria-docs.ts (dev-gated by isDevAgentsEnabled, so it 404s in production). It manages two tenant-scoped namespaces directly from the Worker — no sandbox/CLI round-trip:

  • Documentation — customers/<slug>/docs/ in BACKUP_BUCKET (markdown, text, and binary files like PDFs/images; content type from R2 metadata or the extension). Store: R2ClientDocsStore (listDocs / getDoc / getDocBinary / putDoc / putDocBinary).
  • Proposals (leads) — <slug>/ in the global LEADS_BUCKET (PDFs and assets), scoped to the caller’s tenant.

The SPA renders markdown client-side (marked + DOMPurify, sanitized) and opens PDFs with the browser’s native viewer through the /raw endpoints. Raw responses set X-Content-Type-Options: nosniff; HTML/SVG are forced to attachment with a sandbox CSP so client-authored files can never execute in the admin origin.

Versioning (DocsGitStore). Every doc write is committed to a Worker-side, git-like object store in BACKUP_BUCKET:

customers/<slug>/docs-git/objects/<blobSha>    content-addressed file bytes
customers/<slug>/docs-git/commits/<commitSha>  commit JSON ({ tree, parents, … })
customers/<slug>/docs-git/refs/heads/dev       mutable HEAD
customers/<slug>/docs-git/refs/promoted        promoted ref

The working namespace always mirrors HEAD. History is exposed in the dashboard’s commit-tree shape (GET /commits); POST /rollback/:sha restores an older commit as a new, non-destructive revert commit.

Backups (DocsBackupStore). POST /backup snapshots the working namespace to customers/<slug>/docs-backups/<id>/ (files + manifest.json), independent of the git store; GET /backups lists (retention keeps the newest 10), and POST /backups/:id/restore re-applies a snapshot as a new commit.

Versioned documentation channels (platform docs)

The platform docs site (webs/fouria-docs, served at fouria.io) is published per channel so an instance reads documentation and release notes that match the code it runs:

ChannelSourceURL
latestnewest v* release tag/latest/ (the root aliases /)
stagingdevelop head/staging/

buildEnvVars forwards FOURIA_DOCS_CHANNEL (latest only for a production-platform production instance; staging for the staging platform and every -dev environment — src/docs-channel.ts), FOURIA_DOCS_BASE_URL (https://fouria.io/<channel>), FOURIA_DOCS_FALLBACK_URL (https://fouria-docs.pages.dev/<channel>), and FOURIA_VERSION (the Worker bundle’s CalVer). The fouria-docs skill and the dev-only Fouria Builder/Q&A personas read those variables, then confirm the build via <base>/versions.json before quoting release notes. versions.json records the channel, source commit, and component versions; release-notes/ is compiled from each component’s CHANGELOG.md, so a channel page only lists what is in that channel’s code.

Deployment (.github/workflows/deploy-pages.yml): both channels are built from their own checkouts and assembled by scripts/build-deploy-tree.ts — the release tag supplies / and /latest/, develop supplies /staging/ — and the combined tree is deployed to fouria-docs on a release and on every develop docs change. The staging project previews the staging channel at its root.

Sandbox SDK 1.0 side-by-side foundation (SandboxV1)

Alongside the default legacy Sandbox path, the Worker carries an additive, not-routed Sandbox SDK 1.0 foundation so the side-by-side move can be built without a big-bang cutover. It is a foundation only — no tenant is on it and it carries no durability claim.

  • Class — SandboxV1 (src/sandbox-v1.ts), a user-owned DurableObject that drives this.ctx.container (the 1.0 model). Exported from src/index.ts together with DirectoryBackupGateway (needed for the DO to pass ctx.exports.DirectoryBackupGateway to DirectoryBackup).
  • Registration (class + migration only) — wrangler.jsonc migration v7 (new_sqlite_classes: ["SandboxV1"]). There is no deploy surface: the SandboxV1 DO binding and its scheduling_policy: "durable_object" container application were removed in every environment (durable-agents audit finding F9), because they served no traffic while every deploy still built/pushed an extra image and registered an extra application. Dockerfile.v1 and the class are kept; restoring a container entry plus the binding is the first step of the cutover (see docs/plans/sandbox-v1-scaffold.md).
  • Default routing unchanged — the legacy Sandbox class + getSandbox() still serve every tenant. No route selects SandboxV1; there is no customer routing change and no cutover.
  • Package / image pairing — root catalog installs @cloudflare/sandbox@1.0.0 (this DO) and @cloudflare/sandbox-legacy@0.13.0-next.776.1 (the still-default legacy class, imported via the -legacy alias). Dockerfile.v1 uses node:24-trixie-slim and COPYs the sandbox-shim helper from the matching cloudflare/sandbox:1.0.0 image (the shim tag must track the package version); the legacy base image no longer applies because 1.0 removes the in-image SDK server.
  • TTL (derived, capped at 6h) and keep-alive — resolveInactivityPolicy() reuses resolveSleepAfter(): non-billable instances get a finite idle timeout (staging 15m, dev 2h) clamped to the 6h platform maximum accepted by setInactivityTimeout(). Billable production sets the 6h maximum and renews it from a recurring alarm on the same DO (keep-alive purpose): SDK 1.0 has no “never stop” timeout, so timeoutMs: null would mean stop shortly after the DO goes inactive, not stay awake. The renewal alarm keeps the DO active (which resets the inactivity timer), re-applies the timeout in case the DO incarnation lost it, and never starts a stopped container. The snapshot cadence and a queued conversion share the same alarm slot, so clearing one purpose never deletes another’s alarm. There is no SANDBOX_SLEEP_AFTER binding, and the provisioned instance size is carried in CONTAINER_INSTANCE_TYPE (a durable_object container chooses its size in ctx.container.start(), including every wake).
  • Real DirectoryBackup with F4E1 at-rest wrapping — the additive DO instantiates the real DirectoryBackup (from @cloudflare/sandbox@1.x) against ctx.container + ctx.exports.DirectoryBackupGateway with a tenant-scoped sandbox-v1/<encoded-do-name>/ R2 prefix in BACKUP_BUCKET. The SDK writes one flat plaintext object per record (<prefix><id>.tar.zst); the Worker then post-encrypts it in place to <prefix><id>.tar.zst.enc (F4E1/AES-GCM) plus a readable <prefix><id>.meta.json sidecar, and deletes the plaintext canonical object. Restore transiently decrypts .enc back to the canonical key for the SDK, then always deletes that plaintext in a finally block; delete removes all three objects (strict for explicit user deletion, best-effort on cleanup). Plaintext therefore exists only in the short post-upload window and the restore/decrypt window. The master key stays Worker-only (env.BACKUP_ENCRYPTION_KEY, never passed to the container); the record’s sha256/size remain plaintext and the SDK DirectoryBackup.restore() verifies them on restore. With BACKUP_ENCRYPTION_REQUIRED=true a missing key/meta fails closed before any upload or restore.
  • Still no scheduling, conversion, or cutover — Worker-owned periodic snapshot scheduling + retention is not implemented (no approved defaults wired; the intended targets are a 5m snapshot interval, retaining the latest interval snapshots plus a 24h rolling window and daily 30d history, once a scheduler exists). There is no 0.13 .sqsh.enc → DirectoryBackup conversion, no customer cutover, and no default-routing change: the legacy Sandbox class + getSandbox() still serve every tenant, no route selects SandboxV1, and the complete restore-before-Gateway path is not wired for the v1 line. This is not a final durability/no-loss guarantee. See docs/plans/sandbox-v1-scaffold.md.

Sandbox SDK runtime lines

The default tenant runtime is the legacy Sandbox class, installed via the @cloudflare/sandbox-legacy catalog alias of npm:@cloudflare/sandbox 0.13.0-next.776.1, paired with the docker.io/cloudflare/sandbox:0.13.0-next.776.1 image. The Worker package and the container image must stay on the same line — the stable and 1.0 control protocols are mutually incompatible.

The side-by-side SandboxV1 Durable Object uses the stable @cloudflare/sandbox 1.0.0 package (see the Sandbox SDK 1.0 section above). The intended runtime is selected by the SANDBOX_RUNTIME binding (legacy default | v1); until V1 implements the full Sandbox surface the legacy class keeps serving and resolveSandboxRuntime() reports the requested V1 as pending (see src/sandbox-runtime.ts).

Every tenant-facing site that needs a Sandbox obtains it through the single runtime acquisition boundary acquireSandbox() in src/sandbox-contract.ts — the request middleware (src/index.ts), the task buffer (src/tasks/index.ts), the ActivityScheduler Durable Object (src/activity/scheduler.ts), and the per-customer gateway launcher (src/gateway/process.ts). The boundary resolves the runtime but always returns the legacy stub for now (a v1 request is still pending), preserves the container identity (openclaw for the single-tenant instance, customer-<slug> for a per-customer gateway) and the legacy idle policy (defaults to buildSandboxOptions(env)), and never starts a container — getSandbox() returns a Durable Object stub, so acquiring a runtime never wakes a sleeping tenant or touches the v1 DO namespace. This is the single flip point for the cutover.

Sandbox v1 migration gate (S1)

The first cutover slice (S1, contract C1/C2/C10) adds the authority and identity gate, not the conversion (S2 owns convertToV1 and the DO-owned restore). The gate is src/sandbox-migration/platform.ts:

  • canMigrate(env) is true only when the trusted WORKER_PLATFORM binding is exactly staging. Absent, empty, whitespace, production, dev, local, STAGING, and a request carrying Host: staging-<slug> or X-FourIA-Role: root all refuse.
  • migrationRefusal(env) resolves the canonical tenant with tenantSlugFromEnv (never ctx.containerId, never resolveTenantSlug’s context fallback) and refuses null and the reserved client slug.
  • The /control entry runs the gate in a middleware before the sandbox stub is acquired, so a refused migrate performs no R2 write and no DO/container addressing. It returns 403 MIGRATION_PLATFORM_FORBIDDEN, 409 MIGRATION_SLUG_UNRESOLVED, or 409 MIGRATION_RESERVED_CLIENT. An allowed staging request is served by the queue-only entry (S5): it enqueues the durable conversion on the SandboxV1 DO and answers 202 (queued/running), 200 (committed + recordId) or 502 (SANDBOX_V1_CONVERSION_FAILED); the read-only migration-status action reports absent/queued/running/committed/failed and can never start work. See § “Sandbox v1 conversion orchestration” below.
  • The gate is migrate-only: freeze, unfreeze, backup, checkpoint, checkpoint-park, and resume keep their existing behavior on every platform and on the reserved client tenant.
  • sanitizeCustomerSlug rejects the default sentinel in every resolver branch (C10), and SANDBOX_V1_CUTOVER defaults to false — unselected tenants stay byte-identically on the legacy runtime.

Lerma’s provisioning and upgrade paths now both add the platform bindings (WORKER_PLATFORM from cf_environment, SANDBOX_V1_CUTOVER=false) and, on the staging platform only, the SandboxV1 DO binding (Lerma.Cloudflare.SandboxV1). The per-tenant v1 container application is created by Containers.deploy_script_and_container/1 from the same platform/image predicate, with the v1 image supplied as a digest-pinned named image (ctx.container.images.sandbox) because the durable_object policy takes no application-level image. The upgrade path also re-asserts the tenant’s full binding set, so it can rebuild a script that no longer exists, and an upgrade rollback never deletes the live tenant’s script. Verified on staging (2026-11-05); the staging Lerma was then restored to :latest-pre, so a promotion from develop Lerma would drop the v1 bindings until the branch lands. See sandbox-v1-staging-observations.md.

Sandbox v1 conversion (S2)

SandboxV1.convertToV1() (src/sandbox-v1.ts) owns the data movement and the conversion barrier; the Worker never runs the primitives in a request context.

  • Intent (C5, C6). src/sandbox-migration/intent.ts writes sandbox-migration/<tenant>/intent.json before the first converting step. Reads are tri-state (absent / present / unknown); only an explicit absent permits legacy. present or unknown means v1, permanently.
  • Lease (C4). src/sandbox-migration/lease.ts is a per-tenant R2 lease: create-only acquire (etagDoesNotMatch: '*'), etag-CAS takeover of an expired/released tombstone, owner-scoped renew/release, and fail-closed parsing. The TTL is a liveness bound only.
  • Conversion (C7). convertToV1() acquires the lease, writes/validates the intent, opens the legacy recovery point (stable-backup.json, F4E1-decrypting data.sqsh.enc when needed), streams the plaintext squashfs into the v1 container, extracts it with unsquashfs -f -d /home/openclaw, takes a v1 DirectoryBackup (tar+zstd/1), writes sandbox-migration/<tenant>/conversion.json, and publishes sandbox-migration/<tenant>/stable.json. Dockerfile.v1 installs squashfs-tools.
  • Barrier (C8). verifyConversionRecord() completes only when the stable pointer names a shape-valid committed record. Readiness is a separate input (S3/S4); startGateway(env) is the DO-owned owner restart and waits for a real /health, but S2 only calls it when the caller supplies the launch env.
  • No fallback (C5). Once the intent exists, the Worker acquires the v1 runtime (S3 below); it never silently falls back to the legacy gateway. ensureGateway on a legacy stub still throws SandboxV1RuntimePendingError as a defense-in-depth check.

Sandbox v1 serving (S3 / OCM-1)

Runtime selection is the durable conversion intent, resolved once per request in the global middleware (resolveRuntimeIntent). acquireSandbox(env, id, options, runtime) then returns the matching runtime:

  • legacy → the 0.12 getSandbox() stub (unchanged, byte-compatible).
  • v1 → V1SandboxClient (src/sandbox-v1-client.ts) wrapped around the SandboxV1 DO. unknown → 503 SANDBOX_V1_RUNTIME_UNKNOWN before any acquisition. Control actions on a converted tenant fail closed with 503 SANDBOX_V1_CONTROL_UNSUPPORTED (S4 owns them) rather than acting on the wrong container.

Transport. The DO exposes a fetch() entrypoint: the Worker client sends the request to the DO stub with X-FourIA-Sandbox-Port, and the DO rewrites it to http://localhost:<port> before calling ctx.container.getTcpPort(port).fetch(...). The entrypoint (not an RPC method) is what carries a WebSocket upgrade Response back to the Worker. It never starts the container: a stopped container answers 503 SANDBOX_V1_SLEEPING, so a probe cannot wake an idle tenant.

Process RPC. ExecProcess handles cannot cross RPC, so the DO keeps them and exposes v1Exec / v1ProcessOutput / v1ProcessWait / v1ProcessSignal / v1ProcessState / v1ListProcesses (output is decoded to strings inside the DO). V1SandboxProcess presents that as the SandboxProcess shape, so startShell / runShell / waitForProcess and the CLI-backed admin routes keep working.

Lifecycle. ensureGateway detects the v1 client and delegates to ensureV1Gateway → DO v1EnsureGateway, which (1) adopts a gateway already answering /health, (2) restores the committed v1 DirectoryBackup when the container holds no tenant state (OPENCLAW_RESTORED=true for the launch), and (3) runs start-openclaw.sh start as the single supervised owner, polling a real /health before returning. The DO is the serialization point, so there is no second owner even across Worker isolates. killGateway delegates to v1KillGateway.

Health. getHealthSnapshot branches on the v1 client: v1Status() is the non-waking ctx.container.running read (a passive probe on a stopped runtime is sleeping, HTTP 200 at the route), readiness is a real /health response, 503 until ready, and /healthz stays the sub-second non-waking liveness read.

Not yet implemented on the v1 line (each fails with SANDBOX_V1_UNSUPPORTED_OPERATION instead of an undefined-method crash): createBackup / restoreBackup / directory backups (PER-2), file tools such as writeFile / readFile / mkdir (OCM-3), and the lifecycle/eviction hooks stop / stopIfIdle / onActivityExpired (OCM-2). Durable boot jobs and scheduling still acquire the legacy stub (PER-3).

Sandbox v1 gateway proxy attribution

OpenClaw attributes proxy-shaped traffic (any Forwarded, X-Forwarded-* or X-Real-IP header) against gateway.trustedProxies before Gateway auth. It accepts the request only when the source address is trusted and the forwarded client headers resolve to a non-loopback client; otherwise the Gateway-authenticated routes answer 403 {"error":{"type":"proxy_attribution_required", …}}.

The Worker always rebuilds X-Forwarded-For / X-Real-IP from CF-Connecting-IP (see rebuildForwardedClientHeaders in src/index.ts), so the second condition holds. The first one depends on the ingress path, which a tenant cannot configure:

RuntimeGateway sees the request fromIn trustedProxies?
legacythe Cloudflare Sandbox container bridge (10.1.0.x)yes — moltlazy unions 10.0.0.0/8
v1the ctx.container.getTcpPort() ingress owned by the platformno (unknown to any tenant config)

That made every proxied request on a converted/born-v1 tenant fail (the reported symptom: proxy_attribution_required on all dev instances deployed with the v1 runtime), including WebSocket upgrades and the admin-RPC reads, because gateway.trustedProxies can only list addresses the tenant can know.

src/gateway/proxy-attribution.ts owns the fix. A request the Gateway refuses cannot be attributed as a proxy, but a direct-remote request is always attributed (OpenClaw only rejects proxy-shaped traffic it cannot resolve), and it is served with the same token/device auth. SandboxV1.fetch() therefore:

  1. probes attribution once per gateway incarnation with a GET / carrying the same rebuilt forwarded headers (X-Forwarded-For: 203.0.113.1). / is the Gateway-authenticated route that fails with the rejection, so its 403 is the definitive signal. A probe that cannot reach a still-booting Gateway leaves the mode unknown (traffic keeps its headers) and is retried by the next forward;
  2. when the Gateway refuses attribution, strips every forwarded header from the request before forwarding it, so OpenClaw attributes it as direct-remote;
  3. caches the verdict for the Durable Object’s lifetime and re-probes after a gateway restart / container destroy (v1KillGateway, v1Destroy, startGatewayOwner), so a healed gateway.trustedProxies restores client-IP fidelity without a Worker deploy.

Requests that carry no forwarded headers (container health probes, CLI reads) skip the probe entirely, and the probe result is single-flighted so concurrent requests at boot cannot fan out probes. Structured event: sandbox.v1.proxy_attribution.direct (warn) — the direct fallback is active for this gateway.

Trade-off: in direct mode the Gateway attributes the container-internal source instead of the real client IP (rate-limit subject and audit client IP). That is acceptable for a single-tenant gateway whose credential is the shared token, and it is strictly better than a 403. When attribution is accepted, nothing changes.

Provisioning an instance directly on Sandbox v1

A tenant can skip the legacy conversion entirely: Lerma provisions it on v1 (“born v1”), the Worker serves it on SandboxV1 from the first request, and the tenant onboards inside the v1 container. The decision is durable in R2 and written once, at provisioning time.

Marker contract. sandbox-migration/<canonical-slug>/runtime.json in the tenant’s BACKUP_BUCKET:

{
  "version": 1,
  "tenant": "<canonical-slug>",
  "runtime": "v1",
  "mode": "fresh",
  "provisionedAt": "<ISO8601>"
}
  • Lerma writes it create-only, before the script deploy; the Worker only ever reads it (src/sandbox-migration/runtime-marker.ts). There is no env var, no endpoint and no Worker-side write path for it.
  • The read is tri-state like every other migration record (readJsonObject): a transport error, malformed JSON, a non-matching tenant, a version other than 1, another runtime, a converted mode, a missing/empty tenant or a provisionedAt that is not ISO 8601 with an explicit zone (Z or ±hh:mm) is unknown, never absent.
  • readFreshRuntime(env) is the route-facing convenience: it resolves the canonical tenant with tenantSlugFromEnv and reads the marker; an unresolved identity is unknown, and no BACKUP_BUCKET is absent (a marker cannot exist without the bucket, so local/dev instances without R2 keep the historical behavior).

Resolution matrix (resolveRuntimeIntent, src/sandbox-migration/intent-gate.ts):

PlatformConversion intentMarkerSANDBOX_RUNTIMERuntime
not staginganyanyanylegacy (unchanged; the platform is the only migration authority)
stagingpresent/unknown + C8 record+pointer completeanyanyv1
stagingpresent/unknown + barrier incompleteanyanypending
stagingabsentpresent (valid fresh marker)anyv1
stagingabsentunknownanyunknown (fail closed)
stagingabsentabsentv1pending (fail closed)
stagingabsentabsentunset/legacylegacy

The conversion intent wins over a marker: a tenant that was converted (and historically also carried a marker) stays behind the C8 record+pointer barrier.

Fail-closed pending. SANDBOX_RUNTIME=v1 with no marker means the instance asked for v1 but was never provisioned for it (the marker write and the binding disagree). The Worker answers the existing 503 SANDBOX_V1_CONVERSION_PENDING and never falls back to legacy — a legacy boot would create the exact legacy state the tenant is not supposed to have. There is no “assume the marker” path and no automatic recovery: re-provision or remove the binding.

Control plane. Neither migration action applies to a fresh tenant:

  • POST /control X-FourIA-Control: migrate → 409 { ok: false, action: 'migrate', code: 'MIGRATION_ALREADY_V1', error: 'tenant is provisioned on SandboxV1; there is nothing to convert' }, without addressing the DO or enqueuing (an unreadable marker → 503 SANDBOX_V1_RUNTIME_UNKNOWN, since converting a tenant that may already be v1 would be destructive). The platform/slug gate still runs first, and a converted tenant has no marker, so it keeps its idempotent enqueue behavior.
  • POST /control X-FourIA-Control: migration-status → 200 { ok: true, action: 'migration-status', state: 'absent', mode: 'fresh' } without touching the DO (never enqueues, never wakes). A converted tenant answers mode: 'converted' alongside its durable state.
  • backup / checkpoint / freeze / unfreeze / checkpoint-park / resume run the existing v1 branches (snapshot through the DO, park/resume), so a fresh tenant is a first-class v1 tenant for the rollout machinery.

Boot and persistence. SandboxV1.restoreTenantStateIfNeeded() returns false when the container holds no config and the recovery point is pointer-absent with no conversion record, so a fresh tenant onboards (start-openclaw.sh start runs the setup body) with OPENCLAW_RESTORED unset.

The first recovery point is taken right after that first successful boot. v1EnsureGateway dispatches runInitialSnapshot() detached (registered with ctx.waitUntil, never awaited), so the tenant no longer waits for the first idle-cadence tick to become durable:

  • When. Only for a tenant with no recovery point: verifyRecoveryPoint must report pointer-absent and conversion.json must be absent. A converted tenant, a tenant that already snapshotted (cadence, backup, checkpoint-park), an unreadable pointer and any inconsistent state are left alone — the boot never snapshots blindly over an existing point. A parked tenant is skipped too (its park already committed the rollout record) and a container that is not running is never woken.
  • Guards. A single-flight promise (#initialSnapshotRun) makes concurrent boots (the owner run and the adopted-gateway path both schedule it) fold into one attempt, a settled flag short-circuits the check once there is nothing to do (at most one R2 read per DO incarnation), and takeOwnedSnapshot() gives the idle cadence and this attempt one shared owner so they cannot snapshot the directory concurrently — the loser skips and the cadence keeps its alarm armed.
  • Best effort. Any failure is caught and logged as sandbox.v1.initial_snapshot_failed (bounded message); the boot stays ready and the container keeps running. A failure is retried on a later boot: a new DO incarnation retries immediately, the same incarnation after INITIAL_SNAPSHOT_RETRY_MS (10 min), so a broken backup cannot be hammered on every request. A success logs sandbox.v1.initial_snapshot with recordId/size/took.

From then on the normal v1 persistence path applies: v1CreateSnapshot writes the DirectoryBackup, catalogs it, advances sandbox-migration/<slug>/stable.json (pruning by retention), and that pointer is the recovery point everything restores (rollout park, restore by record id, cold boot). A conversion record without a pointer still refuses to onboard over it.

Residual window. The snapshot is detached, so a tenant that dies between /health becoming ready and the archive committing (seconds to minutes on a large directory, and longer on a failed + retried attempt) still has no durable copy; the cadence remains the backstop for that sliver, and readiness is never delayed by the backup.

Policy. Provisioning a fresh v1 tenant is limited to the staging platform (including dev instances on it), like every other v1 lane: the resolution matrix returns legacy outside WORKER_PLATFORM === "staging", so a marker written on a production tenant is inert. The reserved client slug never migrates and Lerma refuses to write a marker for it; the Worker deliberately does not special-case client here — a marker is authoritative for serving.

Sandbox v1 idle snapshot cadence (PER-2)

A converted tenant’s container belongs to SandboxV1 (ctx.container), and raw ctx.container has no eviction hook: the legacy onActivityExpired → archive → stop path does not exist on SDK 1.0. The platform stops the container after its inactivity timeout (applyInactivityTimeout, derived from resolveSleepAfter), so without a cadence everything written since the last committed record would be lost. The DO therefore owns a periodic snapshot:

  • Arming. startContainer() arms the cadence after the display/outbound setup succeeds (best-effort: a storage error is logged as sandbox.v1.snapshot.cadence_arm_failed and never fails the boot). Arming writes a durable purpose marker (snapshot:cadence-pending) plus ctx.storage.setAlarm(now + interval). The DO has one alarm slot and a queued conversion arms now + 1s, so arming never postpones an earlier alarm — the marker, not the alarm time, tells the handler what to do. A restarted DO keeps the alarm (it is durable) and startContainer() re-arms on the next boot.
  • Interval. resolveSnapshotIntervalMs(env, inactivityTimeoutMs): SANDBOX_V1_SNAPSHOT_INTERVAL_MS (ms, 0 disables) → else the configured BACKUP_INTERVAL_MINUTES → else 1h. The result is capped at half the instance’s inactivity timeout when one is applied, because a snapshot due after the container has already idle-stopped is pointless (and the cadence must never wake it): staging (15m) → 7.5m, dev (2h) → 1h. The legacy 5-minute default is deliberately not inherited — it was tuned for the interval-gated cron path, and a 5-minute v1 cadence would snapshot back-to-back.
  • The alarm turn (alarm() → runConversionAlarm() then runSnapshotCadence(); neither throws out of the handler):
    1. disabled → clear the marker and stop;
    2. #restoring (a manual restore owns the directory) → skip this turn, keep the cadence armed, so one overlap cannot silently switch the loss bound off;
    3. container stopped → never wake it: clear the marker (the next startContainer() re-arms);
    4. the recovery point (sandbox-migration/<tenant>/stable.json) is newer than the interval → skip (this dedupes with the ActivityScheduler’s own interval-gated v1 snapshot);
    5. otherwise v1CreateSnapshot({ trigger: 'backup' }), logging sandbox.v1.snapshot.cadence with recordId/took;
    6. re-arm for the next interval.
  • Teardown. v1Destroy() clears the marker and the alarm — unless a conversion is queued on the same slot (deleting that alarm would strand it).
  • Parking. The cadence is deliberately park-agnostic: a parked tenant is drained, so a fresh snapshot is unnecessary but harmless (the catalog write is idempotent and touches R2 only).

Failures log sandbox.v1.snapshot.cadence_failed and re-arm; the previous recovery point stays intact and the container keeps running.

Sandbox v1 restore by record id (PER-2)

POST /api/admin/storage/restore/:id?setCurrent=0 on a converted tenant applies a catalogued v1 record instead of the legacy .sqsh archive:

  • the route opens the durable maintenance window (restore) before the DO touches the container, so a request arriving mid-restore cannot restart the gateway under the swap (ensureGateway’s v1 branch honors the window);
  • SandboxV1.v1RestoreRecord stops the single gateway owner, restores the record’s directory into HOME_DIR (the SDK swaps it atomically), and — by default — advances the stable pointer so a cold boot restores what the operator chose (?setCurrent=0 applies the record without promoting it);
  • an unknown or unreadable record fails closed (404) and never touches the container; a restore already in progress is refused by an in-DO flag;
  • the Worker then boots the restored state (OPENCLAW_RESTORED semantics) and reports gatewayReady. A boot failure does not undo the restore: the next request retries the boot, and the caller must not retry the destructive step.

v1 restores are synchronous: there is no legacy backup-job queue for the record format, and the SDK restore is bounded by the archive size.

Sandbox v1 outbound virtual hosts (OCM-1 / BIL-1)

The container reaches the Worker-owned services through virtual hosts instead of the public internet: releases.r2.internal, docs.r2.internal, leads.r2.internal, software.r2.internal (R2 namespaces resolved with the Worker’s own binding), otel.internal (OTLP ingest from the tenant’s exporter), cdp.internal (the container’s local CDP proxy forwarding Browser Rendering commands to the tenant’s Sandbox DO through src/sandbox-cdp-bridge.ts → src/cdp-session.ts — the browser credential never enters the container), and ai.internal (the inference proxy in src/sandbox-ai-bridge.ts). The map lives in src/sandbox-outbound.ts (SANDBOX_OUTBOUND_ROUTES), the single source of truth for both runtimes:

  • the legacy Sandbox class installs it as its outboundByHost statics;
  • SandboxV1 registers each host with ctx.container.interceptOutboundHttp(host, ctx.exports.SandboxOutboundGateway({ props: { host } })) in startContainer(). SDK 1.0 has no class statics — an intercept belongs to the container instance and disappears when it is replaced — so the routes are installed on every start, including a restarted DO that finds its container still running. A Worker image without the export logs a warning and leaves the container on direct egress instead of failing the boot.

Without this, a converted tenant silently loses OTLP export (AI cost attribution), the R2 bridges and the agent browser (cdp.internal); the container has no other way to resolve those names.

AI Gateway inference proxy (ai.internal)

AI Gateway capacity is account-scoped (20 gateways on the paid plan), so tenants share pools rather than owning a gateway. A gateway total is therefore never a client’s spend, and the account-scoped inference credential must not sit in a tenant container.

The route is opt-in (AI_GATEWAY_PROXY=true). It only takes effect when the tenant’s primary model resolves under cloudflare-unified-billing/*; the official cloudflare-ai-gateway provider ignores CF_AI_GATEWAY_CUSTOM_DOMAIN and would send the derived token to Cloudflare, which rejects it. Confirm the active model ref (Agent Studio’s model pane, or openclaw models status) before enabling it on a tenant. When enabled, tenant inference is routed through the Worker:

  1. buildEnvVars sets CF_AI_GATEWAY_CUSTOM_DOMAIN=ai.internal and hands the container CLOUDFLARE_AI_GATEWAY_API_KEY / CLOUDFLARE_API_TOKEN set to HMAC-SHA256(MOLTBOT_GATEWAY_TOKEN, "fouria-ai-proxy/v1") (src/gateway/ai-proxy-token.ts). The unified-billing plugin then calls https://ai.internal/compat/....
  2. The sandbox intercepts that host and runs handleAiRequest (src/sandbox-ai-bridge.ts) in the Worker.
  3. The handler verifies the derived token, resolves the tenant from the Worker env (resolveTenantSlug, never the request), maps /compat/* → /ai/v1/*, and forwards to https://api.cloudflare.com/client/v4/accounts/{account}/ai/v1/... with the real credential.
  4. It always sets cf-aig-metadata to the trusted tenant_id, environment and platform (plus the plugin’s openclaw_run_id), and forces cf-aig-gateway-id from the trusted binding. A tenant cannot override either by editing its config.

Consequences:

  • Attribution. Lerma’s development-expense report sums only log entries whose metadata.tenant_id matches the client and whose environment is one of that client’s non-billable environments; legacy entries without metadata are counted as ai_gateway_unattributed_requests instead of being assigned to a client. Cloudflare’s Analytics API can group the same metadata with metadataValue(key: "tenant_id"), and spend-limit rules can split budgets by it.
  • No Access credential pair. CF_ACCESS_TOKEN, CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET are no longer forwarded to containers. An Access-protected custom domain needs a per-tenant service token and service tokens do not produce cf.user_id, so the pair could not deliver the identity attribution it was meant for.
  • Rollout to deployed instances. Lerma persists the opt-in on instances.ai_gateway_proxy and re-deploys an already-running instance’s current bundle from the instance page (AI proxy toolkit action → Instances.set_ai_gateway_proxy/2). Only the binding changes; the image, gateway and container application are untouched, and the instance status is preserved. Upgrades carry the flag forward.
  • Custom-domain opt-out. Setting CF_AI_GATEWAY_CUSTOM_DOMAIN explicitly keeps the direct path with the real credential; that domain must not be Access-protected.
  • Tests. src/sandbox-ai-bridge.test.ts pins the identity, metadata, gateway-forcing and path-mapping contract; src/gateway/env.test.ts pins the derived-token wiring and the removal of the Access credentials.

Sandbox v1 rollout park (SEC-3)

A v1 tenant can now be parked for a container replacement, with the same guarantees as the legacy lane and one deliberate difference in ordering:

  • POST /control { X-FourIA-Control: checkpoint-park } on a converted tenant:
    1. switches live sessions to read-only and waits for their turns to end (drainLiveSessions over the v1 gateway WS bridge; the previous modes are recorded durably with recordPendingSessionModes),
    2. commits the recovery point with v1CreateSnapshot — fail-closed: a failed snapshot returns 502 SANDBOX_V1_PARK_SNAPSHOT_FAILED and the rollout never starts, because the replacement container has no other copy,
    3. only then opens the durable park (maintenance/replacement.json { parked: true }) and arms the exec lock. A stopped container is already parked (skipped: 'not-running').
  • The park blocks writes through the existing layers: tenantParkGuard refuses mutating /api/admin/* and /debug/* requests with 409 TENANT_PARKED, the /control entry refuses every action except resume and checkpoint-park, and assertTenantExecAllowed stops Worker-initiated exec. Reads stay available.
  • resume clears the park record, unlocks exec, requests the boot through the v1 DO (which restores the committed record) and applies the recorded session modes once the gateway answers a real /health probe; if it is not ready yet the response says sessionModes: "pending" and the durable map stays for the next boot. Resuming an unparked tenant is a no-op that never wakes it.

killGateway remains process-scoped on v1 (v1KillGateway signals the gateway process); nothing in the park path destroys or recreates the container.

Sandbox v1 boot, owner and repair (OCM-2)

The durable lanes that run outside a request must resolve the tenant’s runtime themselves — they have no request-scoped sandbox to inherit:

  • The ActivityScheduler resolves the runtime on every operation. Its sandbox() helper calls resolveRuntimeIntent and then acquireSandbox(..., runtime). Before this, the scheduler always acquired the legacy client: after a conversion, an alarm-driven wake or a scheduled snapshot would have booted the old container, archived the legacy filesystem and advanced the legacy stable-backup.json — silent divergence from the v1 recovery point. pending/unknown tenants are skipped (logged as activity.sandbox.skipped), never guessed.
  • Scheduled snapshots on v1 write a v1 record. A snapshot backup job on a converted tenant calls SandboxV1.createSnapshot({ trigger: 'backup' }) (one call: DirectoryBackup + catalog + pointer + retention) instead of the legacy stage → archive → finalize pipeline. restore/deprovision jobs on v1 are reported as skipped with an explicit reason until those paths exist.
  • restoreIfNeeded no-ops on v1. The DO restores the committed DirectoryBackup (or onboards a fresh tenant) inside v1EnsureGateway/launchRestoredState. The legacy state probe (exists) is intentionally unsupported on the v1 client, so running the legacy restore step would read unknown and wedge the durable boot job. The boot job’s restore step therefore returns immediately for v1 and the launch step performs the restore; this is what lets the automation wake reuse the durable boot job instead of blocking the alarm on ensureGateway.
  • The DO’s launch supervisor mirrors the legacy boot guards. v1Exec returns a process handle, so startGatewayOwner races the supervised launch’s exit against the /health poll: a non-zero exit before the port opens is a terminal launch failure reported in seconds (a zero exit stays ignored — the “lost the startup-lock race” path is judged by the health probe). On failure it tails /tmp/gateway-stdout.log; when the text matches a SQLite corruption signature it queues the durable doctor-repair job (gateway-launch-state-database) and fails with a retryable error, exactly like the legacy path. Relaunching cannot fix a malformed page, and the job re-queues the boot when the repair succeeds.

Sandbox v1 snapshots and the recovery point (PER-2)

On the legacy lane a snapshot is a .sqsh archive catalogued by backup-handle.json; on v1 it is a DirectoryBackup record. The tenant needs the same guarantees, so the v1 lane has its own catalog under the tenant’s migration scope:

ObjectPurpose
records/<id>.jsonOne catalog entry per committed record (size, sha256, format, trigger)
stable.jsonThe stable pointer naming the committed recovery point (shared with the conversion)
conversion.jsonThe migration artifact; doubles as the first catalog entry’s fallback
  • v1CreateSnapshot() (DO). backupDirectory({ dir: '/home/openclaw' }) writes the record and post-encrypts it in place with F4E1 (failing closed when BACKUP_ENCRYPTION_REQUIRED=true and no key is set). The catalog entry is written immediately after the archive, so a killed invocation leaves a findable record instead of an unlisted orphan, then the stable pointer advances and retention prunes oldest-first beyond BACKUP_MAX_VERSIONS (default 3) while never removing the record just taken. A failed snapshot leaves the container running and the previous recovery point intact. A stopped container is skipped, never started: a 1.0 instance loses its disk on an inactivity stop, so ensureRunning() here would snapshot an empty /home/openclaw and advance the recovery point over real data. v1CreateSnapshot returns { state: 'skipped', reason: 'not-running' }; /control freeze/backup/checkpoint report the skip, and the scheduler’s snapshot job skips the cycle.
  • /control routing. backup, checkpoint and freeze on a v1 tenant are served by the DO (the legacy .sqsh orchestration never runs for a converted tenant). freeze takes a snapshot and only then destroys the container — a failed snapshot keeps the tenant running (SANDBOX_V1_FREEZE_SNAPSHOT_FAILED) because the container is the only live copy until the record is committed. unfreeze is a no-op (the next request cold-boots from the recovery point). Other control actions still answer 503 SANDBOX_V1_CONTROL_UNSUPPORTED.
  • Cold boot restores the pointer, not the conversion artifact. A snapshot advances stable.json, so restoreTenantStateIfNeeded restores whatever the pointer names (records/<id>.json, falling back to conversion.json for pre-catalog records) and then launches with OPENCLAW_RESTORED=true. A conversion record without a pointer is an inconsistent C8 state and is refused — onboarding over it would silently discard a converted tenant.
  • The C8 barrier is the pointer→record pair. verifyRecoveryPoint is the single read used by the runtime gate and the boot path: no pointer is pointer-absent, a pointer without a readable record is record-absent, an unreadable record is record-unknown, and a pointer naming a different record is pointer-record-mismatch. Only a matching, shape-valid pair is complete.

Sandbox v1 conversion orchestration (S5)

POST /control with X-FourIA-Control: migrate (forwarded by the Dispatch Worker’s /_control/:slug/migrate) starts the legacy → v1 conversion. The work streams the legacy squashfs, extracts it in the v1 container and uploads a v1 DirectoryBackup — minutes of I/O, far longer than a Worker request lives (its background continuations are cancelled shortly after the response). The control entry is therefore queue-only: it enqueues a durable job on the SandboxV1 DO and answers in milliseconds; the conversion runs on the DO alarm.

This is deliberate rather than incidental: the idle-snapshot cadence alarm (§ Sandbox v1 idle snapshot cadence) fired on the live tenant lane and committed record 4437c278-… with no operator action, which is the proof that alarms fire on the Workers-for-Platforms tenant script. Queueing removes the request’s wall-clock, CPU and disconnect exposure from a destructive data migration.

Contract

OutcomeStatusBody
queued or in flight202{ ok: true, action: 'migrate', state: 'queued' | 'running' }
already committed200{ ok: true, action: 'migrate', state: 'committed', recordId }
durable job recorded failed502{ ok: false, action: 'migrate', state: 'failed', error, code: 'SANDBOX_V1_CONVERSION_FAILED' }
SandboxV1 binding missing, or the DO unreachable503{ ok: false, action: 'migrate', error, code: 'SANDBOX_V1_UNAVAILABLE' }

The staging/reserved-slug gate (403/409) is unchanged and still runs in middleware before any DO access, so a refused migrate performs no R2 write and no container addressing. The route logs a structured sandbox.v1.conversion.requested event (state, attempt, recordId; no secrets). A transport failure is reported as 503 and not as failed: nothing was recorded, so the same call is safe to retry, whereas failed is a durable, terminal state an operator must act on.

Reading the status without starting a conversion (migration-status)

migrate is an idempotent enqueue (it starts work from absent/failed), so a poll that reused it would convert a tenant just by asking. The read-only action migration-status (X-FourIA-Control: migration-status, forwarded by the Dispatch Worker as /_control/:slug/migration-status) exists for polling:

Tenant runtimeOutcome
legacy (no conversion intent)200 { ok: true, action: 'migration-status', state: 'absent' }, no v1 DO addressed
pending / v1200 { state: 'queued' | 'running' | 'committed' | 'failed', recordId?, error? }
v1 binding missing, or the read fails503 { ok: false, action: 'migration-status', error, code: 'SANDBOX_V1_UNAVAILABLE' }

It calls only v1ConversionState() — never v1RequestConversion(), so it cannot enqueue, never starts work and never wakes the container. The one write it may perform is the lost-alarm re-arm (ensureConversionArmed() on a non-terminal job), which arms the alarm slot but launches nothing. error (the job’s bounded failure message) is included only for state: 'failed'. The action is allowed through the runtime middleware for pending and v1 tenants and is not part of the staging-only gate: reading a conversion state is not an action.

Durable job

  • The job is the DO storage marker migration:convert-request ({ requestedAt, attempt }); the state the control plane polls is migration:convert-status (queued → running → committed | failed, plus the 1-based attempt). Both survive eviction and code updates.
  • v1RequestConversion() is idempotent: committed returns as-is (including when only the R2 conversion record survives a DO storage reset), an in-flight job returns queued/running, and only absent/failed enqueues new work — failed starts the next attempt.
  • Arming never postpones an earlier alarm: the conversion and the cadence share the DO’s single alarm slot, so armConversionAlarm() only writes when there is no pending alarm or the pending one is later. Conversion arming wins.
  • alarm() runs the conversion first, then the cadence, each in its own try/catch so one failing never starves the other, and finally re-arms a job that is still non-terminal (a backstop for a crash inside a turn).
  • A lost alarm is healed on the read paths: v1ConversionState() (the poll), v1RequestConversion(), v1Status() and v1EnsureGateway() all call ensureConversionArmed(). It re-arms only when the durable marker exists and the state is non-terminal, so a queued job can never sit forever; a terminal state with a leftover marker is cleared instead. The read paths themselves never wake the container — only the conversion turn does, by design.
  • Concurrency: #conversionRun is the single owner, mirroring #gatewayStart. A second alarm turn (or a read path that armed one) folds into the in-flight run instead of starting a second destructive conversion. The conversion’s own lease/intent rules keep a crashed attempt idempotent on the next turn.
  • Ordering is the crash contract: running is durable before the conversion starts, committed is durable before the marker is dropped and before the gateway launch is attempted, and a failure records failed (bounded message: 500 chars) without throwing. A launch failure therefore can never retract a commit — the next v1EnsureGateway retries the boot, whereas failed would re-run the destructive conversion. The container environment is rebuilt inside the DO from its own bindings (no secrets in DO storage).
  • There is no synchronous conversion entry point: it was removed from the DO, the RPC surface and the client, so a request cannot await the migration.

The conversion still depends on a committed legacy recovery point (stable-backup.json); a tenant without one fails closed with a recorded reason and must be checkpointed first. Because the intent is a point of no return, the runtime gate requires the C8 barrier (conversion record + stable pointer) before a tenant serves on v1: an intent without it is pending and the Worker answers 503 SANDBOX_V1_CONVERSION_PENDING (only the idempotent migrate entry stays reachable), so a failed conversion can never wake a fresh empty v1 container over the legacy recovery point.

Key differences from the 0.12 stable line, all funneled through src/gateway/sandbox-exec.ts:

  • No startProcess / execStream / getLogs(). sandbox.exec(argv) starts a process and resolves once it is launched, not once it finishes; it never implies a shell. startShell(sandbox, script) wraps a shell command line in ['/bin/bash', '-lc', script] and returns the handle; runShell(sandbox, script) waits and buffers {stdout, stderr, exitCode}; outputOf(proc) / collectLogs(proc) read an existing handle.
  • listProcesses() returns ProcessStatus[] (.state, argv .command, .startedAt) with no methods. findExistingGatewayProcess fetches a live handle via getProcess(id) and returns a GatewayProcess wrapper (.state, .command string, .refresh(), .waitForPort(), .kill(), .getLogs()).
  • waitForProcess delegates to waitForExit({ timeout }) and kills the process on a local wait timeout (the SDK timeout cancels the wait only).
  • Busy-DO hardening for gateway reads. A contended container-control channel can hold an exec launch for the SDK default (120s) and stall the Sandbox DO. Read routes therefore bound their sandbox work and degrade instead of hanging past the client timeout:
    • ensureGateway(waitForReady:false) (every API read route) is bounded by ENSURE_GATEWAY_READ_TIMEOUT_MS (30s) and returns null on expiry; the caller serves its normal “not ready” payload. Cold-start (waitForReady:true) is exempt.
    • A short-TTL positive readiness result (GATEWAY_READINESS_CACHE_TTL_MS, 5s) collapses concurrent read-path probes onto one; it is cleared on kill/restart/recovery.
    • The gateway port probe (isGatewayPortOpen) is bounded by PORT_PROBE_TIMEOUT_MS (10s) with a container-side curl -m, and marks the expected closed-port exit (curl 7 = connection refused) with reportNonZeroExit: false (operation gateway.port-probe) so the normal “gateway down, start it” outcome is not logged as a sandbox.process.failed error on every cold start — matching the gateway.health-probe behavior.
    • Expected CLI non-zero exits are not logged as failures either: runShell (via reportNonZeroExit) and waitForProcess (same option) let a call site mark an exit it handles as normal. The knowledge wiki status probe uses it (operation: 'knowledge.wiki-status') because an uninitialized vault exits non-zero and the handler reports initialized:false. Timeouts always log.
    • Route flows that poll the container wrap their whole flow in a budget (/api/admin/devices 25s, /debug/container-config 20s).
  • No transport selection. SANDBOX_TRANSPORT was removed; the preview uses RPC automatically.
  • SandboxOptions.labels was removed upstream (worker identity is no longer attached at container start).

Deploy cutover: because the protocols are incompatible, a production deploy must roll containers in one step — wrangler deploy --containers-rollout=immediate — with rollout_active_grace_period at 0. In-flight processes/terminals stop and pre-deploy process IDs become invalid.

Debug tools on Sandbox v1 (ROOT diagnostics)

The ROOT /debug/* surface (DEBUG_ROUTES=true, ROOT only) is the operator’s escape hatch when a tenant misbehaves. A converted tenant is served by V1SandboxClient (SandboxV1 DO), not the legacy Sandbox, so each route was audited against the v1 surface (exec, getProcess, listProcesses, containerFetch, wsConnect, killGateway, destroy, status, isRuntimeActive) and adapted where the legacy call had no v1 equivalent.

RouteWhat it callsv1Notes
GET /debug/versionrunShell('openclaw --version' / 'node --version')✅ worksexec → waitForExit → output() are all implemented on the v1 client.
GET /debug/processeslistProcesses()✅ worksv1 returns serializable ProcessStatus-shaped entries (.state, argv .command, no methods). The list is the DO’s tracked map, so it is empty after a DO eviction — not a container inventory.
GET /debug/processes?logs=truegetProcess() + logs✅ worksA finished process is read with output() (the v1 handle has no replayable logs() stream). A running process is never read (logs_error: 'process running — live log stream disabled').
GET /debug/gateway-api?path=containerFetch on the gateway port✅ worksv1 fetch forwards to ctx.container.getTcpPort(); a stopped container answers 503 SANDBOX_V1_SLEEPING instead of waking.
GET /debug/cli?cmd=startShell + waitForProcess + outputOf + status() + exitCode✅ worksSame CLI semantics as legacy; DEV_MODE instances still need OPENCLAW_GATEWAY_TOKEN forwarded (already done).
GET /debug/logs[?id=]gateway tail / finished-process output✅ worksNo id → findExistingGatewayProcess → getLogs() tails /tmp/gateway-stdout.log. With id → output() fallback when logs() is unsupported.
GET /debug/gateway-logsfindExistingGatewayProcess + getLogs()✅ worksAlways the file tail (a live SDK log stream never settles). The response now carries path and runtime.
GET /debug/envWorker bindings only✅ worksAdds sandbox_runtime_actual (the durable per-tenant lane this request was served by) next to the env-level sandbox_runtime switch.
GET /debug/container-configexec cat …/openclaw.json + outputOf✅ works20s end-to-end budget unchanged.
POST /debug/sqlite-integrityrunShellResilient (inline) / ActivityScheduler (durable)✅ worksThe durable path resolves the tenant’s runtime itself (scheduler.sandbox()), so the repair job runs on v1 too.
GET /debug/sqlite-integrity/jobs/:idActivityScheduler read✅ worksDO-only. .../jobs/:id/ws is the streaming variant.
GET /debug/gateway-tokensecret binding✅ worksNo container access.
POST /debug/stop-gatewayfindExistingGatewayProcess + killGateway✅ workskillGateway delegates to v1KillGateway (process-scoped, with a best-effort pkill sweep).
POST /debug/restart-openclawkillGateway + cache resets✅ worksv1 branch: maintenance window → kill → resetStartupCache + clearPersistenceCache + resetV1GatewayStart → close window; the next request relaunches. No checkpoint and no restore-needed: a v1 restart destroys nothing and the marker is legacy-only. Same response fields + runtime: 'v1'.
POST /debug/destroy-containerdestroy()✅ worksv1 destroy tears down the container and clears the DO’s tracked processes/alarm; the route also clears the persistence/startup caches and the isolate start dedupe.
POST /debug/reprovisionv1 snapshot + C8 recovery-point check + destroy()✅ worksThe checkpoint is v1CreateSnapshot({ trigger: 'backup' }) (bounded by REPROVISION_CHECKPOINT_TIMEOUT_MS); “restorable” is verifyRecoveryPoint (pointer→record), not backups/<id>/meta.json. Response carries recordId and runtime: 'v1'; still fails closed (409) without a restorable copy.
POST /debug/backup/:modev1CreateSnapshot (v1) / scheduleBackupIfNeeded (legacy)✅ worksOn v1 the snapshot is one DO call (DirectoryBackup + catalog + pointer + retention) and needs no gateway, so the route never boots one. sync returns recordId.
POST /debug/rescue-containercontainerFetch probe + Containers API + destroy()⚠️ partialDiagnosis and destroy work, but v1 containerFetch never wakes a stopped container: a sleeping tenant reads as unreachable. Use it for an orphaned/Error container, not to inspect a sleeping one.
POST /debug/r2-put?key=R2 binding✅ worksNo container access.
POST /debug/sleepdurable deprovision job (legacy) / stop()❌ not supportedv1 has no stop primitive (the tenant sleeps on the platform inactivity timer). Answers 503 { code: 'SANDBOX_V1_SLEEP_UNSUPPORTED' } instead of a 202 for work that never happens.

Operational caveats:

  • Access blocks headless /debug/* calls on tenant hostnames (the per-tenant Access app answers 302 to the login page). Drive these routes from a ROOT browser session on https://<tenant>/_admin/ (the Debug Tools cards), or from Lerma’s service token — not from a container-side curl.
  • DEBUG_ROUTES=true is Worker-local and /debug/* is ROOT-only; neither is provisioned by Lerma for a client instance, so this is a staging/dev/operator surface.
  • Process ids are per DO incarnation. After an eviction/redeploy the v1 process map is empty (/debug/processes shows nothing, /debug/logs?id= 404s) even though the gateway is healthy — use /debug/gateway-logs (the file tail) instead.
  • Nothing here is destructive beyond what it says: restart/destroy/ reprovision never touch R2 on v1 except reprovision’s own snapshot (which advances the recovery point) and rescue’s Containers-API image repair.

Container Image Build Profiles (dev vs production)

The Dockerfile takes a BUILD_PROFILE build arg (default production):

  • production (default) installs only xz-utils — the slim image (~1.36 GB) used by production and by E2E.
  • dev additionally installs the optional PDF/document tooling (pandoc + weasyprint — a large Python/cairo/pango stack, ~760 MB) that the dev-only Fouria Builder fouria-leads skill needs (apps/fouria/skills/agent-only/fouria-leads/SKILL.md; the Builder is injected only when OPENCLAW_DEV_MODE=true).

The profile is selected per wrangler environment via image_vars (Cloudflare’s build-time variables, equivalent to docker build --build-arg):

EnvironmentBUILD_PROFILEImage
env.staging (dev/test)devwith pandoc/weasyprint
env.productiondefault (production)slim
default / local Vite dev (local + E2E)default (production)slim

There is a single image definition (apps/fouria/Dockerfile); CI (test.yml) builds it with no build args when it pushes unstable-<sha>, and the release CD (deploy-fouria.yml) promotes that same image by retagging it, so released images are the slim production profile. The E2E basic (PR) and production (develop) profiles both provision the slim production image (only instance type/env differ). To build a dev image locally:

docker build --build-arg BUILD_PROFILE=dev -f apps/fouria/Dockerfile .

Sandbox Recovery (stale + orphaned)

When the platform terminates a container behind a live Sandbox DO (idle recycle, maintenance, SIGTERM), every RPC into it fails with the SDK’s HTTP error! status: 500 wrapper — and retrying against the same instance fails forever, bricking the tenant. gateway/sandbox-recovery.ts classifies these dead-container errors (isStaleContainerError) and recovers (recoverStaleSandbox): destroy the instance, reset the startup-probe and persistence caches. The next RPC auto-provisions a fresh container and the normal cold-start flow runs (R2 restore → start-openclaw.sh → gateway).

A container destroyed out-of-band (Cloudflare Containers dashboard “Destroy”, failed operator force-stop) is a second failure class: the DO is alive but points at a placement that no longer exists, so RPCs fail with the SDK’s retryable admission errors (no container instance available, maximum number of running container instances exceeded) instead of the 500 signature. Those are legitimate during a cold start, so they are not treated as stale on their own. isContainerUnavailableError + trackProvisioningFailure escalate them to a recovery once the streak reaches ORPHANED_INSTANCE_MIN_FAILURES and spans ORPHANED_INSTANCE_GRACE_MS (120s — longer than the SDK’s ≈110s provisioning budget, so a slow-but-healthy cold start is never cancelled mid-provision). The streak is cleared whenever a sandbox RPC succeeds. maybeRecoverSandbox unifies both classes for callers.

Recovery is wired into three failure points in gateway/process.ts: findExistingGatewayProcess (listProcesses failures), checkGatewayHealth (exec failures), and the ensureGateway startShell path (recover + retry once). It is cooldown-guarded (STALE_RECOVERY_COOLDOWN_MS, 60s) and never fires for deploy-window OperationInterruptedError. Permanent image errors still require the rescue tool (rescueContainer, see OPERATOR-GUIDE §5.1). Logs use the [SandboxRecovery] prefix.

Transient transport drops (retry, don’t reprovision)

A third failure signature is not a dead container: Network connection lost. (and similar socket/WebSocket transport messages). It appears when the DO↔container connection drops while the container is alive — most visibly while the Gateway is applying a config/plugin reload (e.g. adding a model provider in the Control UI), which restarts the Gateway and briefly tears down the container RPC/WS transports.

isContainerTransportError (gateway/sandbox-recovery.ts) classifies these, and isStaleContainerError explicitly excludes them (even when the SDK wraps the message in its sandbox.<op> error prefix) so a transient drop never destroys a live instance. ensureGateway instead retries with a short backoff: the port probe (isGatewayPortOpenResilient, up to TRANSPORT_RETRY_ATTEMPTS) and the startShell spawn (once). Before this, a reload that dropped the transport mid probe fell straight through to start-openclaw.sh → Failed to start process → a 502 to the user, even though the Gateway came back seconds later. The retry keeps the request alive across the reload window.

A Durable Object code update (a normal Worker/DO redeploy, or a Lerma redeploy of the tenant script) resets the control session between the DO and its container even though the container itself is not replaced. The cold start then loses the transport on the supervised port wait: the atomic start-openclaw.sh start process (config setup + gateway under one lock) keeps running, but waitForPort() on the stale handle rejected and previously aborted the cold start — killing a gateway that would have completed (observed as Network connection lost.). The atomic start path has no separate setup-process wait, so only the gateway port wait reattaches via getProcess(id) and resumes (waitForPortResilient, gateway/sandbox-exec.ts), bounded by the shared STARTUP_TIMEOUT_MS deadline. The generic waitForExitResilient helper remains in sandbox-exec.ts for the legacy/boot-smoke supervised waits. Only a non-transport error, an un-reattachable process, or an exhausted deadline still fails the cold start.

The openclaw doctor diagnostics run through the same transport-resilient path: runShellResilient, which retries the exec launch (startShellResilient) and reattaches the wait (waitForExitResilient). Previously a transient The operation was aborted drop on the doctor startShell surfaced as a hard 500 { ok: false, error: "The operation was aborted" }; it now reattaches to the process and returns the structured diagnostic instead. GET /api/doctor still runs inline this way; POST /api/admin/config/fix and POST /debug/sqlite-integrity now enqueue a durable maintenance job on the ActivityScheduler DO (activity/maintenance-jobs.ts + activity/maintenance-executor.ts) and return 202 { accepted, jobId }, because a large integrity check or doctor repair can exceed a Worker request’s lifetime (the old 30s/90s shell timeouts surfaced as “timeout in under 2 minutes”). The DO alarm runs the command under a 14-minute lease with bounded retries (runMaintenanceOperation, still runShellResilient internally) and stores the bounded result on the job record, polled via GET /debug/sqlite-integrity/jobs/:id / GET /api/admin/config/fix/jobs/:id. Both endpoints fall back to an inline run when the ACTIVITY binding is absent (local dev / tests). The doctor repair runs openclaw doctor --fix --non-interactive (the supported command for index-only integrity_check corruption — missing or wrong-count index entries — which verifies the table data, preserves a backup, and REINDEXes) and quiesces the gateway first: runMaintenanceOperation opens the cross-isolate maintenance window (beginMaintenance(..., 'doctor-repair')), kills the gateway so the repair can take exclusive state ownership, then closes the window so the next request re-launches the gateway.

Restore → repair ordering (restore-config-pending.json). A restore replaces the container’s writable layer but launches nothing, so a config repair applied around a restore is silently overwritten by the next re-apply of the recovery point (observed on tenant-tarjetero-dev: a config fix was lost on the next gateway restore). Every successful restore (restoreIfNeeded and restoreBackupById) now writes a durable restore-config-pending.json marker. The durable boot job and the durable restore job launch the gateway on the restored state through the shared launchRestoredState helper (src/post-restore.ts); the admin sync/async-fallback restore route ends there too, so every restore path has the same ordering guarantee. The helper calls ensureGateway({ waitForReady: true }) and then captureRestoredConfigVersion — writing configs/<ts>.json and stable/config.json, so the restored config becomes the newest and stable config version and the dashboard’s diff/history reflects what is actually live. The marker is cleared only after that capture. A gateway-launch failure never re-runs the destructive restore: restore jobs stay succeeded with detail: "gateway-launch-pending" and the sync route returns gatewayReady: false; the pending marker drives the next boot / maintenance tick. A doctor-repair job checks, before running: an active restore job, an open maintenance window / in-flight restore (getPersistenceBusyReason), or a pending restored-config marker. In any of those cases it defers (MaintenanceJobDeferredError): the job keeps its attempts and queue position (60s re-check) instead of failing its three attempts while the restore runs, and executes only after the gateway is up on the restored state.

Automatic repair when the launch is blocked by a corrupt state database. A warm R2 restore can bring back a damaged state/openclaw.sqlite (or an agent openclaw-agent.sqlite). OpenClaw’s schema preflight reads PRAGMA user_version before anything else, so the Gateway dies within seconds with Error: database disk image is malformed (readSqliteUserVersion → inspectStateDatabaseSchema) and exits — no amount of relaunching helps. Two things made that a multi-hour loop where the loading page sat on “Starting OpenClaw Gateway”:

  1. The crash is now detected immediately. The readiness wait races a third signal, watchGatewayEarlyExit: the atomic cold start execs the Gateway, so a non-zero exit of the supervised process before the port opens is a terminal launch failure. Previously the Worker waited out the whole shared STARTUP_TIMEOUT_MS budget (≈5 min) before even reading the diagnostics. A zero exit is deliberately ignored — that is the “lost the startup-lock race to a healthy owner” path, decided by the existing port/lock adoption check — and transport/plumbing errors from waitForExit (a DO code update resets the control session) park the watcher instead of failing a healthy boot.
  2. The documented repair is queued. When the launch diagnostics match a corruption signature (gateway/state-db.ts: database disk image is malformed, file is not a database, SQLITE_CORRUPT, …), ensureGateway enqueues the durable doctor-repair maintenance job and throws GatewayStateDatabaseRepairError (retryable). The job carries reason: 'gateway-launch-state-database', which makes it (a) skip the post-restore gateway gate in preflight — waiting for a gateway the corruption prevents is a permanent deferral; the restore-safety checks stay — and (b) re-queue the durable boot when the repair succeeds, since the launch itself is what failed and no request would retry it. Re-requesting is deduped (MAINTENANCE_JOB_BUSY → keep the active job), so repeated failed launches cannot pile up repairs. Without the ACTIVITY binding (local dev) the launch reports no-scheduler and the operator runs Repair Environment.

The loading page follows it: /api/startup/ws pushes repair (the DO’s active maintenance job) and the page renders “Repairing the OpenClaw state database (openclaw doctor –fix, attempt N)…”, suppressing the slow-step notice — the container’s own startup-progress.json still names the launch step the repair is recovering from, which is exactly the stale “Starting OpenClaw Gateway (311s)” message.

Job status is pushed over a WebSocket — GET /api/admin/config/fix/jobs/:id/ws(and the /debug/sqlite-integrity/jobs/:id/ws variant) — instead of the dashboard polling the JSON endpoints above every 2s (one request + Worker log line per tick for the whole multi-minute repair). The streamer (routes/api/maintenance-ws.ts) reads the durable record from the ActivityScheduler DO, sends a frame only when it changes, and closes 1000 once the job is terminal; the dashboard’s subscribeMaintenanceJob falls back to a slow HTTP poll only while the socket is down. While a job is active the streamer also calls the DO’s rearm() self-heal, so a job whose alarm was lost is scheduled again immediately.

The shared alarm is written through an optimistic-concurrency guard (armWithGenerationGuard): the ActivityScheduler owns one alarm, so three job enqueues, the warm-path schedule refresh, and the alarm handler’s own trailing re-arm can interleave at await points. A stale arm that read “no pending job” could execute setAlarm(later)/deleteAlarm() after a just-enqueued job’s setAlarm(now) and strand the job until the later time — observed as a doctor-repair job enqueued at 12:57:30 that only began at 13:02:11 (exactly the previous boot job’s 14-minute lease). The guard bumps a stored generation and abandons a superseded arm’s write, so the newest arm (fresh read) owns the alarm.

While the control session reconnects, the loading page’s GET /api/startup/ws stream treats a transport-drop snapshot as transient: it re-sends the last known state (or a synthetic starting view) and backs off exponentially (STARTUP_WS_POLL_INTERVAL_MS → STARTUP_WS_MAX_BACKOFF_MS, 30s) instead of re-probing — and re-logging the same error — every 5s. A successful probe resets the cadence.

The manual lever is POST /debug/reprovision (DEBUG_ROUTES, ROOT): it checkpoints the live container first and fails closed — unless it just committed a snapshot or a restorable backup already exists it refuses with 409 rather than silently destroying tenant state; {"force":true} is the explicit override for an already-unrecoverable container. The checkpoint is only attempted when isRuntimeActive() reports a live runtime (so an idle container is never woken) and is time-bounded (REPROVISION_CHECKPOINT_TIMEOUT_MS); the restorability check HEADs the newest catalog entries’ meta.json rather than trusting the entry count. On success it destroys the instance, clears the startup/persistence caches, and resets the orphan streak without ever contacting the gateway — so it works when the container is entirely gone. Compare POST /debug/restart-openclaw (gateway-aware, checkpointed) and POST /debug/destroy-container (teardown only, used by E2E).

Activity-Driven Wake (ActivityScheduler)

Automations (“jobs”) run inside the Gateway, so the Gateway must be running for a schedule to fire. Historically a Worker cron trigger woke every tenant on every tick (expensive and since removed). src/activity/ replaces it with a per-tenant activity clock:

  • ActivityScheduler Durable Object (SQLite-backed, binding ACTIVITY) — stores a mirror of the tenant’s automation schedule and owns a single alarm. Deliberately a separate DO from the SDK Sandbox DO, which already uses its own alarm for sleepAfter.
  • Mirror — warm paths (GET /api/health, only when readiness.ok) read openclaw automations list --json and push the earliest nextRunAtMs into the DO. The DO throttles this to once per 5 min, and it never wakes a sleeping container.
  • Mirror bootstrap — the first mirror must not wait for a dashboard poll: the durable boot job mirrors the schedule right after launchRestoredState (activity.schedule.mirrored). A converted tenant that never opens the dashboard therefore still has a wake plan before it sleeps.
  • Alarm (activity/alarm.ts) — fires CRON_WAKE_AHEAD_MINUTES (default 10) before the next job. On legacy it calls ensureGateway({ waitForReady: false }) (fast; the schedule read retries until the gateway serves). On v1 it enqueues the durable boot job (deduped) and returns { ready: false } instead of blocking the alarm turn on a restore + launch; the boot job’s launch step mirrors the schedule when the gateway is ready, and a 60s fallback check covers a lost mirror. No CLI read runs against a booting container, and the read is gated on the non-waking isRuntimeActive() read. No enabled jobs → alarm cleared → the container sleeps. A due-but-pending job is woken once per occurrence: wokenFor survives mirror writes and resets only when the gateway reports a different next run, so a wake-ahead cannot turn into a 60s wake/CLI loop while the job is pending. A woken occurrence that does not advance backs off 5m → 30m; a failed wake/read keeps the schedule and backs off (bounded), it never deletes the only trigger a sleeping tenant has. Every activity re-arm goes through the shared combined alarm, so it cannot postpone a boot/maintenance/backup job.
  • Non-waking diagnostics — the alarm’s memory watch checks isRuntimeActive() before its /proc exec. On v1 that exec would start a stopped instance (the disk is gone), so a skipped scheduled snapshot or a failed wake can never warm a sleeping tenant by accident.

Net: healing/wakeup is independent of user traffic, gated on real job activity, and free when idle. Traffic may cold-start the container; jobs get a pre-warmed, healed one. Requires the ACTIVITY binding in wrangler.jsonc and in the lerma provisioner (tenant_provisioner.ex resource_bindings) + the new_sqlite_classes migration (dispatch_namespace.ex @do_sqlite_classes).

Change-Aware Backups

Scheduled snapshots are gated on whether tenant state changed. Detection is on by default; the watch loop is opt-in:

FlagEffect
BACKUP_CHANGE_DETECTION (true default)Phase 1 (cost). The scheduled snapshot path (scheduleBackupIfNeeded) runs a container-side scan first. Unchanged state ⇒ no squashfs, no legacy staging, no lease, no upload. Only an explicit false/0/off disables it.
BACKUP_CHANGE_WATCH=true + BACKUP_WATCH bindingPhase 2 (RPO). The per-tenant BackupWatch DO checks every BACKUP_CHANGE_CHECK_MINUTES (2) and requests a durable snapshot from ActivityScheduler once a change has been quiet for BACKUP_CHANGE_DEBOUNCE_MS (60 s) and the floor (BACKUP_INTERVAL_MINUTES) has elapsed.

Detection is default-on because a snapshot is not a cheap probe: it runs mksquashfs over the whole tree, uploads the archive, and re-streams it through the Worker for F4E1 encryption. Measured on fouria-backup-tarjetero-dev, two snapshots 37 minutes apart differed by 0.08 % (348 KB of 428 MB) — every tick that re-uploaded it paid for the full archive. Only an explicit opt-out turns the gate off, so a tenant that omits the var still gets the saving.

The scan itself costs one find, so the trade is deliberately lopsided: an unnecessary scan is negligible, an unnecessary snapshot is not.

Eviction compensates. Because a quiet tenant now takes no scheduled snapshot, its newest archive legitimately ages past the idle window — which would make TenantSandbox.onActivityExpired() re-archive the whole tenant once per sleep cycle. The hook therefore runs the same scan and treats unchanged as coverage, stopping inline (see Backup before deprovisioning).

Scan (src/change-detection.ts). Runs find over the real state roots (/home/openclaw, /root/openclaw/.openclaw, /root/clawd; the home directory is symlink-only for state, and find does not follow symlinks). It prunes the transient legacy stage and .npm. It compares against the baseline marker /tmp/.fouria-last-backup and prints unchanged, changed, or active (changed within the debounce window). Only entries inside the roots count (-mindepth 1), so a root’s own mtime, which moves when the legacy stage is created, cannot read as a change. Known limitation: a pure deletion with no accompanying write is not detected until the next write in the same tree, because find -newer sees mtimes, not removals. Every ambiguous outcome is changed: a missing baseline, no roots, a non-zero exit, a timeout, or a thrown error.

Baseline. /tmp/.fouria-pending-backup is stamped before a snapshot captures state and promoted to the baseline only after the catalog commits. A failed snapshot leaves the baseline untouched. This runs on every snapshot, whether or not a flag is set, so enabling a flag later has a valid baseline.

A restore forgets the baseline. Restored files keep their archived (older) mtimes, so a -newer <baseline> scan would call the replaced tree unchanged and skip the first post-restore snapshot. Both successful restore paths (restoreIfNeeded, restoreBackupById) call resetChangeBaseline, which removes both markers so the next gate takes a snapshot unconditionally.

Archive size and restorable TTL

Two createBackup options in src/persistence.ts decide how large an archive is and whether it stays restorable. Both matter more than they look: the archive is uploaded by the container, re-read and re-encrypted by the Worker, downloaded again on restore, and stored BACKUP_MAX_VERSIONS times, so every byte is paid for maxVersions + 2 times.

  • SNAPSHOT_EXCLUDES — passed to mksquashfs as wildcard patterns (the SDK normalizes ** away and warns). Excludes .npm, *.log, *.tmp, .cache, and workspace/*node_modules*. It deliberately does not exclude plugin extensions/<id>/node_modules (a restore has no reinstall step of its own — start-openclaw.sh only installs when the extension directory is missing, so stripping the deps leaves a plugin that loads and then fails at import), SQLite databases, or session transcripts. Opt-in (BACKUP_ARCHIVE_EXCLUDES, default false). This is a fidelity trade, not a pure win: the archive is the recovery tier, and an archive restored without those paths no longer matches the filesystem a Tier 1 workspace snapshot restores (or the pre-eviction tree). The default keeps the archive a complete image; enable it per instance once you know the tenant regenerates those paths.
  • SNAPSHOT_ARCHIVE_TTL_SECONDS (10 years) — the SDK enforces the TTL at restore time (doRestoreBackup refuses an archive past createdAt + ttl with BackupExpiredError), so it is a restorability gate, not a storage-lifecycle hint. The previous 7-day value silently made a retained recovery point unrestorable — cleanupOldBackups prunes by count, so an idle or frozen tenant could keep maxVersions archives that no boot could apply, and isUnrecoverableArchiveError then dropped the stable pointer and failed the boot closed. The durable repair step heals existing tenants: extendArchiveTtl rewrites meta.json for every archive below the minimum, preserving dir/createdAt/encrypted/encryptionVersion, and reports the ids as ttlExtended / ttlFailed from POST /api/admin/storage/reconcile. The restore path heals its own archive (ensureArchiveRestorable, called right after restore-target.json is published in both restoreIfNeeded and restoreBackupById): the repair step only runs inside a snapshot job, so a tenant that cannot boot could never reach the repair that would make it bootable — the wedge observed on staging-client-dev, where all three catalogued archives (7-day TTL) expired on 2026-10-29 and every boot then failed in ~250 ms on the expiry check after its container was replaced. The SDK’s expiry message (Backup <id> has expired (created: …, TTL: …s)) is now also classified as an unrecoverable archive, so a metadata rewrite that cannot take effect drops the dead entry instead of retrying it forever.

Compression is still the SDK default (lz4, chosen for speed, not ratio); moving to zstd trades container CPU for bytes and is tracked separately.

Watch (src/backup-watch/). watch.ts is the pure state machine. durable-object.ts is the adapter, and arm.ts holds gating and policy with no DO imports, so warm paths can call it without an import cycle.

  • Never wakes a container: liveness is the non-waking isRuntimeActive(). A not-running tenant is disarmed and not scanned.
  • Keepalive guard. Every scan is an exec that renews the container’s sleep timer. A tenant unchanged for BACKUP_CHANGE_IDLE_DISARM_MS (10 min, below the staging 15 min sleep window) is disarmed, and re-arming never resets the idle clock. The watch can therefore delay a quiet container’s sleep by at most that window.
  • No starvation. An active tenant defers snapshots for at most BACKUP_CHANGE_MAX_DEFER_MS (10 min), so constant writes still get snapshotted.
  • No hot-path work. Snapshots are never taken from the watch directly. It enqueues the existing ActivityScheduler durable snapshot job, which keeps that job’s gateway start, lease, retention, and restore-pending guards.
  • Gated before probing. Each tick first asks the scheduler for getSnapshotGate() — one DO storage read, no container RPC — and re-arms without any probe or scan when a snapshot job owns the durable slot or the floor has not elapsed since the newest committed snapshot (whichever trigger produced it: dispatch cron, operator, or this watch). A gate read failure is fail-open (logged, then the normal tick). Previously a busy tick still ran a liveness probe plus a find scan every two minutes (~1,440 container operations per tenant per day) to discover there was nothing to do (durable-agents audit finding F7). A changed tenant inside the floor is no longer scanned, so dirtySince starts at the first scan after the floor; maxDeferMs still bounds starvation.
  • Arming happens on a running tenant only: the dispatch backup control action (/control) and the warm refreshActivitySchedule path (gateway healthy).

Wiring. wrangler.jsonc declares the BackupWatch class (migration v8) and the BACKUP_WATCH binding in every environment. In production the tenant’s bindings come from lerma, which binds BACKUP_WATCH only when the backup_change_watch provisioning flag is on (see apps/lerma). Both flags are plain-text vars that lerma always sets to true or false, so turning a flag off explicitly reverts it on redeploy.

Persistence policy telemetry (Phase 0)

Before changing the backup policy (see docs/plans/persistent-workspace-phase0.md) the current behaviour is measured, not guessed. Three Analytics Engine events are the input; they are observability only and never gate a snapshot, restore, or stop:

EventEmitted fromdouble1double2double3
persistence-snapshotfinalizeSnapshot (success), ActivityScheduler (fail/skip)durationMsarchiveDurationMsbytes
persistence-restorerestoreIfNeeded (recovery point), restoreBackupByIddurationMsbytes—
backup-change-windowBackupWatch tick (runBackupWatchTick observation)dirtyWindowMsscanMs—

bytes is the captured workspace size from the archive’s meta.json (one small metadata read at archive), so “the artifact is large” and “the pipeline is slow” are separable. trigger (scheduled / manual / deprovision) and the skip reason (interval-not-elapsed, unchanged, not-running, backup-lock-busy) are blobs, so a quiet series has a visible denominator. backup-change-window records the continuous dirty time the watch accumulated (dirtySince), which is the RPO distribution the checkpoint cadence will be chosen from.

Queries and the phase gates are in docs/plans/persistent-workspace-phase0.md.

Workspace snapshots on eviction (Phase 1, flag-gated)

WORKSPACE_SNAPSHOT=true makes TenantSandbox.onActivityExpired() capture a platform filesystem snapshot immediately before eviction and commit the handle at workspace/snapshot.json in BACKUP_BUCKET (one head handle; image-pinned; 30-day TTL). A new workspace-snapshot Analytics Engine row records durationMs, bytes, trigger=evict and the outcome.

Capture is observability-grade in Phase 1: boot still uses the archive, so a capture failure (or an unsupported runtime) never blocks the archive/stop path — unsupported and failed are warnings, not errors.

The fallback must restore the same tree. Phase 2 may boot from the handle and fall back to the archive, so the two tiers have to hold the same content. The archive is a complete image today because SNAPSHOT_EXCLUDES is behind BACKUP_ARCHIVE_EXCLUDES (off by default); an instance that opts into the excludes makes an archive restore diverge from a snapshot restore by exactly those paths. Do not enable the excludes on an instance whose snapshot boot must be able to miss into the archive without regeneration.

Runtime prerequisite. Cloudflare only supports snapshots on a container application with scheduling_policy: "durable_object". The tenant container is on the default policy, and that cannot be changed in place: the policy is immutable, a new Durable Object class/namespace is required, and the legacy Container class is unsupported under it. Tier 1 therefore depends on the Sandbox 1.0 workload adapter, tracked in #380 (S2–S4), not on a config change. SandboxV1 already runs on durable_object and implements the same snapshotWorkspace() method, so the capability is real where the runtime allows it. Keep the flag off until an instance is opted into v1; see docs/plans/persistent-workspace-phase1.md.

Backup before deprovisioning

A container is never deprovisioned without a committed snapshot. The durable backup-job queue (src/activity/backup-jobs.ts) gains a deprovision operation (mode: 'sleep' | 'freeze'); the ActivityScheduler alarm runs it as snapshot → stopIfIdle() (sleep) or snapshot → destroy() (freeze), and a failed snapshot retries rather than tearing the container down (fail closed).

Reuse a snapshot that already covers the state. onActivityExpired() stops inline instead of enqueueing a snapshot when the container has already been idle for a whole window and an existing archive already captures the current filesystem — a second archive is pure cost. Coverage is proved two ways, cheapest first:

  1. Age. hasRecentSnapshot() (a pure catalog read in src/persistence.ts) answers whether the newest committed archive is newer than the instance’s idle timeout (resolveSleepAfterMs(): staging 15m, dev 2h, production 6h) plus IDLE_SNAPSHOT_REUSE_SLACK_MS (5m, so the archive finalized just before the previous stop attempt still counts).
  2. No change since the last commit. With BACKUP_CHANGE_DETECTION on — the default — a quiet tenant takes no scheduled snapshot at all, so its newest archive legitimately ages past the idle window and the age test alone would re-archive it on every sleep cycle. When the gate is enabled, the hook runs the same scanTenantChanges() the scheduled path uses and treats unchanged as coverage: the baseline is promoted only after a catalog commit, so unchanged means that archive already holds this state. changed/active, a missing baseline, or detection being off fall through to the archive.

A read failure is treated as “no recent snapshot” (fail closed). This also breaks the loop where the deprovision snapshot’s own RPCs renewed the activity window, stopIfIdle() refused, and the next expiry snapshotted again (observed on tenant-tarjetero-dev: three 422 MB archives per idle expiry while the container never stopped).

Deprovision archives stamp the schedule. finalizeSnapshot receives stampScheduled: true for deprovision jobs as well as trigger: 'scheduled' ones, so a committed deprovision archive resets lastScheduledBackup and the next cron tick does not archive the same state again.

Stop only on inactivity (stop-time re-check). The snapshot can take minutes on a large tenant, so the container may be used again between enqueue and stop. The sleep branch therefore does not call sandbox.stop() directly: it calls TenantSandbox.stopIfIdle(), an RPC on the Sandbox DO (the activity authority). That re-reads the DO’s own idle window — renewed by every request, including a proxy request for an in-flight chat — and the container process list, and stops only when the window has genuinely elapsed and no transient work is running. If the tenant became active during the snapshot the job returns skipped (detail: active:<reason>) and leaves the container running; the next idle expiry enqueues a new job. Refusals: activity (window renewed), transient-activity, probe-failed (unknown is not idle), billable-production (never auto-sleeps), not-running.

  • Idle sleep. TenantSandbox.onActivityExpired() stops inline when an existing snapshot covers the state (a recent archive, or unchanged from the change scan when detection is on), and otherwise enqueues a deprovision (sleep) job, so the Sandbox DO leaving memory never loses an in-flight multi-minute snapshot. With no ACTIVITY binding it falls back to the previous stop-now path.
  • Freeze. POST /control freeze enqueues a deprovision (freeze) job and acknowledges immediately (202 { ok, accepted: true, frozen: false, jobId }), failing closed at enqueue (409 when a durable job is already active). The caller does not hold a multi-minute request open: it polls the tenant’s GET /api/health/backup/:jobId until the job reaches a terminal status. A snapshot failure surfaces as a failed job and the container stays running, so lerma never marks a still-running instance frozen.
  • Platform stops (an inactivity timeout firing after the DO left memory, a container crash, or main-process exit) have no Worker hook; the periodic + change-watch snapshots bound that residual RPO.

The pollable endpoint (GET /api/health/backup/:jobId) carries no in-Worker authentication — the same posture as /api/health — and is reachable by the lerma control plane’s health-monitor service token through the health Access tunnel (the *.fouria.io/api/health path application in apps/fouria/iac/access.tf). Cloudflare Access path inheritance covers the /backup/* sub-path, and it is also listed explicitly so the tunnel allowlist documents the endpoint.

Health & Status Endpoints

GET /api/health is the unified health endpoint: it returns the startup/readiness/liveness probes plus the canonical status (running|starting|stopped), processId and processType. It requires no authentication inside the Worker, but at the Cloudflare edge it is Access-protected by path-scoped applications (*.fouria.io/api/health) that carry no role requirement: they admit any authenticated user (browser) and the lerma health-monitor service token (headless). GET /api/status, GET /api/admin/gateway/status/polling and GET /api/admin/gateway/status/ws are thin aliases over the same snapshot. Browser clients use the sibling paths outside /api/health/* — GET /api/health-ws (WebSocket stream every ~2s), GET /api/health-snapshot (HTTP snapshot), and GET /api/health-components (Cloudflare component states) — while /api/health, /api/health/ws, and /api/health/components remain mounted for headless service-token consumers (lerma). The browser-facing sibling paths exist because /api/health and /api/health/* are covered by the path-scoped FourIA Health - <slug> Access app whose immutable AUD differs from the host app’s: the browser’s host-app session is invalid there, so Access answers with a 302 login redirect to *.cloudflareaccess.com — which WebSocket handshakes cannot follow and browser fetch() calls reject on cross-origin CORS.

By default /api/health warms the container (lerma keep-alive). ?passive=1 probes without warming — the ROOT /_admin/diagnostics surface uses it so an operator can debug an instance whose container is down. The admin dashboard subscribes to the health WebSocket (with an HTTP polling fallback) and never blocks the UI on gateway status.

GET /healthz is a Kubernetes-style liveness probe: one non-waking isRuntimeActive() read, always 200 { ok, ts, status } (status = active | idle), never warming the container or probing the gateway. It exists because /healthz otherwise fell through to the gateway proxy (app.all('*')), which warmed the container and waited for the port — ~108 s on a cold instance, so the platform killed the request before it answered. Readiness stays on /api/health.

A single /api/health request issues one gateway /health curl exec, shared by the readiness and liveness probes. They previously each ran their own, so every poll spawned a pair of identical curl processes (visible in /debug/processes) — a duplicate on the frequent lerma probe. The process scan and the /health probe are now both fetched once, in parallel, and reused across the snapshot.

Gateway memory (gateway: { pid, rssBytes, threads })

The gateway is one Node process in a 4 GiB / no-swap container whose RSS is dominated by per-worker_thread V8 heaps that an in-process restart never reclaims (docs/plans/sandbox-memory-usage.md). A dev tenant “rebooted” once and the only way to answer “was it memory?” was to SSH in and read /proc, dmesg and the gateway log by hand.

src/gateway/memory.ts makes that a first-class signal:

PieceBehaviour
buildGatewayResourceScript()One shell exec: reads the cgroup memory limit (mem:<bytes>; v2 memory.max, v1 fallback) and pgrep -x 'openclaw-gatewa' → gw:<pid>:<rssKb>:<threads> per live gateway server (same exact comm as the soft reload, so it never measures the openclaw gateway run launcher; zombies skipped).
readGatewayMemory()Runs it (gateway.memory-read, bounded) and returns { resources, containerLimitBytes }; readGatewayResources() is the resources-only view. Never throws: a failed read is nulls, not a broken health probe.
resolveMemoryRestartBytes()Derives the watchdog threshold: 75% of containerLimitBytes, or null below 3 GiB (too small for a healthy gateway; the gate is below the 4 GiB standard-1 class to tolerate runtime accounting). Unknown limits fall back to 3 GiB (the 4 GiB standard-1 value).
sampleGatewayResources()Throttles the read to GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS (60s), so the frequent health poll issues at most one extra exec per interval and only a fresh sample writes a metrics row. A cached null is served inside the window too, so a broken probe is not retried every poll.
maybeRestartGatewayForMemory()The dev-gated watchdog (below).

attachGatewayResources() in routes/public.ts attaches the sample to the snapshot as gateway and writes a gateway-resources Analytics Engine row (double1 = RSS bytes, double2 = threads, blob1 = gateway-resources, blob2 = customer, trailing platform/environment blobs like every other cost row). It is applied by both /api/health (and /api/health-snapshot) and the /api/health-ws stream, and only when the snapshot is running — a sleeping/stopped container is never woken for a diagnostic.

Dev-gated memory watchdog. When the sample reaches the instance-aware threshold — resolveMemoryRestartBytes(): 75% of the container’s cgroup memory limit (3 GiB on a 4 GiB standard-1, 4.5 GiB on a 6 GiB standard-2, 6 GiB on an 8 GiB standard-3, 9 GiB on a 12 GiB standard-4) — on a dev instance (isDevAgentsEnabled: -dev hostname, OPENCLAW_DEV_MODE, or local DEV_MODE), the gateway is replaced — killGateway + resetStartupCache + ensureGateway({ skipRestore: true, bypassMaintenance: true }). That is the only mechanism that actually returns RSS to baseline.

The limit is read from the container on every sample (one extra local read in the same exec), so an instance resized by an instance-type rollout picks up the new threshold without a Worker deploy. Instances smaller than 3 GiB (i.e. basic/lite) are skipped (instance-too-small): a restart cannot make an undersized box viable, and a 3 GiB fixed threshold on an 8 GiB standard-3 would be only 37% of the limit and restart a healthy gateway.

Design constraints worth keeping:

  • The threshold guards the container limit, not OpenClaw’s warning. OpenClaw’s own rss_threshold is 1.5 GiB and fires permanently on a healthy-but-large dev gateway (~2.1 GiB observed, ~2.6 GiB peak). Using it as the action threshold would restart the gateway in a loop.
  • Scale with the instance, never hardcode. A fixed byte threshold means a different headroom on every instance type; the ratio keeps ~25% of the limit free for the kernel and the next allocation burst.
  • No soft-reload step. A measured in-process SIGUSR2 restart does not reclaim RSS (1.14 → 1.38 → 1.18 GiB), so reloading first would only add a failed attempt.
  • skipRestore. A kill leaves the filesystem untouched; re-entering the R2 restore barrier would deadlock on the backup lock for no benefit.
  • Cooldown claimed before acting. GATEWAY_MEMORY_WATCHDOG_COOLDOWN_MS (30 min) is recorded before the replacement, so two concurrent health polls cannot start two restarts.
  • Detached. The watchdog is dispatched through executionCtx.waitUntil (or left detached when there is no context), so a multi-minute replacement never delays or fails a health response.

Every decision returns a stable machine reason (not-dev, no-resources, instance-too-small, below-limit, cooldown, rss-above-limit), and an actual restart logs a structured gateway.memory.pressure event with pid, rssBytes, rssMib, threads, the resolved limitBytes/limitMib, and the measured containerLimitBytes/containerLimitMib.

Two-stage policy and continuous sampling (2026-11-03). The watchdog above only runs when something calls attachGatewayResources, and nothing polls /api/health while the gateway is healthy — the Control UI talks to the gateway directly. During the tenant-tarjetero-dev outage that blind spot was total: gateway-resources had no rows for ~17 h, so the crash had no memory evidence at all.

  • checkGatewayMemoryHealth() (gateway/memory.ts) is now the single sampler used by both /api/health* and the durable ActivityScheduler.alarm() (runMemoryWatch()), so a tenant is observed whether or not a browser is polling. The throttle and the gateway-resources row are unchanged.
  • Every fresh sample also writes gateway-memory-sample (blob1, blob2 = customer, blob3 = running | absent, double1 = RSS, double2 = threads, double3 = container limit). The row exists even when no gateway process is running, so a gap in the series means “nobody sampled” instead of “nothing happened”.
  • decideMemoryAction() is the pure policy behind both callers: ≥ 60 % of the cgroup limit (resolveMemoryCheckpointBytes, GATEWAY_MEMORY_CHECKPOINT_RATIO) → checkpoint; ≥ 75 % (resolveMemoryRestartBytes) → restart, escalating to checkpoint-and-restart when the newest archive is older than GATEWAY_MEMORY_LOSS_BUDGET_MS (15 min). maybeRestartGatewayForMemory delegates to the same function (mapping its reasons back to the vocabulary above), so the thresholds cannot drift between the request path and the durable alarm.
  • The durable alarm performs the checkpoint by enqueuing a snapshot job with trigger: 'memory-pressure' before restarting, so the work is on disk regardless of what the kernel does next. The checkpoint has its own cooldown (EMERGENCY_SNAPSHOT_COOLDOWN_MS, 10 min) and is surfaced as a gateway.memory.policy event.
  • The request path may still only restart: it cannot own a durable job, and blocking a health response on a 450 MB snapshot is worse than restarting.
  • healthWsHandler now passes its executionCtx to attachGatewayResources. Without it the detached watchdog promise was cancelled as soon as the frame was sent, so the watchdog was a no-op on the one path a browser actually keeps open.
  • Sampling never wakes the container, and it yields to every durable job. sandbox.exec auto-starts a stopped container, so an unbounded /proc read from the alarm raced container.start() during a local restore (gateway.memory-read failed (launch-failed) / Container failed to start, and a workerd jsgInternalError that aborted the start). checkGatewayMemoryHealth now takes the non-waking isRuntimeActive() read first (bounded by PASSIVE_PROBE_TIMEOUT_MS, like the passive health probe) and returns { action: 'skipped', reason: 'runtime-inactive' } with a fresh: false sample when the runtime is off — so a sleeping tenant is never billed for a diagnostic and never woken by an alarm tick. runMemoryWatch additionally returns before sampling while a boot, backup, or maintenance job is active (queued/running/retrying): those own the container, and the watchdog is observability, never a participant in a boot.

Design and remaining work: docs/plans/graceful-shutdown-and-emergency-checkpoints.md.

A destructive restore is never triggered by an unreadable container. probeContainerState() returns present | absent | unknown, and restoreIfNeeded only restores on absent. On unknown it throws the retryable ContainerStateUnknownError and writes no restore-needed latch, so the durable boot job retries instead of replacing the filesystem of a container it cannot read. This is what turned the 2026-11-03 wedged container (dead control channel) into a destructive 450 MB restore that timed out and pinned every later boot behind RestoreInFlightError (restore-outcome-unknown).

Cloudflare component states

The dedicated GET /api/health/components endpoint returns a quick view of the Worker-bound platform pieces rendered by the ROOT diagnostics horizontal Cloudflare Components widget. GET /api/health also carries the same (redacted) components entry — but only on the non-passive path. Lerma’s HealthMonitor polls /api/health?passive=1 frequently and never reads components, so the passive probe skips the component probe entirely (no isRuntimeActive read, no R2 HEAD) to keep the frequent poll slim:

ComponentProbe
ContainerNon-waking sandbox.isRuntimeActive() (up / sleeping / unknown)
Durable ObjectSandbox + ACTIVITY bindings present and the control channel answered
R2Bindings-shaped BACKUP_BUCKET plus a HEAD of backup-handle.json
AI GatewayPresent only when all three gateway credentials are configured (up = configured)

Every probe is non-waking and individually bounded (3s), so the endpoint is safe on a sleeping instance and can never hang the health response; on failure the components entry is omitted instead of blocking. The AI Gateway is inclusion-gated on credentials and reported as up (“configured”) — the probe deliberately does not issue an authenticated Cloudflare API call on every poll, and a missing account-scoped token must not read as a false outage.

Each component also carries binding-derived facts (a { label, value }[] shown in the widget’s hover panel) and a Cloudflare dashboard logsUrl (a per-product logs/metrics deep link opened on click). Both are derived from Worker bindings/env only — never an authenticated Cloudflare API call — so the surface stays non-waking and free (src/gateway/cloudflare-links.ts builds the links: Container → Workers Logs/observability, Durable Object → DO metrics/logs, R2 → the BACKUP_BUCKET_NAME bucket page, AI Gateway → the gateway page). When the account id is not bound the links fall back to the dashboard’s documented ?to=/:account/… form; a component without a resolvable link renders as a plain (non-clickable) card. The unified GET /api/health snapshot redacts facts/logsUrl (redactComponentDetails) — it is a broadly-consumed public probe — and serves only the state summary; the enriched view is exclusive to GET /api/health/components.

On phones the widget collapses into a compact status widget that shares a row with the gateway and persistence-lock status widgets at the top of the diagnostics page (not floated or overlaid): a single pill carrying the Cloudflare icon, a worst-case rollup status dot and an up/total summary. Tapping it expands the same component strip as a compact panel. To build that parallel row, AdminLayout omits the topbar gateway + persistence indicators on the diagnostics page on phones (App.tsx) and DiagnosticDashboard renders GatewayStatusLed + PersistenceLockIndicator beside the Cloudflare widget. The gateway pill gains an identifying icon and the lock an Idle/Backing up/Restoring label (its color still encodes the state); both use compact phone sizing so the three widgets stay on one line on normal phones and wrap gracefully on very narrow ones. The desktop layout is unchanged — the topbar keeps its indicators and the Cloudflare card renders full-width.

The ROOT /_admin/diagnostics page renders its operational tools as separate clickable Fouria DevOps widget cards (backup connectivity + configuration editor + paired devices). Each card’s title row toggles its body while the action buttons stay live, and the cards load their data asynchronously and render their chrome immediately — there is no blocking loading animation for the data that populates them. On phones each option collapses into a vertical icon + name menu row (collapsed by default; tap to open) with the option’s actions revealed inside the opened body. The Configuration Editor reflows to the viewport: action buttons stack full-width, the section tabs scroll horizontally, long JSON wraps, and the raw textarea shrinks. The Debug Tools grid drops to a single column so nothing forces sideways scrolling, and the admin bottom nav uses flexible items (with safe-area insets) so every nav entry fits. The floating dev role switcher (DevUserSelectorCompact) is lifted above the bottom nav on phones so it neither covers the last nav item nor overflows the viewport.

The gateway status indicator lives in the admin topbar beside the persistence-lock indicator and the notification bell (GatewayStatusLed, mounted in App.tsx’s AdminLayout). Its color-coded LED reflects the gateway status (running/starting/sleeping/stopped/unknown); clicking it reveals the complete status value (status, process, process type, the startup/readiness/ liveness probes, restore state and last update) and a reboot icon that performs the same soft-reload/restart as the removed Gateway Controls banner. The reboot action is hidden behind an Admin badge when /debug/* is disabled, matching the previous gating.

Live gateway detection

On the Worker-driven cold start the gateway is a supervised sandbox.exec process (/bin/bash -lc 'exec openclaw gateway run …'), so it is visible to listProcesses() for the life of the container. The legacy full mode (manual runs, test:gateway:boot) still launches it as a detached background child of start-openclaw.sh (… &), and a container provisioned by an older image may still be serving a detached gateway during a rolling deploy — for those, after the script exits the gateway binary is invisible to listProcesses(). To avoid false no_process reports on log/status surfaces, those endpoints (GET /api/admin/logs, GET /debug/logs, GET /debug/stop-gateway, POST /api/admin/gateway/restart, GET /api/admin/audit-logs) call findExistingGatewayProcess(sandbox, { probeLive: true }), which falls back to a bounded /health probe and — when the gateway responds — returns a synthetic gateway-live process whose logs are read from /tmp/gateway-stdout.log. The passive health snapshot path deliberately does not probe so a down container fails fast.

Log Levels

LevelWhen to use
debugLocal development
infoProduction default
warnHigh-traffic production
errorSecurity-sensitive production

TypeScript & Tooling

Type Generation

Worker types are auto-generated:

bun run types   # wrangler types → worker-configuration.d.ts

Never manually edit worker-configuration.d.ts — it’s regenerated from wrangler.jsonc.

Testing

bun run test             # Unit tests (Vitest)
bun run test:coverage    # With coverage report
bun run test:client      # SPA component/page tests (jsdom)
bun run test:ui          # Playwright UI suite (Vite dev server + mocked API)
bun run test:ui:headed   # Same, with a visible browser
bun run test:e2e         # E2E tests (cctr + Playwright)
bun run test:e2e:cli     # CLI-only E2E (cctr)

The Playwright UI suite (test/ui/, config playwright.config.ts) drives the real SPA served by bun run dev with per-test /api/* mocks — no Worker, container, or Cloudflare credentials. It covers the app shell, role-based navigation and redirects, every admin page’s render, and the notification bell. See docs/TESTING.md for the full description and the Nix PLAYWRIGHT_BROWSERS_PATH notes.

Formatting & Linting

bun run format          # oxfmt (auto-format)
bun run format:check    # Check formatting
bun run lint            # oxlint
bun run typecheck       # tsc --noEmit

Common Tasks

Adding a New API Endpoint

  1. Create route handler in src/routes/api/<domain>.ts
  2. Register in src/routes/api.ts
  3. Add client method in src/client/api.ts
  4. Write tests (unit + integration)
  5. If new dependencies: update package.json

Adding a New Environment Variable

  1. Add to OpenClawEnv interface in src/types.ts
  2. If passed to container: add to buildEnvVars() in src/gateway/env.ts
  3. Update .dev.vars.example
  4. Document in ../../docs/ENV-VARIABLES.md