# Fouria

## 2026.10.29

### Features

- **fouria:** expose gateway RSS and add a dev-gated memory watchdog (#526) ([441fdae](https://github.com/0xCAB0/fouria/commit/441fdae76deff4d3ff8f4dbc3f76e08d970dd896))
- **fouria,moltlazy:** reconcile declared software on demand and report its real state (#525) ([5476899](https://github.com/0xCAB0/fouria/commit/5476899a04baa2f41ed732fccf40cebeec373149))
- **fouria:** renew durable snapshot leases (#512) ([5a38cd5](https://github.com/0xCAB0/fouria/commit/5a38cd500f9eb1a426f81da52b1868fce43a1874))
- **fouria-dispatch,fouria:** scale fleet fan-out and make catalog writes conflict-safe (#510) ([c4ef484](https://github.com/0xCAB0/fouria/commit/c4ef484e1831af27ec92682f05a94de4f66041ed))
- **fouria:** checkpoint durable tenant jobs and harden the backup step budget (#509) ([2114c1f](https://github.com/0xCAB0/fouria/commit/2114c1f8125126022a8294a878830ac5b39a81f4))
- **lerma,fouria:** price semantic billing actions with trace precedence (#507) ([85f8701](https://github.com/0xCAB0/fouria/commit/85f8701ecfe0fce2b92fadd24468c09d0633e337))

### Bug fixes

- **lerma:** make dispatch deploys idempotent around migrated DO classes (#516) (#520) ([d3c2f60](https://github.com/0xCAB0/fouria/commit/d3c2f60fe78bda2930ddf470840c3d69e0ce0b53))
- **fouria:** stop the idle-sleep snapshot loop and reconcile dev heartbeats (#518) ([9b324bc](https://github.com/0xCAB0/fouria/commit/9b324bc82e60017388380f27a294da9e48fa7b2d))
- **fouria:** make backup catalog reads pure and healing explicit (#511) ([7470376](https://github.com/0xCAB0/fouria/commit/7470376cec2257be3107ac608b6c97003e3608de))
- **fouria:** keep environment repair available while the gateway is down (#506) ([1321226](https://github.com/0xCAB0/fouria/commit/1321226bec316e8dc2f56c25acff22174ab8ba8e))
- **fouria:** defer repairs until the restored gateway is up (#505) ([579532b](https://github.com/0xCAB0/fouria/commit/579532b41af9049fe3639c53534d0294d9c467b3))

### Chores

- **openclaw:** migrate to 2026.10.8 (#524) ([66a46da](https://github.com/0xCAB0/fouria/commit/66a46dabba340316d9a21316af7bf241d0b10ff6))
- **openclaw:** migrate to 2026.10.6 (#515) ([0df3522](https://github.com/0xCAB0/fouria/commit/0df352287694025dc3690edbe64641ae4867f266))

## 2026.10.21

### Features

- **website:** add FourIA Terms of Service page (#499) ([892282c](https://github.com/0xCAB0/fouria/commit/892282c955f898838b80c47a7fcb3029bef0593b))
- **observability:** ingest Cloudflare OTLP destinations + uptime stats (#337) (#478) ([1a4630b](https://github.com/0xCAB0/fouria/commit/1a4630b2087b1e42e4b74d9b5c73bb768f588f19))
- **fouria,lerma:** backup before container deprovisioning (#456) ([54ef3f6](https://github.com/0xCAB0/fouria/commit/54ef3f67489a13eb65aa1ece7b5e81ce5593c4d1))
- **fouria:** Telegram webhook ingress with cold-start message queue (#364) (#449) ([b95acd8](https://github.com/0xCAB0/fouria/commit/b95acd8dc4b5abacc53535a2aa30ac9d10f0972e))
- **fouria,moltlazy:** declarative software provisioning for agent skills (#459) ([a7b623b](https://github.com/0xCAB0/fouria/commit/a7b623bc5752594f186cb79a295cf8773aac4678))
- **fouria:** dev→production agent push (promotion) + local two-worker E2E (#453) ([3dc5863](https://github.com/0xCAB0/fouria/commit/3dc586374a647aebe12fc32d064843e56aed3c5c))

### Bug fixes

- **fouria:** defer request-scoped restores and stop only on inactivity (#503) ([7171192](https://github.com/0xCAB0/fouria/commit/7171192e99b434a8f22abfab21a712d9c6e6c9de))
- **observability:** decompress gzip OTLP destination batches (#337) (#484) ([7bd792b](https://github.com/0xCAB0/fouria/commit/7bd792b461cfd4fdc0cf4de42bf7e88a71178816))
- **gateway:** soft reload targets the server and keeps OPENCLAW_NO_RESPAWN (#463) (#477) ([a883054](https://github.com/0xCAB0/fouria/commit/a8830542d54fbaf326d728cd6d76a566ba9b110d))
- **fouria:** durable maintenance jobs over WebSocket, release promotion, homepage crash (#468) ([90680ba](https://github.com/0xCAB0/fouria/commit/90680ba18ea6791c446874e6772bd658efe09be8))
- **fouria:** run SQLite integrity + doctor repair as durable jobs (#467) ([f841ac4](https://github.com/0xCAB0/fouria/commit/f841ac45d6b90b7fe927c6b969cafcfa0ca1a891))
- **fouria,moltlazy:** mitigate dev sandbox memory OOM (#462) ([132355e](https://github.com/0xCAB0/fouria/commit/132355e2cfab071bd384ed1c756e0a969ae7d596))

### Tests

- expand auth backup and billing coverage (#471) ([43a8b18](https://github.com/0xCAB0/fouria/commit/43a8b18d419046f7c352c7165bd178c6b0f73c64))
- **e2e:** drop cctr-incompatible '#' corpus headers and add a guard ([cc95f96](https://github.com/0xCAB0/fouria/commit/cc95f96bf2a2a219f9f720821287ac98233cdd9f))

### Chores

- **observability:** enable production destination export (#337) (#486) ([98078dc](https://github.com/0xCAB0/fouria/commit/98078dcdd4b320064bc22e6bb75bc7679a191f13))
- **release:** adopt Changesets for semver versions and changelogs (#483) ([67a91ed](https://github.com/0xCAB0/fouria/commit/67a91ed750f6a66d068de599d0470b93706066e7))
- **openclaw:** migrate to 2026.10.5 (#476) ([9991907](https://github.com/0xCAB0/fouria/commit/9991907e3aa0979e5a6a1d50bcf6a63adcd35235))
- **deps:** bump the bun-all group across 1 directory with 56 updates (#474) ([471ddad](https://github.com/0xCAB0/fouria/commit/471ddad5864d20869e6aba1fa706bd9a093c6553))
- **fouria:** pin cctr to the 0xCAB0/cctr fork (0.31.0-paso4.1) ([9b5ea28](https://github.com/0xCAB0/fouria/commit/9b5ea283662b243c511a3e2258bd858441227435))
- **fouria:** enable software provisioning on staging via MOLTLAZY_FEATURE_FLAGS (#464) ([dd3c417](https://github.com/0xCAB0/fouria/commit/dd3c41794eaef92ce8a0adee9912c0be27e297ac))

### Other changes

- Feature/dev instances can emit billable (#502) ([d8bc76d](https://github.com/0xCAB0/fouria/commit/d8bc76d5e5ce7718c68a804c42165d7e3372d81d))
- Feature/enhance fouria builder (#501) ([95a1107](https://github.com/0xCAB0/fouria/commit/95a1107978a5b82460d61c5344177d6fc145ba03))
- Feature/migrate dev to vite plugin (#475) ([a8cad6f](https://github.com/0xCAB0/fouria/commit/a8cad6f18e9f8a07330ca3b70919be5d15033f59))
- Fixed healtz quick endpoint ([7e76734](https://github.com/0xCAB0/fouria/commit/7e767347b60fb0d6964bc5d382ef13aff680b856))
- Moved components snapshot from Healthcheck endpoints ([b2d9679](https://github.com/0xCAB0/fouria/commit/b2d9679381e61ab3c77e81411966870b171d9c62))

## Historical changes (imported from `apps/fouria/AGENTS.md`)

These entries predate the generated release sections above. They were maintained
by hand in `apps/fouria/AGENTS.md` and are kept here verbatim for traceability.
New entries belong in a changeset (see `docs/VERSIONING.md`); the sections above
are generated by `scripts/release.mjs`.

- **Adversarial/fuzz pass on the Sandbox v1 critical pipelines closed six defects.** A second attack run against zero data loss, secrets, memory and boot found: **(1)** `v1CreateSnapshot` could race an operator restore (it only checked container liveness; the worker's maintenance window covers route-driven calls but not the primitive) — it now returns `skipped: 'restoring'`, so a half-swapped directory can never become the recovery point. **(2)** `startGatewayOwner` re-checks `#restoring` before the launch and inside the readiness loop, so a boot that overlaps a restore aborts retryably instead of launching over the swap or reporting readiness for a gateway the restore is about to kill. **(3)** `v1ProcessOutput` decoded whatever a process wrote after the SDK had buffered it in the DO isolate — reads are now truncated at `MAX_PROCESS_OUTPUT_BYTES` (1 MiB per stream) with `truncated: true`. **(4)** A `start-openclaw.sh start` wrapper that exits **0** (lost startup lock) with no gateway process anywhere waited the whole `timeoutMs` (up to 5 min); after `SANDBOX_V1_ZERO_EXIT_GRACE_MS` (default 15s, `0` disables) the DO proves the absence of an owner (`pgrep -f` on the setup wrapper **or** the gateway) and fails fast, while a booting lock winner resets the grace and is adopted. **(5)** A failed conversion left the streamed squashfs and the partial extraction in `/tmp` (~2× the archive on a 4–12 GB disk), so retries could ENOSPC the container: failures now clean up best-effort and rethrow the original error. **(6)** The durable `migration:convert-status` and its log stored `error.message` verbatim; `redactSecrets` (known env values ≥ 8 chars, longest-first) scrubs both before they are written. Tests: `src/sandbox-v1-adversarial.test.ts` (restore/snapshot overlap, restore retry + in-progress refusal, 12-snapshot → 24-restore → boot round-trip fuzz, R2 secret scan, redacted conversion failure, output cap, zero-exit fast-fail / live-peer adoption), `src/sandbox-migration/conversion.test.ts` (scratch cleanup, cleanup failure never masks the original error), `src/logging.test.ts` (`redactSecrets`, 60 fuzz-embedded credentials, `envSecretValues`); fuzz harness `bun run --cwd apps/fouria test:fuzz:sandbox-v1 -- 4000` (all invariants held); corpus `test/business-rules/operational-modes.txt`; docs `docs/plans/sandbox-v1-adversarial-fuzz.md`, `docs/ENV-VARIABLES.md`.
- **Sandbox v1 presents gateway traffic as direct-remote when the Gateway refuses proxy attribution (`proxy_attribution_required`).** Every dev instance deployed on the v1 runtime served `{"error":{"type":"proxy_attribution_required"}}` on the root HTML/WS proxy (and the admin-RPC reads fell back to the CLI): OpenClaw attributes proxy-_shaped_ traffic (any `Forwarded` / `X-Forwarded-*` / `X-Real-IP`) against `gateway.trustedProxies` **before** Gateway auth and rejects a source it cannot trust, and moltlazy's heal can only list the ranges the **legacy** runtime reaches the container from (the Cloudflare Sandbox bridge, `10.1.0.x`). The v1 transport forwards through `ctx.container.getTcpPort(port)` from the `SandboxV1` Durable Object, whose ingress address is owned by the platform — not an address any tenant config can know (and blindly unioning more private ranges is unsafe: a range that swallows the `X-Forwarded-For` walk makes attribution _worse_). A **direct-remote** request, by contrast, is always attributed and is served with the same token/device auth, so `SandboxV1.fetch()` now probes attribution **once per gateway incarnation** (`GET /` with the rebuilt forwarded headers — exactly the Gateway-authenticated route that fails) and, when the Gateway refuses it, strips every proxy-shaped header before forwarding. The verdict is cached for the DO lifetime and re-probed after `v1KillGateway` / `v1Destroy` / `startGatewayOwner`, so a healed `gateway.trustedProxies` restores client-IP (rate-limit/audit attribution) fidelity without a Worker deploy; requests with no forwarded headers skip the probe, and a probe that cannot reach a booting Gateway leaves the mode `unknown` (traffic keeps its headers) and is retried. `src/gateway/proxy-attribution.ts` owns the header set and the payload classifier (403 + `type`/message) so the proxy, the admin-RPC path and the transport cannot drift. Structured event: `sandbox.v1.proxy_attribution.direct` (warn). Tests: `src/gateway/proxy-attribution.test.ts`, `src/sandbox-v1.test.ts` (`gateway proxy attribution`: direct fallback, cached verdict, accepted attribution keeps headers, failed probe never degrades traffic, no probe without forwarded headers, re-probe after destroy); corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` § "Sandbox v1 gateway proxy attribution", `packages/moltlazy/AGENTS.md`.
- **Sandbox v1 waking/sleep policy and automation wake are lifecycle-correct (audit follow-up).** Four gaps between the legacy policy and SDK 1.0's `durable_object` lifecycle are closed. **(1) Production keep-alive is real:** `resolveInactivityPolicy()` now returns the 6h maximum + `keepAlive: true` for billable production instead of `null` — with no timeout the platform stops the container shortly after the DO goes inactive, so the old "never sleeps" contract was the opposite of what it claimed. A keep-alive purpose (`sandbox-v1:keepalive-pending`) shares the DO alarm slot, re-applies the timeout and re-arms at `timeout/2`; the cadence and a queued conversion keep their own markers, and clearing one purpose never deletes another's alarm. **(2) Wake keeps the provisioned size:** `startContainer()` reads `CONTAINER_INSTANCE_TYPE` (allowlist, fallback `standard-1`); lerma deploys it from the instance type. **(3) No empty recovery point:** `v1CreateSnapshot` skips a stopped container (`{ state: 'skipped', reason: 'not-running' }`) instead of calling `ensureRunning()` and archiving an empty `/home/openclaw`; `/control` `freeze`/`backup`/`checkpoint`, `/debug/*` and the scheduler's snapshot job report the skip. **(4) Reliable automation wake:** the occurrence guard (`wokenFor`) survives mirror writes and resets only when the next run advances; a woken occurrence that does not advance backs off 5m→30m instead of re-waking every 60s, and a failed wake/read keeps the schedule with a bounded retry instead of deleting it. The durable boot job mirrors the schedule right after `launchRestoredState` (no dashboard poll needed); on v1 the wake **enqueues that boot job** (deduped) and returns `{ ready: false }` instead of blocking the alarm turn on a restore + launch, so no CLI read runs against a booting container (legacy keeps the fast `waitForReady:false` path). Activity re-arms go through the shared combined alarm, and the alarm's memory watch is gated on `isRuntimeActive()` so a diagnostic never starts a stopped v1 instance. The task buffer and Telegram ingress queue resolve `resolveRuntimeIntent()` before acquiring a sandbox (they defaulted to legacy on a converted tenant). Operators get a non-waking `GET /debug/lifecycle` (policy + `keepAliveArmedAt`/`snapshotCadenceArmedAt`/`stableRecordId`) and `POST /api/admin/activity/refresh` (mirror the CLI schedule deterministically). The live drill `apps/fouria/test/e2e-v1/03_passive_health_and_wake.txt` (`bun run --cwd apps/fouria test:e2e:v1:live`, skip unless `E2E_V1_LIVE=1`) covers idle stop → automation wake, freeze/backup on the sleeping tenant, and billable keep-alive on a temporary billable binding. Tests: `src/sandbox-v1.test.ts` (keep-alive, instance type, stopped-snapshot skip, lifecycle), `src/activity/{alarm,scheduler}.test.ts` (occurrence guard/backoff, no give-up, boot mirror, durable v1 wake, non-waking memory watch), `src/index.test.ts` (freeze/backup skip on a stopped v1 tenant), `src/routes/api/activity.test.ts`, `src/tasks/index.test.ts` (v1 wake), `src/telegram-ingress/queue.test.ts` (explicit runtime); lerma `tenant_provisioner_pipeline_test.exs`; docs `WORKER-DEVELOPMENT.md`, `ENV-VARIABLES.md`, `sandbox-v1-scaffold.md`, `TESTING.md`, `test/e2e-v1/README.md`.
- **A tenant can be provisioned directly on the Sandbox v1 runtime ("born v1"), with no conversion.** Until now the only path to v1 was `migrate`, so a brand-new instance always started legacy. A fresh tenant is now identified by a durable marker written by Lerma at provisioning (`sandbox-migration/<canonical-slug>/runtime.json` in `BACKUP_BUCKET`: `{ version: 1, tenant, runtime: "v1", mode: "fresh", provisionedAt }`, create-only before the script deploy; the Worker only reads it). `src/sandbox-migration/runtime-marker.ts` owns the contract: `runtimeMarkerKey`, a strict validator (version 1, `runtime: v1`, `mode: fresh`, a tenant **matching the scope it was read from**, an ISO 8601 timestamp with an explicit zone), tri-state `readRuntimeMarker(bucket, scope)` via the shared `readJsonObject`, and `readFreshRuntime(env)` for the routes (no bucket ⇒ `absent`, unresolved identity ⇒ `unknown`). `resolveRuntimeIntent` resolves `absent` intent + `present` marker → `v1`; `unknown` marker → `unknown`; `absent` marker **with `SANDBOX_RUNTIME=v1`** → `pending` (fail closed: the instance asked for v1 but was never provisioned for it, so it must never boot the legacy container and create legacy state); otherwise `legacy`. The conversion intent still wins over a marker (the C8 record+pointer barrier decides when a converted tenant serves). `/control` `migrate` on a fresh tenant answers `409 MIGRATION_ALREADY_V1` ("there is nothing to convert") without addressing the DO or enqueuing (an unreadable marker → `503 SANDBOX_V1_RUNTIME_UNKNOWN`, since converting a tenant that may already be v1 is destructive), and `migration-status` answers `200 { state: 'absent', mode: 'fresh' }` without touching the DO (a converted tenant reports `mode: 'converted'`). `restoreTenantStateIfNeeded()` returns `false` for a pointer-less, record-less container so the tenant **onboards** in the v1 container with `OPENCLAW_RESTORED` unset, and the **first recovery point is taken right after that first successful boot**: `v1EnsureGateway` dispatches `runInitialSnapshot()` detached (`ctx.waitUntil`, never awaited, so readiness is not delayed), only when `verifyRecoveryPoint` is `pointer-absent` **and** `conversion.json` is absent — a converted tenant, a tenant that already snapshotted (cadence/`backup`/`checkpoint-park`), an unreadable/inconsistent pointer and a parked tenant are left alone, and a stopped container is never woken. A single-flight promise folds concurrent boots (owner run + adopted-gateway path) into one attempt, a settled flag keeps the adopting boot path at one R2 read per incarnation, and `takeOwnedSnapshot()` gives the cadence and this attempt one shared owner so they cannot snapshot the directory concurrently (the loser skips; the cadence keeps its alarm). Failures are caught and logged as `sandbox.v1.initial_snapshot_failed` with the boot still `ready`, and retried on a later boot (a new DO incarnation immediately, the same one after `INITIAL_SNAPSHOT_RETRY_MS` = 10 min); success logs `sandbox.v1.initial_snapshot`. The committed `DirectoryBackup` + `stable.json` pointer becomes the recovery point for park/restore/boot (retention applies; the cadence stays the backstop for the few seconds between readiness and the archive committing). Policy is unchanged: the matrix returns `legacy` outside `WORKER_PLATFORM === "staging"`, and the reserved `client` slug is never provisioned with a marker (the Worker treats a marker as authoritative for serving). Tests: `src/sandbox-migration/runtime-marker.test.ts`, `src/sandbox-migration/intent-gate.test.ts` (full matrix), `src/index.test.ts` (fresh serving, `409`, `absent`+`fresh`, pending without a marker, unreadable marker), `src/sandbox-v1.test.ts` (fresh onboarding without `OPENCLAW_RESTORED`; first recovery point committed after the boot, once under concurrent boots, a failed attempt tolerated + retried on a later boot, and skipped for a converted tenant with a complete pointer or a parked tenant); corpus `test/business-rules/sandbox-v1-cutover.txt`; docs `docs/WORKER-DEVELOPMENT.md` § "Provisioning an instance directly on Sandbox v1", `docs/ENV-VARIABLES.md`.
- **The ROOT debug tools work on Sandbox v1.** `/debug/*` (`DEBUG_ROUTES=true`, ROOT) runs against whatever client the request acquired, so on a converted tenant every route that touched the container was audited against `V1SandboxClient` and fixed where the legacy call has no v1 equivalent. A shared `readFinishedProcessLogs()` tries the legacy `logs()` stream and falls back to `output()` (the v1 handle has no replayable log stream), so `/processes?logs=true` and `/logs?id=` report a finished process's stdout instead of `Failed to retrieve logs`; the gateway log routes tail `/tmp/gateway-stdout.log` and now return `path` + `runtime`. `/restart-openclaw` gained a v1 branch (maintenance window → `killGateway` → `resetStartupCache`/`clearPersistenceCache`/`resetV1GatewayStart`, **no** legacy checkpoint and no `restore-needed` — a v1 restart destroys nothing); `/reprovision` checkpoints with `v1CreateSnapshot` and treats a restorable copy as the C8 `verifyRecoveryPoint` (not `backups/<id>/meta.json`), returning `recordId`; `/backup/:mode` snapshots through the DO and never boots a gateway on v1; `/destroy-container` also clears the startup cache and the isolate start dedupe; `/env` reports `sandbox_runtime_actual`. One refusal: `/debug/sleep` answers `503 { code: 'SANDBOX_V1_SLEEP_UNSUPPORTED' }` because v1 has no stop primitive (the tenant sleeps on the platform inactivity timer) — a `202` would claim work that never happens. One caveat: `/rescue-container` diagnoses/destroys, but v1 `containerFetch` never wakes a stopped container, so a sleeping tenant reads as unreachable. Process ids are per DO incarnation (empty list after an eviction → use `/debug/gateway-logs`). Tests: `routes/debug.test.ts` (`Sandbox v1 (converted tenant) debug routes`, 15 cases with a `__fouriaSandboxV1`-branded double) plus the dashboard card test; corpus `test/business-rules/sandbox-v1-serving.txt`; compatibility matrix in `docs/WORKER-DEVELOPMENT.md` § "Debug tools on Sandbox v1 (ROOT diagnostics)", QA drill in `docs/plans/sandbox-v1-migration-report.md` § 10.
- **Sandbox v1 idle snapshot cadence (PER-2).** A converted tenant's container is owned by `SandboxV1`, and raw `ctx.container` has no eviction hook (the legacy `onActivityExpired` → archive → stop path does not exist on SDK 1.0), so the platform's inactivity stop silently dropped everything written since the last committed record. The DO now runs its own cadence: `startContainer()` arms a durable purpose marker (`snapshot:cadence-pending`) plus `ctx.storage.setAlarm` (never postponing an earlier alarm, so a queued conversion keeps its slot), `alarm()` runs `runConversionAlarm()` then `runSnapshotCadence()`, and `v1Destroy()` clears both (unless a conversion is queued). `resolveSnapshotIntervalMs(env, inactivityTimeoutMs)` resolves `SANDBOX_V1_SNAPSHOT_INTERVAL_MS` (ms, `0` disables) → else the configured `BACKUP_INTERVAL_MINUTES` → else 1h, capped at **half** the inactivity timeout (staging 7.5m, dev 1h), so it never fires after the container would have slept. The turn never wakes a stopped container, skips while `#restoring`, skips when the recovery point (`stable.json`) is newer than the interval (deduping with the `ActivityScheduler`'s interval-gated v1 snapshot), then calls `v1CreateSnapshot({ trigger: 'backup' })` and re-arms; failures log `sandbox.v1.snapshot.cadence_failed`/`cadence_arm_failed` and never throw. `selectRecordsToPrune` now breaks a same-millisecond `createdAt` tie by record id, so retention is deterministic for back-to-back snapshots. Tests: `src/sandbox-v1.test.ts` (arm, disable, fire + re-arm, dedupe, never-wake, destroy), `src/sandbox-migration/records.test.ts` (tie-break); corpus `test/business-rules/sandbox-v1-persistence.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` (§ Sandbox v1 idle snapshot cadence), `docs/ENV-VARIABLES.md`.
- **Sandbox v1 restore by record id (PER-2).** `POST /api/admin/storage/restore/:id` on a converted tenant no longer falls into the legacy `.sqsh` path: the route opens the durable maintenance window (`restore`) first (the v1 `ensureGateway` branch honors it, so a request cannot restart the gateway under the swap), then `SandboxV1.v1RestoreRecord` stops the single gateway owner, restores the catalogued record's directory into `HOME_DIR` (SDK atomic swap), and by default advances the stable pointer so a cold boot restores the chosen record (`?setCurrent=0` applies without promoting). An unknown/unreadable record fails closed (`404`) without touching the container, a restore already in progress is refused by an in-DO flag, and the Worker then boots the restored state and reports `gatewayReady` (a boot failure never undoes the restore). v1 restores are synchronous — there is no legacy job queue for the record format. Tests: `src/sandbox-v1.test.ts` (restore + promote, no-promote, unknown record), `src/routes/api/storage.test.ts` (v1 branch, 404, maintenance window), `src/gateway/process.test.ts` (maintenance guard); corpus `test/business-rules/sandbox-v1-persistence.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` (§ Sandbox v1 restore by record id).
- **Sandbox v1 outbound virtual hosts (OCM-1 / BIL-1).** A converted tenant could not reach the Worker-owned virtual hosts: `TenantSandbox.outboundByHost` is a static on the **legacy** `Sandbox` class, and SDK 1.0 has no such statics, so `SandboxV1`'s container had no intercept for `otel.internal` (OTLP export / AI cost attribution), the `*.r2.internal` bridges (leads/docs/software) or `cdp.internal` (Browser Rendering via the container's local CDP proxy). The host map now lives in `src/sandbox-outbound.ts` (`SANDBOX_OUTBOUND_ROUTES`), used by both runtimes: the legacy class installs it as its `outboundByHost` statics, and `SandboxV1.startContainer()` registers every host with `ctx.container.interceptOutboundHttp(host, ctx.exports.SandboxOutboundGateway({ props: { host } }))`. Intercepts belong to the container instance, so they are re-registered on every start (including a restarted DO that finds the container running); a bundle without the export warns and keeps direct egress instead of failing the boot. Each intercepted request is dispatched by host to the existing bridge handler; an unknown host answers a structured `404` rather than throwing (an escaping exception surfaces in the container as `ECONNRESET`). Tests: `src/sandbox-outbound.test.ts`, `src/sandbox-v1.test.ts` (one intercept per host, re-registered on restart, missing export warns); corpus `test/business-rules/sandbox-v1-serving.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` (§ Sandbox v1 outbound virtual hosts).
- **Sandbox v1 rollout park (SEC-3).** A converted tenant can be parked for a container replacement. `/control` `checkpoint-park` on v1 drains live sessions to `read-only` (over the v1 gateway WS bridge, recording the previous modes durably), commits the recovery point with `v1CreateSnapshot` **fail-closed** (`502 SANDBOX_V1_PARK_SNAPSHOT_FAILED`, so no rollout starts without a restorable copy), and only then opens the durable park (`maintenance/replacement.json` `{ parked: true }`) and arms the exec lock; a stopped container is already parked (`skipped: 'not-running'`). The existing guards block writes (`tenantParkGuard` → `409 TENANT_PARKED`, `/control` refuses every action except `resume`/`checkpoint-park`, `assertTenantExecAllowed`), reads stay available. `resume` clears the park, unlocks exec, boots through the v1 DO and applies the recorded session modes after a real `/health` probe (or reports `sessionModes: "pending"`); resuming an unparked tenant never wakes it. `killGateway` stays process-scoped on v1. Tests: `src/index.test.ts` (park → 409 + durable record, resume, unparked no-op); corpus `test/business-rules/sandbox-v1-cutover.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` (§ Sandbox v1 rollout park).
- **Sandbox v1 boot, owner and repair (OCM-2).** The durable lanes that run outside a request now resolve the tenant's runtime themselves. `ActivityScheduler.sandbox()` is async and calls `resolveRuntimeIntent` before `acquireSandbox(..., runtime)`; a `pending`/`unknown` tenant is skipped (`activity.sandbox.skipped`) instead of being acted on. Previously the scheduler always acquired the **legacy** client, so after a conversion an alarm-driven wake or a scheduled snapshot would have booted the old container, archived the legacy filesystem and advanced the legacy `stable-backup.json` — silent divergence from the v1 recovery point. A `snapshot` backup job on a converted tenant now calls `SandboxV1.createSnapshot({ trigger: 'backup' })` (DirectoryBackup + catalog + pointer + retention) instead of the legacy stage/archive/finalize pipeline; v1 `restore`/`deprovision` jobs report `skipped` with an explicit reason. The DO's `startGatewayOwner` mirrors the legacy boot guards: it races the supervised launch's exit against the `/health` poll (a non-zero exit before the port opens fails in seconds; a zero exit is ignored as the lost-race path), tails `/tmp/gateway-stdout.log`, and queues the durable `doctor-repair` (`gateway-launch-state-database`) with a retryable error when the log matches a SQLite corruption signature. Tests: `src/activity/scheduler.test.ts` (v1 acquisition, v1 scheduled snapshot), `src/sandbox-v1.test.ts` (early exit fails fast, corrupt DB queues the repair); corpus `test/business-rules/sandbox-v1-cutover.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` (§ Sandbox v1 boot, owner and repair).
- **Sandbox v1 snapshots and the recovery point (PER-2).** A converted tenant now has a v1 persistence path of its own. `src/sandbox-migration/records.ts` is the catalog: `records/<id>.json` per committed `DirectoryBackup` entry plus the shared `stable.json` pointer, with `verifyRecoveryPoint` as the single C8 read (pointer→record: `pointer-absent`, `record-absent`, `record-unknown`, `pointer-record-mismatch`, or `complete`). `SandboxV1.v1CreateSnapshot()` writes the record through `backupDirectory` (post-encrypted in place with F4E1, failing closed on `BACKUP_ENCRYPTION_REQUIRED`), catalogs it **immediately**, advances the pointer, and prunes oldest-first beyond `BACKUP_MAX_VERSIONS` (default 3) while protecting the record just taken; a failed snapshot never stops the container. `/control` `backup`/`checkpoint` on a v1 tenant are served by the DO (`SANDBOX_V1_SNAPSHOT_FAILED` on error) and never enter the legacy `.sqsh` orchestration; other control actions still answer `503 SANDBOX_V1_CONTROL_UNSUPPORTED`. Cold boot restores what the pointer names (falling back to `conversion.json` for pre-catalog records) and refuses a conversion record without a pointer instead of onboarding over converted state. Tests: `src/sandbox-migration/records.test.ts`, `src/sandbox-v1.test.ts` (snapshot + prune, pointer restore, pointerless refusal), `src/index.test.ts` (`/control` v1 snapshot); corpus `test/business-rules/sandbox-v1-persistence.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` (§ Sandbox v1 snapshots and the recovery point).
- **Sandbox v1 conversion orchestration (S5 / PER-3).** `/control` with `X-FourIA-Control: migrate` starts the legacy → v1 conversion, and the Dispatch Worker forwards `migrate` (`CONTROL_ACTIONS`). The entry is **queue-only**: it calls `v1RequestConversion()` on the `SandboxV1` DO and answers immediately — the conversion streams the legacy squashfs, extracts it in the v1 container and uploads a `DirectoryBackup`, which is minutes of work that no Worker request can survive. The contract is `202 { state: 'queued' | 'running' }`, `200 { state: 'committed', recordId }`, `502 { state: 'failed', error, code: 'SANDBOX_V1_CONVERSION_FAILED' }` for a durably recorded failure, and `503 SANDBOX_V1_UNAVAILABLE` when the binding is missing or the DO is unreachable (nothing recorded → safe to retry); the staging/reserved-slug gate (`403`/`409`) still runs in middleware before any DO access and the route logs `sandbox.v1.conversion.requested`. The conversion runs on the DO `alarm()`: conversion first, cadence second, each wrapped so one cannot starve the other, and the gateway is launched afterwards with `OPENCLAW_RESTORED=true`. The job is durable (`migration:convert-request` marker + `migration:convert-status`, with a 1-based `attempt`) and idempotent (committed via DO state or the R2 record, queued/running in flight, new work only from absent/failed); `alarm()` orders `running` → `committed` **before** dropping the marker and before the launch, so a launch failure never retracts the commit, and a failure records `failed` with a bounded message instead of throwing. A lost alarm self-heals: `v1ConversionState()`/`v1RequestConversion()`/`v1Status()`/`v1EnsureGateway()` call `ensureConversionArmed()`, which re-arms only a non-terminal job and never wakes the container; `armConversionAlarm()` never postpones an earlier pending alarm (the cadence shares the slot, so conversion arming wins); `#conversionRun` is the single owner so overlapping alarm turns cannot start a second destructive conversion. `migrate` is an idempotent **enqueue** (it starts work from `absent`/`failed`), so polling reuses a separate **read-only** action, `migration-status` (`X-FourIA-Control: migration-status`; dispatch `CONTROL_ACTIONS` + echo of `state`/`recordId`, no snapshot CPU budget, no tenant-cache invalidation): it calls **only** `v1ConversionState()` — never `v1RequestConversion()` — so it cannot enqueue, start work or wake the container, and answers `200 { state: absent | queued | running | committed | failed, recordId?, error? }` (`absent` for a legacy tenant without addressing a v1 DO; `503 SANDBOX_V1_UNAVAILABLE` when the binding is missing or the read fails), never `202`. There is no synchronous conversion entry point — `v1ConvertNow` was removed from the DO, the RPC surface and `V1SandboxClient`. Tests: `src/sandbox-v1.test.ts` (queue/arm, alarm commit, cadence-after-conversion, failed recording, new attempt from failed, overlapping alarms, re-arm after a DO restart, no re-queue), `src/index.test.ts` (202 queued/running, 200 committed, 502 failed, 503 unreachable/binding, no inline conversion, converted-tenant migrate), `src/sandbox-v1-client.test.ts` (no inline method), `workers/fouria-dispatch/test/unit/control.test.ts` (status forwarding + echo); corpus `test/business-rules/sandbox-v1-cutover.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` (§ Sandbox v1 conversion orchestration), `docs/plans/sandbox-v1-migration-report.md` (drill 12), `docs/plans/sandbox-v1-scaffold.md`.
- **Sandbox v1 serving parity (S3 / OCM-1).** A converted tenant is now served by `SandboxV1` instead of failing closed. `src/sandbox-v1-client.ts` adds `V1SandboxClient` (a `Sandbox`-shaped client over the `SandboxV1` DO) with HTTP/WS transport (`stub.fetch` → `ctx.container.getTcpPort(port).fetch`, never waking a stopped container; `503 SANDBOX_V1_SLEEPING`), `isRuntimeActive`/`status`, a process RPC shim (`exec`/`getProcess`/`listProcesses` + `output`/`waitForExit`/`kill`), `killGateway` and `destroy`; unsupported legacy operations throw `SandboxV1UnsupportedOperationError` with `SANDBOX_V1_UNSUPPORTED_OPERATION`. `acquireSandbox(env, id, options, runtime)` selects legacy/v1 from the durable intent and fails closed on `unknown`; `index.ts` acquires the v1 client for a converted tenant and returns `503 SANDBOX_V1_CONTROL_UNSUPPORTED` for `/control` actions until S4. `ensureGateway` delegates to the DO (`v1EnsureGateway`), which adopts a healthy gateway, restores the committed v1 `DirectoryBackup` into an empty container (`OPENCLAW_RESTORED=true`), and runs `start-openclaw.sh start` as the single owner; `killGateway` delegates to `v1KillGateway`. `getHealthSnapshot` has v1 parity (non-waking `sleeping`, real `/health` readiness, 503-until-ready). Tests: `src/sandbox-v1-client.test.ts`, `src/sandbox-v1.test.ts`, `src/gateway/health-probes.test.ts`, `src/gateway/process.test.ts`, `src/sandbox-contract.test.ts`, `src/index.test.ts`; corpus `test/business-rules/sandbox-v1-serving.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` (§ Sandbox v1 serving), `docs/plans/sandbox-v1-scaffold.md`.
- **Sandbox v1 conversion (S2).** The v1 DO now owns the legacy→v1 data migration and the conversion barrier. `src/sandbox-migration/state.ts` is a tri-state reader (`absent`/`present`/`unknown`; only explicit absent permits legacy), `intent.ts` writes the per-tenant write-ahead intent, and `lease.ts` is a per-tenant R2 lease (create-only acquire, etag-CAS takeover, owner-scoped renew/release, fail-closed parse, TTL is liveness only). `SandboxV1.convertToV1()` acquires the lease, writes/validates the intent, opens the legacy recovery point (`stable-backup.json`, F4E1-decrypting `data.sqsh.enc`), streams the squashfs into the v1 container, extracts it with `unsquashfs -f -d /home/openclaw`, takes a v1 `DirectoryBackup`, writes `sandbox-migration/<tenant>/conversion.json` + `stable.json`, and verifies the pointer/record barrier (C8). The Worker refuses legacy routing/serving with `503 SANDBOX_V1_RUNTIME_PENDING` and `ensureGateway` throws `SandboxV1RuntimePendingError` once the intent is present/unreadable (C5). `Dockerfile.v1` installs `squashfs-tools`; the v1 serving path/readiness is S3 and durable job orchestration is S5. Tests: `src/sandbox-migration/{state,intent,lease,conversion,intent-gate}.test.ts`, `src/index.test.ts`, `src/gateway/process.test.ts`; corpus `test/business-rules/sandbox-v1-persistence.txt`; docs `docs/plans/sandbox-v1-scaffold.md`, `docs/WORKER-DEVELOPMENT.md`.
- **Sandbox v1 release tooling and deploy surface restored.** The `v1-sandbox` track now carries the P5b release tooling (`scripts/release-manifest.mjs`, `release-registry-digest.mjs`, `validate-runtime-image-pins.mjs` + tests), `Dockerfile.v1` copies `sandbox-shim` from `cloudflare/sandbox:1.0.3` (matching `catalog.@cloudflare/sandbox`), and `wrangler.jsonc` restores the `SandboxV1` `durable_object` container application (`fouria-sandbox-v1` local/production, `fouria-sandbox-v1-staging` staging) plus the `SandboxV1` DO binding in all three environments. `bun scripts/validate-runtime-image-pins.mjs` passes (3 v1 apps, 3 legacy apps). This is a deliberate reversal of audit finding F9 **for the migration track only**: the v1 image must be buildable/publishable for the staging lane, but no route selects `SandboxV1` and no tenant is converted. Tests: `scripts/*.test.ts`; docs `docs/plans/sandbox-v1-scaffold.md`.
- **Sandbox v1 migration gate (S1).** The cutover now has its authority and identity gate. `src/sandbox-migration/platform.ts` exposes the single `canMigrate(env)` predicate (true only for the trusted `WORKER_PLATFORM === "staging"` binding) and `migrationRefusal(env)`, which resolves the canonical tenant with `tenantSlugFromEnv` and refuses `null`/`default` and the reserved `client` slug. The `/control` entry runs the gate in a middleware **before** the sandbox stub is acquired, so a refused `X-FourIA-Control: migrate` performs no R2 write and no DO/container addressing (`403 MIGRATION_PLATFORM_FORBIDDEN`, `409 MIGRATION_SLUG_UNRESOLVED`, `409 MIGRATION_RESERVED_CLIENT`); an allowed staging request stops at `501 MIGRATION_NOT_IMPLEMENTED` until S2 adds `convertToV1`. The gate is migrate-only, so freeze/unfreeze/backup/checkpoint/resume keep their behavior on every platform and on `client`. `sanitizeCustomerSlug` now rejects the `default` sentinel in every resolver branch (C10), and Lerma provisions `WORKER_PLATFORM` (from `cf_environment`) plus `SANDBOX_V1_CUTOVER=false`. Tests: `src/sandbox-migration/platform.test.ts`, `src/sandbox-tenant.test.ts`, `src/index.test.ts` (`sandbox v1 migration gate`), corpus `test/business-rules/sandbox-v1-cutover.txt`, Lerma `tenant_provisioner_pipeline_test.exs`; docs `docs/ENV-VARIABLES.md`, `docs/WORKER-DEVELOPMENT.md`, `docs/OPERATOR-GUIDE.md` (§2.6).
- **Local containers no longer look like a platform bug, and the memory watchdog never races a boot.** A local restore failed with `Container failed to start` while `docker ps` showed only a `…-proxy` sidecar churning every ~20 s; the app container was never created. That is `net.ipv4.conf.all.src_valid_mark=1` (which Tailscale sets) making workerd's egress sidecar drop its own marked handshake ([cloudflare/workerd#6860](https://github.com/cloudflare/workerd/issues/6860)), not a FourIA defect. The dev server now warns before it builds images — `src/dev-container-preflight.ts` (pure `describeContainerNetnsProblem` + `readContainerNetnsSettings`, unit-tested) reads the two relevant sysctls and prints the fix (preferred: `net.core.devconf_inherit_init_net = 2`, which keeps Tailscale working; one-session alternative: `src_valid_mark=0`) plus the stale-sidecar cleanup; `FOURIA_SKIP_CONTAINER_PREFLIGHT=1` silences it for CI hosts. The same investigation exposed a real race: the durable memory watch sampled `/proc` with `sandbox.exec` — which auto-starts a stopped container — while a restore was probing it, producing `gateway.memory-read failed (launch-failed)` and a workerd `jsgInternalError` that aborted `container.start()`. `checkGatewayMemoryHealth()` now takes the bounded non-waking `isRuntimeActive()` read first and returns `{ action: 'skipped', reason: 'runtime-inactive' }` when the runtime is off, and `ActivityScheduler.runMemoryWatch()` returns before sampling while a boot/backup/maintenance job is active (queued/running/retrying), so observability never participates in a boot. The persistence lease acquire is now bounded too: a conditional `backup-lock` write that never settles is reported as `persistence.lock.write-timeout` and treated as a lost acquire, instead of holding the durable step (6 min of silence, then `job-step.interrupted`) — observed locally when taking over an expired lease. Tests: `src/dev-container-preflight.test.ts`, `src/gateway/memory.test.ts`, `src/activity/scheduler.test.ts`, `src/persistence.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `docs/DEVELOPMENT.md` § "Local containers never start", `docs/TESTING.md` § "E2E Tests (Local)", `apps/fouria/docs/WORKER-DEVELOPMENT.md` § memory watchdog.
- **The loading page's terminal shows the formatted durable job status instead of an empty `/tmp/startup.log`.** `/tmp/startup.log` does not exist during the durable boot job's restore phase (no `start-openclaw.sh` process yet) and can stay quiet while the gateway binds its port, so the terminal sat blank next to `Still working on "Starting OpenClaw Gateway" (188s)` — exactly when an operator wanted to know what the platform was doing. `src/activity/job-status-log.ts` (pure, unit-tested) formats the state the Worker already reads into `jobLog` lines: the `ActivityScheduler` boot job (`Boot running — launching the gateway (step 2/2, attempt 1/3), 4m in this step`), the restore-progress record (`Restore — Decrypting the backup (32%)`, with the existing idle/terminal annotations), and an active repair (`Repair running — openclaw doctor --fix, attempt 1/3`), including a retry wait and a truncated failure reason. `/api/startup/ws` pushes `jobLog` on every frame and the non-passive not-yet-ready `/api/health` / `/api/health-snapshot` carries it (plus `boot`/`repair`) for the HTTP polling fallback. `loading.html` renders the container log when it has content and otherwise the job log, labels the terminal source (`/tmp/startup.log` vs `boot job status`), and keeps the terminal visible during the whole restore phase — the brand logo now only appears when there is nothing at all to show. Tests: `src/activity/job-status-log.test.ts`, `src/assets/loading.test.ts`, `src/routes/public.test.ts`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` § Startup Progress.
- **The agent browser reaches Cloudflare Browser Rendering through a private `cdp.internal` bridge, with no Cloudflare Access bypass.** The tenant `/cdp` route is behind the host Access application (`302`), so it is no longer the agent transport. The container runs `cdp-proxy.mjs` (`127.0.0.1:9223`), a WS↔HTTP proxy that serves CDP discovery and forwards commands to the Worker's `cdp.internal` handler (`TenantSandbox.outboundByHost`). The handler calls the tenant's **Sandbox Durable Object** (`cdpCommand`/`cdpClose`), which launches the browser from `env.BROWSER` and keeps the session across OpenClaw's per-action WebSockets (serialized command queue). `src/cdp-session.ts` reuses the `/cdp` `handleCDPMethod`, so the transports cannot drift; the Browser Rendering credential never enters the container. Moltlazy writes a fill-only remote-CDP profile (`browser.profiles.cloudflare.cdpUrl=http://127.0.0.1:9223`, `defaultProfile=cloudflare`) and adds `tools.alsoAllow: ["browser"]` (the `coding` profile excludes it); restored boots re-apply `--only gateway,browser`. Tests: `src/cdp-session.test.ts`, `src/sandbox-cdp-bridge.test.ts`, `src/index.test.ts` (host registration/health), `packages/moltlazy/tests/config/browser.test.ts`, `tests/config/tools.test.ts`, E2E `test/e2e/r_browser_remote_profile.txt` (registration, `open`, cross-call `tabs`). Docs: `apps/fouria/docs/WORKER-DEVELOPMENT.md` § Browser Rendering, `apps/fouria/docs/OPERATOR-GUIDE.md` §8.2, `docs/TESTING.md`, `packages/moltlazy/AGENTS.md`. Known limit: Playwright-backed `snapshot`/`evaluate` still need CDP session multiplexing; raw `open`/`tabs`/navigation/screenshots work.
- **A wedged container is no longer read as an empty one, and memory pressure is observed from the durable alarm.** `tenant-tarjetero-dev` lost ~7 h 45 m of work on 2026-11-03: the Sandbox DO could not open a control session to a starved container (`ContainerControlConnection.fetchUpgradeAttempt` → `AbortError`), `containerHasState()` swallowed every probe error and returned `false`, and the recovery path therefore started a **destructive 450 MB restore over a live filesystem**. It timed out at 180 s, wrote `restore-target.json { outcome: 'unknown' }`, and every later boot was refused with `RestoreInFlightError (restore-outcome-unknown)` for 11 min at a time. Two defects fixed and one policy added (design: [docs/plans/graceful-shutdown-and-emergency-checkpoints.md](/llms/platform/reference/plans/graceful-shutdown-and-emergency-checkpoints/index.md)). **(1) Absent ≠ unknown:** `probeContainerState()` returns `present | absent | unknown`; `restoreIfNeeded` restores only on `absent` and throws the retryable `ContainerStateUnknownError` on `unknown`, without writing the `restore-needed` latch (a runtime with no `exists()` probe keeps the historical "assume empty" reading). **(2) Continuous observation:** `checkGatewayMemoryHealth()` is the single sampler for `/api/health*` **and** the `ActivityScheduler` alarm (`runMemoryWatch()`), so memory is sampled even when no browser polls health — the reason `gateway-resources` had no rows for ~17 h through the incident. Every fresh sample also writes `gateway-memory-sample` (`state: running|absent`, container limit) so a gap means "nobody sampled", and `healthWsHandler` now passes its `executionCtx` (the detached watchdog promise was previously cancelled with the frame). **(3) Two-stage policy:** `decideMemoryAction()` checkpoints at 60 % of the cgroup limit and restarts at 75 %, escalating to `checkpoint-and-restart` when the newest archive is older than `GATEWAY_MEMORY_LOSS_BUDGET_MS` (15 min); the alarm enqueues the checkpoint as a `trigger: 'memory-pressure'` snapshot job before restarting, and `maybeRestartGatewayForMemory` delegates to the same policy so thresholds cannot drift. Tests: `src/persistence.test.ts` (`probeContainerState` present/absent/unknown/malformed, restore refused with no latch), `src/gateway/memory.test.ts` (`decideMemoryAction` ladder, `resolveMemoryCheckpointBytes`), `src/metrics.test.ts` (`trackGatewayMemorySample`); corpus `test/business-rules/operational-modes.txt`; docs `docs/plans/graceful-shutdown-and-emergency-checkpoints.md`.
- **Platform docs are versioned per channel, and instances read the one that matches them.** `webs/fouria-docs` now publishes `latest` (built from the newest `v*` release tag; `/latest/`, aliased by the root) and `staging` (built from `develop`; `/staging/`). Each build runs with its own `base_url` and a channel-aware `sync-docs.ts` (`--channel`/`--base-path`), so pages, assets, and `/llms/**` mirrors never cross channels; `versions.json` records the channel, commit, and component versions. A new `release-notes/` section is compiled from each component's `CHANGELOG.md`, making the notes factually compatible with the channel. `build-deploy-tree.ts` assembles both channels from their own checkouts (the release tag for latest, develop for staging) and the production deploy carries both, so `/staging` never lags. `buildEnvVars` forwards `FOURIA_DOCS_CHANNEL` / `FOURIA_DOCS_BASE_URL` / `FOURIA_DOCS_FALLBACK_URL` / `FOURIA_VERSION` (`src/docs-channel.ts`), and the `fouria-docs` skill + dev personas confirm the build via `versions.json` before quoting release notes. Tests: `src/docs-channel.test.ts`, `src/gateway/env.test.ts`, `webs/fouria-docs/scripts/channel.test.ts`; docs `webs/fouria-docs/README.md`, `docs/ENV-VARIABLES.md`, `docs/VERSIONING.md`, `docs/TESTING.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- **OTLP ingest and promotion are private transports — neither has a public route.** A staging capability run could not emit OTLP (`302` to the Access login) or push a release, and the first attempt at a fix (path-scoped Access `bypass` apps + a `PROMOTION_PUSH_TOKEN` + a Dispatch `isMachinePath` exemption) still left two internet-facing endpoints gated only by a bearer. Replaced with the two transports the runtime already uses. **(1) OTLP → sandbox bridge:** the container exports to `http://otel.internal/v1/{traces,metrics}`, resolved by `TenantSandbox.outboundByHost` inside the Worker (`src/sandbox-otel-bridge.ts`), so the traffic never leaves the sandbox ↔ Worker control plane; `/api/otel` is no longer mounted, the ingest logic moved to `src/otel-ingest.ts` (framework-free, constant-time secret compare, no query-string token), and `buildEnvVars` always points the exporter at the bridge. `OTEL_SHARED_SECRET` is kept as the bridge's inner credential. **(2) Promotion → R2:** the dev instance stages the sanitized bundle in its **own** bucket (`promotion/staged/<id>.tar.gz` + `promotion/pending.json`), and the production instance reads it through a read-only `PROMOTION_SOURCE_BUCKET` binding to its sibling dev bucket (`services/promotion/r2-source.ts`). Both buckets stay separate; there is no `import` route, no push token, no cross-host fetch. Production applies the pending release **on boot, after the restore barrier and before the launch**, or on `POST /api/admin/releases/apply`; the apply runs under the shared persistence lease (`withPersistenceLease`), is skipped while the tenant is parked or while a maintenance window/restore is in flight, and is idempotent (the container release pointer, which is part of the snapshot, is the applied marker — a restore cannot silently roll it back). Lerma provisions the sibling binding (`promotion_source_bindings/2` in `Instances` and `TenantProvisioner`) only for `environment: "production"` and ensures the dev bucket exists before binding it. Removed: `PROMOTION_PUSH_TOKEN` everywhere, `AccessPolicy.ensure_machine_access/1` and its path apps, the Dispatch `isMachinePath` exemption, `/api/otel` and `/api/admin/releases/import`. Tests: `src/sandbox-otel-bridge.test.ts` + `test/otel-e2e.test.ts` (bridge routing/auth/tenant scoping/limits), `test/otel-e2e` reworked for the virtual host, `src/services/promotion/r2-source.test.ts` (stage, pending validation, target gating, idempotency, park/busy/skip reasons, applied pointer), `src/routes/api/releases.test.ts` (stage 202, dev-only 409, apply+checkpoint, status availability), `src/gateway/env.test.ts`, `src/index.test.ts` (bridge registered, 401 without secret), `tenant_provisioner_pipeline_test.exs` (`PROMOTION_SOURCE_BUCKET` binding); corpus `tenant-isolation.txt`, `operational-modes.txt`; docs `ENV-VARIABLES.md`, `OPERATOR-GUIDE.md` (§5.4f), `WORKER-DEVELOPMENT.md`, `TESTING.md`, `apps/lerma/README.md`, `secrets-manifest.schema.json`.
- **A recovery point whose TTL lapsed is healed on the restore path, and a terminal boot failure is no longer presented as a retry.** `staging-client-dev` sat on `Restore attempt failed during "Restoring the backup" — retrying — no progress for 10 min`: not a slow restore, but three ~250 ms failures followed by silence. The SDK refuses an archive past `createdAt + meta.json.ttl` (`BackupExpiredError`) _before_ touching the container, and all three catalogued archives carried the old 7-day TTL (expired 2026-10-29) — the 10-year `SNAPSHOT_ARCHIVE_TTL_SECONDS` + `extendArchiveTtl` repair from #523 only runs as the `repair` step of a **snapshot job**, which a tenant that cannot boot never gets (chicken-and-egg). `ensureArchiveRestorable` (`src/persistence.ts`) now widens `meta.json.ttl` right after `restore-target.json` is published, in both `restoreIfNeeded` and `restoreBackupById`, and the SDK's expiry wording (`Backup &lt;id&gt; has expired (…)`, the code lives on the error object, not the message) is classified by `isUnrecoverableArchiveError` so a rewrite that cannot take effect drops the dead entry instead of looping. The page-side wedge is fixed too: `readRecoveryState` stops reporting a `restore-target.json` with `outcome: "failed"` as `restoring`, the `ActivityScheduler` records `describeRestoreTerminalFailure` when the boot job (or a manual restore job) exhausts its attempts — label `Restore failed — &lt;reason&gt;`, `terminal: true`, no idle annotation, cleared by the next `boot.start` — and `loading.html` gives a `failed` boot precedence over the leftover progress bar and renders the reason (`Boot failed after N attempts: &lt;reason&gt;. Refresh the page to retry.`). Operator lever (R2-only, works with the gateway down): `POST /api/admin/storage/reconcile` — note the `-X POST`; `GET` falls through to the gateway proxy and answers `503 Gateway not ready`. Tests: `src/persistence.test.ts` (`ensureArchiveRestorable` before the SDK call, already-widened untouched, expiry classified; `readRecoveryState` failed/unknown), `src/activity/restore-progress.test.ts`, `src/activity/scheduler.test.ts` (terminal label, cleared on a new boot), `src/assets/loading.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md` (§4.4, §5.4e).
- **A gateway launch blocked by a corrupt OpenClaw state database now queues a `doctor --fix` repair and relaunches, instead of looping on "Starting OpenClaw Gateway".** A restored tenant can come back with a damaged `state/openclaw.sqlite`: OpenClaw's schema preflight fails with `database disk image is malformed` (`readSqliteUserVersion` → `inspectStateDatabaseSchema`) and the gateway exits within seconds, so relaunching can never help. Two fixes. **(1) The crash is detected immediately:** the readiness wait now races a third signal, `watchGatewayEarlyExit` — the atomic cold start `exec`s the gateway, so a **non-zero** exit of the supervised process before the port opens is a terminal launch failure (previously each attempt waited out the full `STARTUP_TIMEOUT_MS`, the ~5-minute stale step and 311s page counter). A **zero** exit stays ignored (the "lost the startup-lock race to a healthy owner" path is decided by the existing port/lock adoption check), and `waitForExit` plumbing errors (a DO code update resets the control session) park the watcher instead of failing a healthy boot. **(2) The documented repair is queued:** `src/gateway/state-db.ts` classifies the launch diagnostics (`database disk image is malformed`, `file is not a database`, `SQLITE_CORRUPT`, …) and `ensureGateway` enqueues the durable `doctor-repair` job with `reason: 'gateway-launch-state-database'`, then throws the retryable `GatewayStateDatabaseRepairError`. That reason makes the job (a) skip the post-restore **gateway** gate in `preflight` (`awaitRestoredGatewayBeforeRepair({ requireGateway: false })` — waiting for a gateway the corruption blocks is a permanent deferral; the active-restore / maintenance-window checks stay) and (b) **re-queue the durable boot** when the repair succeeds, since the failed launch would otherwise never be retried; re-requests fold into the active job (`MAINTENANCE_JOB_BUSY`) and the `ACTIVITY`-less (local dev) path reports `no-scheduler` instead of throwing. The loading page follows it: `/api/startup/ws` pushes `repair` (`ActivityScheduler.getActiveMaintenanceJob`, a cheap read) and the page renders "Repairing the OpenClaw state database (openclaw doctor --fix, attempt N)…" and suppresses the slow-step notice, instead of the container's stale launch step. Structured events: `gateway.launch.state-database-repair` (outcome `queued`/`busy`/`no-scheduler`/`failed` + `signature`) and `sandbox.process.failed` with `failureReason: 'state-database-corrupt' | 'gateway-exited'`. Tests: `src/gateway/state-db.test.ts`, `src/gateway/process.test.ts` (corrupt launch queues the repair, early non-zero exit is a fast failure, a transport-dropped exit watch does not fail a healthy boot), `src/activity/maintenance-jobs.test.ts` (reason persists, active-job read), `src/activity/scheduler.test.ts` (no gateway gate for the reason, boot re-queued after success, ordinary repair unchanged), `src/routes/public.test.ts` (repair pushed/omitted), `src/assets/loading.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md` (§5.4f).
- **A restored container no longer re-runs the setup body when the launch happens in a different isolate than the restore.** Instances recovered from a recovery point (rollout/reprovision, idle-sleep wake) were launching `start-openclaw.sh start` with the full setup body — re-onboarding and re-installing the plugins over a snapshot that is supposed to be authoritative. The zero-data-loss signal was keyed on the **in-memory** restore status (`getRestoreStatus()` → `hadBackup && restored`), which only exists in the isolate that ran the restore. The launch frequently runs elsewhere: the durable boot job is two steps (`restore`, then `launch` in a later alarm, so a DO eviction loses `_restoreState`), any request that wins the startup lock during the restore→launch window launches from its own isolate (where `restoreIfNeeded` only runs the cheap request-scoped check and records nothing), and `clearPersistenceCache()` (freeze / resume / stale recovery) resets it. New `hasRestoredTenantState(sandbox)` (`src/persistence.ts`) combines the in-memory result with the durable container-state probe (`containerHasState` — the restored `openclaw.json`, legacy `/root` or v1 `/home` layout); the image ships no config, so "state present" is exactly "already initialized/restored". `ensureGateway` now sets `OPENCLAW_RESTORED=true` from that decision, and a brand-new tenant (no state, no backup) still gets the full setup body. Tests: `src/persistence.test.ts` (`hasRestoredTenantState`: in-memory fast path, container-state fallback, brand-new container, probe failure), `src/gateway/process.test.ts` (launch env); corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- **Zero-loss container rollouts: the tenant is parked before an image rollout and resumed after the replacement (`checkpoint-park` / `resume`).** A container replacement always starts from a fresh disk, so it restores the checkpoint taken just before the rollout — anything written after that archive was produced was lost. The Worker now offers a **park** variant of the verified checkpoint: `POST /control { X-FourIA-Control: checkpoint-park }` takes the same snapshot but leaves the tenant drained and **write-blocked**, and `X-FourIA-Control: resume` releases it. Enforcement is two-layered (`src/tenant-park.ts`): `tenantParkGuard` middleware refuses mutating (`POST`/`PUT`/`PATCH`/`DELETE`) `/api/admin/*` and `/debug/*` requests with `409 { code: "TENANT_PARKED" }` (reads stay available; the durable record is the cross-isolate source of truth), and `startShell` calls `assertTenantExecAllowed` before the launch RPC so background/spawned work that never passes a route cannot start a process either. The park's own path (quiesce/snapshot/restore) runs inside `withTenantExecBypass`, the only way past the lock. `maintenance/replacement.json` gained `parked: true` (`beginMaintenance(..., { parked: true })`, `isTenantParked`); an incomplete park un-arms the lock, closes the window and restores service exactly like a failed plain checkpoint, and the 15-minute TTL stays the crash backstop. `resume` deliberately does **not** pass `skipRestore` (the replacement container is empty and must rehydrate), clears the stale in-memory restore barrier (`clearPersistenceCache`, as `freeze` does), and is a no-op that never wakes a sleeping tenant. Other control actions (`backup`/`freeze`/`unfreeze`) are refused while parked. The Dispatch Worker allows/forwards both actions (`CONTROL_ACTIONS`/`isControlAction`), keeps the raised inline-snapshot CPU budget for `checkpoint-park`, and whitelists the `parked`/`resumed` echo fields. Lerma's `@upgrade_pipeline` now parks at `:checkpoint` (**fail-closed**: an unavailable checkpoint aborts before the container is replaced; a sleeping instance is exempt), resumes at a new `:resume` step immediately after `:dispatch_deploy`, and resumes best-effort on both error paths. The bundle-only path `Instances.upgrade_instance/3` (used by the staging validation promotion API) follows the same contract: parked checkpoint first, a failed or non-parked (`parked: false`) response aborts before the script upload, the park is released after the rollout settles and on every error path, and a sleeping instance skips the park entirely (no resume is sent). **Live sessions are switched to read-only first and given a turn to finish:** before the quiesce, `drainLiveSessions` (`src/gateway/session-drain.ts`, one `operator.read`+`operator.write` Gateway WS) reads `sessions.list`, patches every session to `permissionMode: 'read-only'` (exec denied, managed mutation tools omitted), and polls until the running turns end (bounded by `SESSION_DRAIN_TIMEOUT_MS`, 120 s — an outliving turn is reported as `sessionsDrained: false` and the park proceeds, so a slow turn never blocks a release; every write it made is already in the snapshot). Sessions already read-only are skipped and **new** sessions are untouched (the mode is per existing session, never a default). Because the mode lives on the session row it is inside the snapshot, so the previous modes are recorded outside the container (`maintenance/session-modes.json`, `recordPendingSessionModes`) and restored once a gateway is serving — `resume` waits `SESSION_MODE_RESTORE_WAIT_MS` and applies them, and `launchRestoredState` (`post-restore.ts`) applies them after any restore (the map is cleared only when every session was restored, so it retries on the next boot). An aborted park also restores them immediately. **Telegram users are told by the ingress, not the model:** while parked, `POST /telegram-webhook/:account` sends `TELEGRAM_PARKED_NOTIFY_MESSAGE` directly via the Bot API (`notifyTenantParked`), records a `parked` ingress event and returns `200` without enqueuing or waking; the queue consumer answers an already-queued update the same way and acks instead of retrying for the whole rollout. Tests: `src/tenant-park.test.ts`, `src/maintenance.test.ts`, `src/gateway/sandbox-exec.test.ts`, `src/gateway/session-drain.test.ts`, `src/gateway/session-mode-restore.test.ts`, `src/telegram-ingress/{notify,status,route,queue}.test.ts`, `src/index.test.ts`, `workers/fouria-dispatch/test/unit/control.test.ts`, `apps/lerma/test/lerma/cloudflare/containers_test.exs`, `tenant_provisioner_pipeline_test.exs`; corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` (§ Rollout park), `apps/fouria/docs/OPERATOR-GUIDE.md` (§2.5), `apps/lerma/README.md`. Not covered (separate work): the post-rollout readiness gate and fleet cohort/abort policy.
- **The dev memory watchdog is instance-aware, and new tenants default to a viable instance.** `GATEWAY_RSS_RESTART_BYTES` was a hardcoded 3 GiB (justified as "3 of 4 GiB"); on a resized `standard-3` (8 GiB) that is only 37% of the limit, so the watchdog would restart a healthy gateway. The threshold is now `resolveMemoryRestartBytes()` — **75% of the container's cgroup memory limit** (`memory.max`, with a cgroup v1 fallback), read in the same `/proc` exec as the RSS sample: 3 GiB on a 4 GiB `standard-1`, 4.5 GiB on `standard-2`, 6 GiB on `standard-3`, 9 GiB on `standard-4`. Because the limit is re-read on every sample, an instance-type rollout updates the threshold without a Worker deploy. Instances below 3 GiB are skipped (`instance-too-small`) — `basic` (1 GiB) cannot hold a healthy gateway (~1.1 GiB steady state), so a restart there can only loop; the gate sits under the 4 GiB `standard-1` class so a cgroup that reports slightly less than nominal still gets a watchdog. `Lerma.Instances.InstanceTypes.default/1`, the `Instance` schema default, and `Containers.deploy_script_and_container/1`'s fallback now default to **`standard-1`** instead of `basic` (1 GiB), which OOM-looped any instance provisioned without an explicit type. Resizing is a Cloudflare rollout (`change_instance_type/2` → `full_auto` with `target_configuration`) that replaces the container, so plan one cold start + R2 restore per tenant. `OPENCLAW_NO_RESPAWN=1` is kept (inert on every instance size); a real process-level replacement would need `OPENCLAW_SUPERVISOR_MODE=external` + a supervisor that preserves the single-owner startup lock. Tests: `src/gateway/memory.test.ts`, `src/routes/public.test.ts`, `apps/lerma/test/lerma/instances/instance_types_test.exs`, `apps/lerma/test/lerma/instances_test.exs`; docs `docs/plans/sandbox-memory-usage.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md`, `apps/lerma/README.md`.
- **Phase 1 captures a Tier 1 workspace snapshot on eviction (flag-gated).** `TenantSandbox.onActivityExpired()` now takes a platform filesystem snapshot immediately before eviction when `WORKSPACE_SNAPSHOT=true` and commits the handle at `workspace/snapshot.json` (versioned, image-pinned, one head handle). Capture never throws: `captured` / `unsupported` / `failed` / `skipped` become a `workspace-snapshot` Analytics Engine row, and the archive-then-stop path runs unchanged — Phase 1 never boots from the handle. **Runtime prerequisite:** Cloudflare only supports snapshots on a container application with `scheduling_policy: "durable_object"`; the tenant container is still on the `default` policy (the legacy SDK calls `container.start()` without an image), so the API is unavailable there and every attempt reports `unsupported`. `SandboxV1` (already `durable_object`) implements the same `snapshotWorkspace()`, so the capability is real where the runtime allows it; the flag stays off until the tenant app is migrated. Tests: `src/workspace-snapshot.test.ts`, `src/sandbox-do.test.ts`, `src/sandbox-v1.test.ts`, `src/metrics.test.ts`; corpus `test/business-rules/cost-tracking.txt`; docs `docs/plans/persistent-workspace-phase1.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md` §"Workspace snapshots on eviction".
- **Persistence policy is measured before it is reworked (Phase 0).** The backup-policy rework (two tiers: native container snapshots as the routine save path, the F4E1 archive as the recovery and portability tier) starts with telemetry and no behaviour change. Three Analytics Engine events now describe the current system: `persistence-snapshot` (end-to-end vs `sandbox.createBackup()` duration, captured bytes read once from the archive's `meta.json`, `trigger`, and a skip/failure reason — `interval-not-elapsed`, `unchanged`, `not-running`, `backup-lock-busy`), `persistence-restore` (duration, bytes, `recovery-point` vs `explicit`, and `archive-unusable` recorded as a skip rather than a generic failure), and `backup-change-window` (continuous dirty time from `dirtySince` plus scan time, with the watch decision). `SnapshotPhaseContext.trigger` labels every snapshot; `SnapshotArchive.sizeBytes` is the one added metadata read. No snapshot, restore, retention, or eviction decision reads these rows. Queries and the Phase 1 gates: `docs/plans/persistent-workspace-phase0.md`. Tests: `src/metrics.test.ts`, `src/persistence.test.ts`, `src/activity/scheduler.test.ts`, `src/backup-watch/watch.test.ts`; corpus `test/business-rules/cost-tracking.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md` §"Persistence policy telemetry (Phase 0)".
- OpenClaw migrated to **2026.10.9** (`catalog.openclaw` + the `@openclaw/gateway-client` / `@openclaw/gateway-protocol` pins in the root `package.json`; the plugin repo tracks and publishes the same release). 2026.10.9 is a **security patch release** (vulnerable dependency remediations, credential handling, workspace-only restrictions for Gateway-built delegation and media tools, malformed MCP address rejection) with **no config-schema changes** — the moltlazy `openclaw config validate` cctr contract passes unchanged, so no `packages/moltlazy` source change was needed. The Node engine floor stays `>=24.22.1 <25 || >=26.12.1`, so `apps/fouria/Dockerfile` keeps Node v24.22.1. The same PR **hardens the sandbox memory limits** in moltlazy: `agents.defaults.compaction` now uses `maxActiveTranscriptBytes: '8mb'` (from 20mb), `keepRecentTokens: 12000` (from 20000), and `midTurnPrecheck.enabled: true`; `session.maintenance` now enforces `maxDiskBytes: '512mb'` / `highWaterBytes: '384mb'`, `maxEntries: 500`, `pruneAfter: '14d'`, `resetArchiveRetention: '14d'`, and `coldStorage: { enabled: true, afterDays: 7 }` (OpenClaw's 10 GB / 5000-entry / 30-day defaults do not fit a 4–12 GB container). Research findings: OpenClaw exposes no knob for the worker-thread pool or the RSS threshold, and `basic` (1 GiB) is below the gateway's ~1.1 GiB steady-state RSS, so `standard-1` is the practical instance floor and a hard restart remains the only RSS reset (see `docs/plans/sandbox-memory-usage.md`). No component CalVer bump — the release train handles that.
- **Runnable end-to-end example for declared software across a restore.** `test/e2e/fixture/software/gog-goplaces.json` is the example manifest for a gateway whose `openclaw/openclaw/gog` and `.../goplaces` skills (ClawHub: `@steipete/gog` / `@steipete/goplaces`) need the `gogcli` (`gog`) and `goplaces` binaries, neither of which is in the image. It installs each binary from its pinned GitHub release (`curl … | tar -xz -C /usr/local/bin`, `gog --version` / `goplaces --version` as `verify`), so the packages are **outside** the `/home/openclaw` snapshot root and do not survive a container replacement. Two proofs: the **local-only** harness suite `test/e2e-backup/zzz_software_binaries_restore.txt` (`%skip` unless `E2E_LOCAL=true`: upload via `PUT /api/admin/software/manifest` → `POST /api/admin/software/provision` → `openclaw skills install @steipete/{gog,goplaces} --global` → backup (`?inline=1`) → `/debug/destroy-container` → restore → binaries back with an advanced report `startedAt`, skills still listed), and the dependency-free `scripts/software-restore-smoke.sh` (`bun run test:software:smoke`), which drives the same contract directly against the sandbox image (provision → install skills → snapshot → fresh container → restore → assert binaries gone/skills back → re-provision → assert versions and a fresh report) and passes locally in ~2 min. Docs: `test/e2e-backup/README.md`, `docs/TESTING.md`.
- **Gateway memory is observable from the Worker, and a dev instance self-heals before the container limit.** Investigating a `tenant-tarjetero-dev` "reboot" required SSH-ing into the container to read `/proc`, `dmesg` and the gateway log, because nothing exposed the gateway's RSS. **(1) Observability (plan Phase 1):** `src/gateway/memory.ts` reads `VmRSS` + `Threads` from `/proc/<pid>/status` for the gateway **server** (`pgrep -x 'openclaw-gatewa'`, the same exact `comm` the soft reload targets, zombies skipped) in ONE shell exec, throttled to `GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS` (60s) so the frequent health poll adds at most one exec per interval. `attachGatewayResources()` (`routes/public.ts`) attaches it to `/api/health`, `/api/health-snapshot` and `/api/health-ws` as `gateway: { pid, rssBytes, threads }` — only while the snapshot is `running`, so a sleeping tenant is never woken — and writes a `gateway-resources` Analytics Engine row (`double1` = RSS bytes, `double2` = threads). **(2) Watchdog (plan Phase 2.4):** on a **dev** instance (`isDevAgentsEnabled`) at `GATEWAY_RSS_RESTART_BYTES` (3 GiB of the 4 GiB container) the gateway is replaced via `killGateway` + `resetStartupCache` + `ensureGateway({ skipRestore: true, bypassMaintenance: true })` — the only lever that reclaims the per-`worker_thread` V8 heaps — with a 30-minute cooldown claimed **before** acting so concurrent health polls cannot double-restart, dispatched detached so a multi-minute replacement never delays a health response. Deliberate design: the threshold guards the **container limit**, not OpenClaw's 1.5 GiB `rss_threshold` warning (which fires permanently on a healthy ~2.1 GiB dev gateway and would make the watchdog a restart loop), and there is **no soft-reload step** (a measured in-process `SIGUSR2` restart does not reclaim RSS: 1.14 → 1.38 → 1.18 GiB). Production is excluded. Tests: `src/gateway/memory.test.ts`, `src/metrics.test.ts` (`trackGatewayResources`), `src/routes/public.test.ts` (sample attached + AE row, one read per interval, no sample when not running, no watchdog on a non-dev instance); corpus `test/business-rules/operational-modes.txt`; docs `docs/plans/sandbox-memory-usage.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md` (§5.4d).
- OpenClaw migrated to **2026.10.8** (`catalog.openclaw` + the `@openclaw/gateway-client` / `@openclaw/gateway-protocol` pins in the root `package.json`; the plugin repo tracks and publishes the same release). 2026.10.8 is a **bug-fix/security release**: plugin lifecycle repairs (older plugins can process incoming messages again after an upgrade, scheduled plugin work no longer stops after startup/reload, and Gateway startup repairs stale managed plugin links left by an external OpenClaw reinstall — the state a warm R2 restore can produce), session-store fixes (relative session-store paths no longer fork a second database; `openclaw doctor --fix` repairs an already-split store), Doctor robustness on very large retained session histories (stack overflow / retained memory), and a trusted-proxy browser-auth fix that no longer writes `gateway.auth.password` or restarts the Gateway on first browser use. It also hardens POSIX exec allowlists (`test -v` / `[ -v` are no longer auto-trusted; moltlazy writes no exec allowlist, and the dev-only `security.installPolicy allow` is unaffected). No config-schema changes — the moltlazy `openclaw config validate` cctr contract passes unchanged, so no `packages/moltlazy` source change was needed — and the Node engine floor is unchanged at `>=24.22.1 <25 || >=26.12.1`, so no Dockerfile change. No component CalVer bump — the release train handles that.
- **Software provisioning is now usable mid-life and honest about its state.** Three gaps closed around the `softwareProvisioning` feature. **(1) On-demand reconcile:** `POST /api/admin/software/provision` (ADMIN/ROOT) re-materializes the current R2 manifest, runs `moltlazy software provision` to completion (bounded by `SOFTWARE_PROVISION_RUN_TIMEOUT_MS`, 5 min) and returns the fresh report, so a manifest uploaded or corrected mid-life no longer waits for the next cold start; `409 { code: 'disabled' | 'no_manifest' }` and `502 { code: 'failed', error }` are distinct outcomes. **(2) No cross-boot tick bleed:** `runSoftwareProvision` truncates `/tmp/software-progress.log` before every run (`resetProgressLog`) — `/tmp` survives an in-place `/debug/restart-openclaw`, so the boot terminal used to interleave ticks from earlier boots. **(3) Truthful dashboard states:** `GET /api/admin/software/{manifest,report}` now return `enabled` (the flag), and the homepage widget separates "provisioning is disabled", "instance is asleep", "no report yet" and "no software declared", with a **Run provisioning** button (ADMIN/ROOT) that renders the on-demand report and surfaces its error. `parseSoftwareReport` moved into `src/gateway/software-provisioning.ts` so the report, run and route paths share one shape check. Tests: `src/gateway/software-provisioning.test.ts`, `src/routes/api/software.test.ts`, `src/client/pages/__tests__/HomePage.test.tsx`, `packages/moltlazy/tests/software/run.test.ts` (progress-log reset), E2E `test/e2e/zzz_software_provisioning.txt` (on-demand run must install the new package and **skip** the already-installed one); corpus `test/business-rules/dashboard-api-contract.txt`, `operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- OpenClaw migrated to **2026.10.6** (`catalog.openclaw` + the `@openclaw/gateway-client` / `@openclaw/gateway-protocol` pins in the root `package.json`; the plugin repo tracks and publishes the same release). 2026.10.6 is a **security patch release** (eMudhra certificate-compromise remediation, Windows setup/Doctor/Gateway startup fixes, Discord command and iPhone Talk fixes) with no config-schema changes: the moltlazy `openclaw config validate` cctr contract passes unchanged, so no `packages/moltlazy` source change was needed. It does raise OpenClaw's Node engine floor from `>=24.16.0 <25 || >=26.1.0` to **`>=24.22.1 <25 || >=26.12.1`**, so `apps/fouria/Dockerfile` now installs Node **v24.22.1** (from v24.20.0) and its version gate compares major/minor/patch (24.22.0 is rejected). No component CalVer bump — the release train handles that.
- **Idle-sleep no longer re-archives an unchanged tenant, and deprovision snapshots reset the cadence.** `TenantSandbox.onActivityExpired()` used to enqueue a full `deprovision` snapshot on every idle expiry; on a 422 MB tenant the snapshot's own RPCs renewed the DO activity window, so `stopIfIdle()` refused, the next expiry snapshotted again, and the container never stopped (observed on `tenant-tarjetero-dev`: three 422 MB archives at 02:35/02:39/02:44 UTC plus three more at 05:39/05:43/05:49, ~3 GB/day, container `running` since Oct 21). The hook now reads the catalog first (`hasRecentSnapshot()` in `src/persistence.ts`, a pure read): a committed snapshot **inside the idle window** (`resolveSleepAfterMs()` — staging 15m, dev 2h, production 6h) already covers the unchanged filesystem, so it stops inline through `stopIfIdle()` without a new archive; no bucket or a read failure falls back to the durable backup-then-stop job (fail closed). Deprovision `finalize` now passes `stampScheduled: true` too, so a committed archive resets `lastScheduledBackup` and the next cron tick does not archive the same state again. Tests: `src/sandbox-do.test.ts` (recent snapshot stops inline, probe failure snapshots, no bucket skips the probe), `src/persistence.test.ts` (`hasRecentSnapshot`), `src/activity/scheduler.test.ts` (deprovision stamps), `src/build-sandbox-options.test.ts` (`resolveSleepAfterMs`); corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- **The dev Fouria Builder heartbeat is reconciled to its intended cadence.** OpenClaw 2026.10.5 retired `agents.entries.*.heartbeat`: heartbeats are ordinary automation jobs, and config reloads/restarts never rewrite an existing job. The migrated `heartbeat-fouria-builder` job therefore kept the **default 30-minute** cadence for 9 days on `tenant-tarjetero-dev` despite moltlazy's `heartbeat.every: 12h`, running full agent turns on the **main** session (~317k tokens each) until the OpenCode subscription hit 0% (`402 Insufficient account funds` → `agent-runner-failure`). `start-openclaw.sh` gains a dev-only `ensure_dev_heartbeat_cadence` (post-start) that reads `openclaw automations get <id> --json`, compares `schedule.everyMs` / `sessionTarget` / `payload.lightContext`, and only then edits the job to `--every 12h --session isolated --light-context` (split so a rejected `--light-context` cannot lose the cadence fix). Idempotent: a matching job is never touched, so the schedule anchor is not reset on every boot. Docs `packages/moltlazy/AGENTS.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- Backup catalog reads are **pure**; catalog healing has an explicit entry point (audit finding F3). `GET /api/admin/storage/backups` (`getBackupHistory`) used to run `reconcileOrphanedBackups` on every dashboard poll — listing `backups/`, reading `meta.json`, adopting orphans, deleting over-cap archives, and writing `backup-handle.json` from a `GET`. It now performs three parallel reads (`getPersistenceStore`, `getStableBackup`, `isRecoveryHeld`) and nothing else. The write path is `reconcileCatalogState` (adoption + retention prune inside the etag-CAS mutation, reporting the ids it changed) → `repairBackupCatalog` (sweep metadata-less uploads + in-place encryption) → refreshed history, exposed as `reconcileBackupCatalog`; the durable `repair` step now runs it **before** the `gate` step of every scheduled snapshot, so the catalog still self-heals after a killed invocation (issue #344). Restore/schedule keep their inline reconcile, `debug/reprovision` reconciles before deciding nothing is restorable (a crashed upload must not look unrecoverable), and operators get `POST /api/admin/storage/reconcile` (ADMIN/ROOT). The history panel polls `GET /api/admin/storage/jobs/:id` (`getBackupJobStatus`) while a snapshot is in flight instead of re-listing the catalog every 5s, falling back to the history diff when no job id is returned. Tests: `src/persistence.test.ts`, `src/persistence-encryption.test.ts`, `src/routes/api/storage.test.ts`, `src/activity/scheduler.test.ts`, `src/routes/debug.test.ts`, `src/client/components/__tests__/BackupHistoryPanel.test.tsx`; corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md`, `docs/plans/durable-agents-rules-audit.md`.
- A **config repair is deferred until a restore settles and the gateway is up on the restored state**, and the restored config becomes the stable/latest config version. A restore replaces the writable layer but launches nothing, so a `doctor --fix` applied around it was silently overwritten by the next re-apply of the recovery point (suspected on `tenant-tarjetero-dev` after a config fix). **(1)** Every successful restore (`restoreIfNeeded`, `restoreBackupById`) writes a durable `restore-config-pending.json` marker. **(2)** The boot job, the durable restore job, and the sync/async-fallback admin restore route finish through the shared `launchRestoredState` (`src/post-restore.ts`): `ensureGateway({ waitForReady: true })` on the restored state, then `captureRestoredConfigVersion` writes `configs/<ts>.json` + `stable/config.json` (the newest/stable version) and clears the marker. A gateway-launch failure does not re-run the destructive restore: restore jobs succeed with `detail: "gateway-launch-pending"`, the sync route returns `gatewayReady: false`, and the marker drives the next tick. **(3)** `doctor-repair` checks an active restore job, `getPersistenceBusyReason` (maintenance window / restore in flight), and the pending marker before running; when blocked it throws `MaintenanceJobDeferredError`, and `runMaintenanceJobAlarm` reschedules it every 60s **without consuming an attempt** (previously three deferrals would have failed the job before the restore finished). Tests: `src/activity/maintenance-jobs.test.ts`, `src/activity/scheduler.test.ts`, `src/persistence.test.ts`, `src/routes/api/storage.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md`.
- Diagnostics dashboard is **skip-not-park and non-blocking**. `DiagnosticDashboard` only dispatches its container reads (`/api/admin/config/raw`, `/api/admin/config`) while the gateway is actually running (`useGatewayRunning()` — `snapshot.readiness.ok`, ignoring the ROOT breaker bypass) and re-runs them when it comes back; the debug-routes flag (Worker-local) still loads. Every load is deferred after paint (`scheduleAfterPaint` → `requestIdleCallback`, `setTimeout(0)` fallback), so no widget waits on a container response: `ConfigSection` gained a `gatewayReady` prop and now keeps its tabs/editor visible with a non-blocking "Loading configuration…" / "Waiting for the gateway to start…" notice instead of swapping the whole panel for a spinner; devices render a waiting state; config loads never clobber text typed before the response arrived. Gateway-only actions (device list/approve, config save, channel probe) are guarded, and the Debug Tools that only make sense against a live gateway (`gateway-api`, `stop-gateway`) are skipped with an inline explanation — container diagnostics (logs/processes/version/sqlite) and recovery actions (restart/rescue/destroy) stay available so a down gateway can still be inspected and revived. **Repair stays available outside the editor:** `doctor --fix` is the recovery action for a down gateway, so `runRepair` no longer requires a running gateway, a dedicated **Repair Environment** bar (`data-testid="repair-environment-button"`, ADMIN/ROOT) sits above the widgets and is disabled only while a repair is in flight, and `ConfigSection`'s repair button stays enabled while `gatewayReady` is false. Tests: `src/client/pages/__tests__/DiagnosticDashboard.test.tsx`, `src/client/components/__tests__/ConfigSection.test.tsx`, `src/client/GatewayHealthContext.test.tsx`; corpus `test/business-rules/dashboard-api-contract.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `docs/TESTING.md`.
- Billable emission: **staging always emits**, `BILLABLE_EVENTS` opts a dev instance in. Both emit sites (`POST /api/otel/v1/traces` and the `CRON_BILLABLE_DRAIN` cron drain) previously gated on the strict billable cost axis (`resolveInstanceDimensions(env).billable` = production platform + production environment), so staging and dev instances could never exercise the billing pipeline. They now call `isBillableEventsEnabled(env)` (`src/dev-mode.ts`): billable **production always emits and cannot be opted out**, the **staging platform always emits** in every environment (mirroring production so billing is validated before release; the rows land in the staging Analytics Engine dataset), and a **dev environment** emits only with `BILLABLE_EVENTS=true`. Emission only: `getCostContext` keeps the real `platform`/`environment` dimensions (`staging`/`dev` remain non-billable _cost_ axes) and `shouldAutoSleep` is untouched, so staging keeps its 15m idle timeout and an opted-in dev instance its 2h one. The emitting path omits the guard's `billable: false` marker (the local-E2E skip, whose worker is a dev instance via `DEV_MODE`, is unaffected). Tests: `src/dev-mode.test.ts`, `src/routes/otel-traces.test.ts` (staging emits, dev opt-in emits), `src/index.test.ts` (cron drain: dev drops, staging and dev-with-override emit), `src/build-sandbox-options.test.ts` (sleep policy unchanged); corpus `test/business-rules/dev-expense.txt`; docs `docs/ENV-VARIABLES.md`, `docs/TESTING.md`, `apps/fouria/README.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `.dev.vars.example`.
- Request paths never run a destructive archive restore, and a container stops only on verified inactivity (investigated on `tenant-tarjetero-dev`, which appeared to "reboot" while in active use). **(1)** The container instance had **not** restarted (`started_at` 00:18, still `running`); what died was the **gateway process** (`[Gateway] Gateway process … has terminal state: exited`), and the follow-up boot ran a **350 MB** archive restore under request execution contexts. `sandbox.restoreBackup` was aborted at ~30.1 s (`backup.restore error … The operation was aborted (30177ms)`, plus `sandbox.exec … The operation was aborted (30001ms)` for health probes/`umount` and the workerd "promise resolved in a different request context" warning) because Cloudflare cancels a request's background continuations shortly after the response — so the restore never finished and every reload re-attempted it (6 boot jobs + 8 restore errors in one hour). Fix: `restoreIfNeeded(..., { restoreOwner: 'request' })` (the `ensureGateway` default) performs only the cheap non-destructive checks (the `restore-needed` latch + `containerHasState()`) and otherwise throws `RestoreDeferredToBootJobError`; `ensureGateway` requests the durable boot job and throws `GatewayBootPendingError` (proxy answers 503/loading). Durable callers (`ActivityScheduler` boot/wake jobs, the queue consumer) pass `restoreOwner: 'durable'`. The proxy's three inline `restoreIfNeeded` calls (`proxy.request`, `proxy.websocket-recovery`, `proxy.fetch-recovery`) were removed. **(2)** A container is now stopped only after a **stop-time** inactivity check: `TenantSandbox.stopIfIdle()` re-reads the DO's own idle window (renewed by every request) and the process list, and the deprovision job calls it after its multi-minute snapshot instead of `sandbox.stop()`; if the tenant became active meanwhile the job is `skipped` (`active:<reason>`). Refusals: `activity`, `transient-activity`, `probe-failed`, `billable-production`, `not-running`. Tests: `src/sandbox-do.test.ts` (`stopIfIdle` cases), `src/activity/scheduler.test.ts` (stop-time skip), `src/gateway/process.test.ts` (deferral + `restoreOwner`), `src/persistence.test.ts` (request deferral / state-present short-circuit), `src/index.test.ts` (proxy no longer restores); corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md`, `webs/fouria-docs/.../worker-development.md`, `operator-guide.md`.
- OpenClaw migrated to **2026.10.5** (`catalog.openclaw` + the `@openclaw/gateway-client` / `@openclaw/gateway-protocol` pins in the root `package.json`; the plugin repo tracks the same release). One breaking change hit moltlazy: the 2026.10.5 config schema **retired `agents.defaults.compaction.enabled`** (embedded auto-compaction is no longer toggleable) and **retired `compaction.truncateAfterCompaction`** ("byte-triggered compaction now opts in via `maxActiveTranscriptBytes` alone") — `openclaw config validate` rejects both keys with `Invalid input`, which broke the moltlazy cctr `config-save` contract. `AGENT_DEFAULTS.compaction` now emits only `maxActiveTranscriptBytes: '20mb'` + `keepRecentTokens: 20000`, preserving the sandbox OOM transcript bound. The runtime zod schema still accepts the retired keys, so only the `config validate` contract (and the published schema) caught it. Tests: `packages/moltlazy/tests/agents.test.ts` (asserts neither retired key is emitted); docs `packages/moltlazy/AGENTS.md`.
- Local development runs on the **Cloudflare Vite plugin** instead of `wrangler dev`. `bun run dev:worker` / `bun start` start the Worker with `vite dev` (`vite.config.ts`), the same plugin `bun run build` already used; `bun run dev` serves the admin SPA from `vite.ui.config.ts` (:8787) and proxies to the Worker (:8788). Local E2E fixtures (`test/e2e/fixture/server/start-local`, `start-local-pair`) launch `vite dev`, pass the old `--var` overrides through the plugin's `config` hook (`E2E_WORKER_NAME`, `PERSIST_STATE_PATH`, `VITE_CACHE_DIR`, `VITE_INSPECTOR_PORT`), install `test/e2e/.dev.vars` into the Worker config directory for the run (the plugin has no `--env-file`; `stop-local*` restores the developer's file), and fire cron ticks through `/cdn-cgi/local/scheduled` instead of `/__scheduled`. `--env` is gone from the deploy/dry-run scripts: `CLOUDFLARE_ENV=…` selects the environment at build time and `wrangler deploy` discovers the generated `dist/fouria/wrangler.json` via `.wrangler/deploy/config.json`. Docs: `README.md`, `docs/DEVELOPMENT.md`, `docs/TESTING.md`, `apps/fouria/README.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md`, `test/e2e*/README.md`.
- `/healthz` is now a real, fast liveness route. It previously fell through to the gateway-proxy catch-all (`app.all('*')`), which ran `restoreIfNeeded` + `ensureGateway` + a `waitForGatewayReady` port wait — on a cold/sleeping instance `GET /healthz` took ~108 s and was killed by the platform before responding (Workers Observability on `tenant-client-dev`: `GET /healthz` → 503 @ ~108 s avg, 348 requests in 2 h). The new `GET /healthz` does exactly one non-waking `isRuntimeActive()` read and returns `200 { ok, ts, status }` (`active` | `idle`) sub-second, never warming the container or probing the gateway; readiness stays on `/api/health`. It sits on the tenant hostname, so it stays behind the host Access app (not public). Tests: `src/routes/public.test.ts` (`/healthz` 200 + no `ensureGateway`/exec/listProcesses; `idle` when the runtime is off); docs `apps/fouria/README.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `docs/WORKER-HEALTH.md`.
- Browser health endpoints moved to Access-safe sibling paths (`/api/health-ws`, `/api/health-snapshot`, `/api/health-components`). On tenant hostnames every browser WebSocket and `fetch()` under `/api/health` or `/api/health/*` failed (`WebSocket connection to 'wss://…/api/health/ws?passive=1' failed`, plus CORS/redirect failures on `GET /api/health/components` and `GET /api/health`): the path-scoped `FourIA Health - <slug>` Access app shadows those paths and its AUD is distinct from the host app's (Cloudflare ignores a requested `aud` on create **and** update — verified against the live API — so the AUDs cannot be unified). A browser logged into the host app therefore gets a cross-origin 302 login redirect to `*.cloudflareaccess.com` on `/api/health*`, which WebSocket handshakes cannot follow and browser `fetch()` rejects on CORS. The sibling paths `/api/health-ws`, `/api/health-snapshot`, and `/api/health-components` sit outside the health app's path prefix, so the host-app session the browser already holds passes (verified against live `staging-client-dev.fouria.io`: `/api/health-*` resolves to the host app `kid`, while `/api/health` and `/api/health/*` resolve to the health app `kid`). The Worker mounts the handlers at both path sets (`healthHandler`, `healthComponentsHandler`, `healthWsHandler` in `src/routes/public.ts`); `api.ts` (`subscribeHealth`, `getHealth`, `getCloudflareComponents`) and `loading.html` use the `-` sibling paths, while `/api/health`, `/api/health/ws`, and `/api/health/components` remain for headless service-token clients (lerma). Tests: `src/routes/public.test.ts`, `src/assets/loading.test.ts`, UI stubs `test/ui/support/api.ts` + `test/ui/cloudflare-components.spec.ts`; docs `apps/fouria/README.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `docs/WORKER-HEALTH.md`, `docs/TESTING.md`; lerma `access_policy.ex` documents the constraint.
- Telegram ingress queue observability: notification-bell entry + home status widget. The per-tenant `TelegramIngress` DO now records every ingress event (bounded counters + last 20) — `queued` (a cold update was enqueued and woke the container), `forwarded` (hot path), `delivered`, and `failed` — via `telegram-ingress/status.ts`, wired from the webhook route (queued/forwarded) and the queue consumer (delivered/failed). `GET /api/admin/telegram-ingress/status` returns the snapshot and `GET /api/admin/telegram-ingress/ws` streams it (a frame per change, `TELEGRAM_INGRESS_WS_POLL_INTERVAL_MS`, rotated). The admin shell's `TelegramIngressProvider` opens the socket only while the feature is in use (an account is registered or activity exists) and raises a notification-bell entry ("Telegram message queued — waking your instance", or a delivery-failure alert); the new `TelegramIngressWidget` on the home page shows state + queued/delivered/failed + last event, hidden until in use. Tests: `src/telegram-ingress/status.test.ts`, `src/routes/api/telegram-ingress.test.ts`, `src/client/__tests__/telegram-ingress-context.test.tsx`, `src/client/components/__tests__/TelegramIngressWidget.test.tsx`, `src/client/pages/__tests__/HomePage.test.tsx`; corpus `test/business-rules/dashboard-api-contract.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- Homepage no longer crashes with `Cannot read properties of undefined (reading 'length')`. `GET /api/admin/software/report` ran `cat <report> 2>/dev/null || echo '{}'`, so a tenant whose `moltlazy software provision` had never run got a truthy `report: {}`; the homepage Software widget then read `software.packages.length` and threw on render (only for ADMIN — ROOT lands on diagnostics). The route now returns `report: null` unless the payload is a real report (an object with a `packages` array), and `HomePage` reads `software?.packages ?? []` defensively. Tests: `src/routes/api/software.test.ts` (missing/partial report → `null`), `src/client/pages/__tests__/HomePage.test.tsx` (truthy `{}` renders the empty state instead of crashing).
- Release CD **promotes the already-built artifacts instead of rebuilding**. `deploy-fouria.yml` used to `docker build` the container image from source and re-build the wrangler bundle at release time; it now promotes what CI already built: it resolves the release's develop-mainline build commit (`git rev-list -n1 --first-parent --invert-grep --grep='^chore: release' <tag> -- apps/fouria/ .github/workflows/test.yml`), retags the Cloudflare registry image `fouria:unstable-<sha>` → `fouria:<tag>` + `latest` (minting short-lived pull+push registry credentials via `POST /accounts/{id}/containers/registries/{host}/credentials`, mirroring Lerma's `ImageRegistry`), and copies the CI bundle `fouria-bundles:unstable-<sha>/fouria.mjs` → `<tag>/` + `latest/` in R2. `deploy-dashboard.yml` now tags the develop build with an immutable `lerma:sha-<sha>` in addition to `:latest-pre`, and its `promote` job pins to that per-commit image (resolving the release's lerma build commit the same way) instead of the floating `:latest-pre` (falling back only for the first release after the tag is introduced). Docs: `docs/DEVELOPMENT.md`, `docs/TESTING.md`, `apps/lerma/DEPLOY.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`. Both workflows validated with `actionlint`.
- Durable `doctor-repair`/SQLite jobs start immediately, and their status streams over a WebSocket instead of 2s polling. A `doctor-repair` job enqueued at 12:57:30 only began at 13:02:11 — exactly the previous boot job's 14-minute lease — because the `ActivityScheduler` owns one alarm and a **stale arm** (the alarm handler's trailing re-arm, or a warm-path `refresh`) could read "no pending job", then execute its `setAlarm(later)`/`deleteAlarm()` _after_ the just-enqueued job's `setAlarm(now)`, postponing it. `scheduleCombinedAlarm` now writes through `armWithGenerationGuard` (`activity/alarm.ts`): it bumps a stored generation, and a superseded arm abandons its stale write so the newest arm (fresh read) owns the alarm. The dashboard no longer polls `GET /api/admin/config/fix/jobs/:id` every 2s (one request + Worker log line per tick for the whole multi-minute repair): a new `GET /api/admin/config/fix/jobs/:id/ws` (and the `/debug/sqlite-integrity/jobs/:id/ws` variant) streams the job (`routes/api/maintenance-ws.ts`) — a frame only on status change, close `1000` when terminal, and a `rearm()` **self-heal** while active. The client's `subscribeMaintenanceJob` is WS-first with a slow HTTP fallback only while the socket is down, and `DiagnosticDashboard` renders a live `MaintenanceJobModal` (queued/running/attempt/elapsed, run-in-background) instead of a bare spinner, with a banner to reopen it after backgrounding. Tests: `src/activity/alarm.test.ts`, `src/activity/scheduler.test.ts`, `src/routes/api/maintenance-ws.test.ts`, `src/client/__tests__/maintenance-job-subscription.test.tsx`, `src/client/pages/__tests__/DiagnosticDashboard.test.tsx`; corpus `test/business-rules/dashboard-api-contract.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- Sandbox memory OOM mitigation (dev instance). A dev instance grew RSS past OpenClaw's 1.5 GiB `rss_threshold` (worker-thread V8 heaps + native arenas, **not** the main isolate) driven by cron-replayed heavy `agentTurn` jobs. **(1)** `packages/moltlazy` `AGENT_DEFAULTS.compaction` sets `enabled: true`, `maxActiveTranscriptBytes: '20mb'`, `keepRecentTokens: 20000` so a provider-side-managed session can no longer grow its persisted transcript unbounded (the `fouria-builder:main` ~407k-token session inflates the SQLite history and worker heaps). **(2)** The dev-only Fouria Builder entry sets `heartbeat.every: '12h'` plus `isolatedSession: true` / `lightContext: true` — OpenClaw's default 30m heartbeat is a full agent turn ~48×/day that also triggers memory-core dreaming reconciliation. **(3)** `start-openclaw.sh` slows memory-core's dreaming sweep to weekly (`frequency: "0 3 * * 1"`, dev-gated) — the ~6-min daily sweep is replayed on every wake (`cron: running missed jobs after restart`) and produced the 1.6 → 2.08 GiB RSS spike. **(4)** `ensure_opencode_credits_automation` dedupes via `openclaw cron list --all --json` in addition to the legacy `automations list` (which missed a prior boot's job, creating two duplicate `opencode-credits-check` cron jobs). Tests: `packages/moltlazy/tests/agents.test.ts`; docs `packages/moltlazy/AGENTS.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- Boot log streams gateway-launch progress (fixes "Launching the gateway — no progress for 6 min"). `start-openclaw.sh` now starts a `start_boot_log_stream` mirror in `launch_gateway_process` that tails the gateway's own `--verbose` stdout (`/tmp/gateway-stdout.log`) into the curated boot log (`/tmp/startup.log`) — so the loading page can follow SQLite migration → plugin load → port bind instead of going blank the moment `start`/`launch` `exec`s the gateway. `post-start`/`full` stop the mirror (`stop_boot_log_stream`) and truncate the buffer at readiness. The Worker's `/api/startup/ws` stream now reads the boot log while `processType === 'gateway'` (not just `'script'`), and `loading.html` shows the log panel during gateway startup — restyled as a vintage CRT terminal (title bar + traffic lights + phosphor text + scanlines) — instead of hiding it behind the logo. Tests: `src/assets/loading.test.ts`, `src/routes/public.test.ts`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- Loading page no longer gives up on its own. `loading.html` had two client-side budget specs — a 5-error retry cap (`maxConsecutiveErrors`, `Retrying... (n/5)`) and an 8-minute wall-clock deadline (`MAX_WAIT_MS` / `deadline` → `runDoctorAndShow()`). Since the cold boot moved to the durable `ActivityScheduler` boot job (which owns retries and terminal failure), the page's self-imposed give-up only interrupted a legitimately long startup: it now waits indefinitely (WebSocket push, with an unlimited 5s HTTP-poll fallback) and reports terminal failure solely from `boot.status === 'failed'` and permanent `degraded` container errors. Removed the now-dead `runDoctorAndShow`/`showStructuredError`/`showError` helpers. Tests: `src/assets/loading.test.ts`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- Restore decrypt no longer exhausts the Worker CPU budget, and interrupted restores are labelled honestly. A large recovery archive (`tenant-tarjetero-dev`) was killed with `Worker exceeded CPU time limit` mid-`Decrypting backup …` and retried in a loop. `encryptStream`/`decryptStream` (`src/crypto/backup-encryption.ts`) grew each 4 MiB chunk with `concat` on every ~64 KB R2 read, recopying the pending chunk ~64× over; a `ByteQueue` now copies each byte once (96 MiB decrypt: 1.5 s → 0.14 s at 64 KB reads, 5.6 s → 0.14 s at 16 KB). The R2 binding delivers the archive in **4 KiB reads** (64 MiB → 16,385 reads), which is why the per-read copy dominated; each restore now logs `[crypto] Decrypt read profile` (reads, bytes, max/avg read, chunks) so the pattern can be checked on a live tenant. The loading bar also showed the last durable phase (`Waiting for the backup lock 3%`) as if still live: a failed attempt is now labelled `Restore attempt failed during "<phase>" — retrying` (`recordRestoreFailure`), and an idle record gets `— no progress for N min` (`describeRestoreProgressIdle`); the fraction never moves backward. Tests: `src/crypto/backup-encryption.test.ts`, `src/activity/restore-progress.test.ts`, `src/activity/scheduler.test.ts`; docs `docs/WORKER-DEVELOPMENT.md`.
- Recovery progress bar synchronized with the durable object's events. During a restore the loading page showed a generic spinner; it now shows a **progress bar driven by the `ActivityScheduler` DO's own phase events**, not a client-side timer. `restoreIfNeeded`/`restoreBackupById` call their `onProgress(phase)` hook on every `persistence.phase`; the DO wires it to `recordRestoreProgress`, which maps the phase to a label + a monotonic fraction (`describeRestoreProgress` in `activity/restore-progress.ts`, cumulative phase weights) and exposes it via `getRestoreProgress()`. `/api/startup/ws` (and non-passive `/api/health`, plus `/api/admin/boot-state`) push `restoreProgress: { phase, label, fraction }`; `loading.html` renders the bar and hides the stepper while a restore is in flight, then switches back to the container's `startup-progress.json` stepper once the gateway launch begins (an indeterminate bar covers a request-scoped restore with no DO feed). Also declares `getBootJob`/`getRestoreProgress` on `ActivitySchedulerStub` (fixing the `public.ts` type error). Tests: `src/activity/restore-progress.test.ts`, `src/activity/scheduler.test.ts`, `src/routes/public.test.ts`, `src/routes/api/misc.test.ts`, `src/assets/loading.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md`.
- Restore legibility: name the backup + date being loaded (and mark the recovery point). A cold restore showed only a generic "Restoring data from backup…" spinner, so an operator could not tell _which_ snapshot was loading. **(1) Durable recovery state.** `restoreIfNeeded`/`restoreBackupById` now write `restore-target.json` (the archive being applied) just before `sandbox.restoreBackup` and clear it on success; `readRecoveryState(bucket)` returns `{ restoring, committed }` (`committed` = `stable-backup.json`). It is read from **R2**, not the per-isolate `restoreStatus`, because the durable boot/restore runs in the `ActivityScheduler` DO whose in-memory state the request isolate cannot see. **(2) Surfaced on the loading page.** `/api/health` (non-passive, not-yet-ready) and `/api/startup/ws` attach `recovery` (bounded by `RECOVERY_STATE_TIMEOUT_MS`, fail-soft to null; lerma's passive poll never pays the read); `loading.html` renders "Restoring backup <id> — <date>" (or "Recovery point …") and its `isRestoring` also honours `recovery.restoring`. **(3) Dashboard.** `BackupsPage` threads `stableBackupId` to `BackupHistoryPanel`, which marks the recovery point with a **recovery point** badge + a "A cold boot restores <id> — <date>" summary. Tests: `src/persistence.test.ts` (`readRecoveryState`, restore-target publish/clear), `src/assets/loading.test.ts`, `src/client/components/__tests__/BackupHistoryPanel.test.tsx`, `src/routes/public.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md`.
- Manual-restore hardening: recovery-point flag, writer quiesce, and no restore loop. Restoring an older archive from the Backups page could not be applied without either making it permanent or being silently undone. Three fixes in `src/persistence.ts` + the storage route/durable job. **(1) `setCurrent` flag.** `POST /api/admin/storage/restore/:id?setCurrent=0` (also `?async=1&setCurrent=0`) passes `{ setRecoveryPoint: false }` to `restoreBackupById` so the archive is applied **without** force-promoting it to `stable-backup.json` (the default still promotes; the durable `BackupJobRequest`/`BackupJobRecord` carry the flag so a retried job keeps the intent). The Backups page exposes it as a **Set as recovery point** checkbox (checked by default). **(2) Stop writers on the lease.** `restoreBackupById` now opens a cross-isolate maintenance window (`maintenance/replacement.json`, operation `restore`) and stops the gateway the moment it takes the shared restore lease (the route no longer pre-kills before the lock), closing the window in a `finally`; a live gateway/automation/channel therefore cannot keep writing while `sandbox.restoreBackup` replaces the writable layer. **(3) No remount loop.** `restoreIfNeeded` refuses to run while any maintenance window is active (and `ensureGateway` already refuses to launch), so a health/proxy refetch cannot restart the gateway and re-run the restore against the committed recovery point — the restore → stop → refetch → remount-the-latest loop. Tests: `src/persistence.test.ts` (set-recovery-point, maintenance window, deferred restore), `src/persistence-encryption.test.ts`, `src/routes/api/storage.test.ts`, `src/activity/scheduler.test.ts`, `src/activity/backup-jobs.test.ts`, `src/client/components/__tests__/BackupHistoryPanel.test.tsx`; corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md`.
- Durable cold-boot job + slim health probe (observed on `tenant-client-dev`: `start-openclaw.sh` never ran; `phase=restore, age=686365ms`; 141 `waitUntil() tasks did not complete`; `/api/health?passive=1` at error level ×98). **(1) Durable boot.** The cold boot (R2 restore → gateway launch) was driven from a request's `ctx.waitUntil()`, which Cloudflare cancels shortly after the response — so a restore that outlived it was killed mid-flight (`Decrypting backup …` ×14) and `start-openclaw.sh` was never reached while every poll parked on the stuck start promise. It is now a durable job on the `ActivityScheduler` DO (`src/activity/boot-job.ts`): `/api/warmup`, non-passive `/api/health`, and `/api/startup/ws` only _request_ it (deduped) and return; the DO alarm performs `restoreIfNeeded` → `ensureGateway` and retries on a bounded lease. Falls back to the request-scoped boot when `ACTIVITY` is absent. **(2)** `sandbox.restoreBackup` is bounded by `RESTORE_BACKUP_TIMEOUT_MS` (a hung control channel fails fast instead of holding the `restore` phase for 11 min). **(3)** The restore reports sub-phase progress so `isGatewayStartStuck` judges `restore` on `GATEWAY_RESTORE_NO_PROGRESS_MS` of **no progress** (`setGatewayStartProgress`) — a slow-but-advancing restore is not killed. **(4)** `withLease` releases the shared R2 lease when its request aborts (`PersistenceDiagnosticContext.signal`), so a cancelled operation can't leave `restore-needed` + a held `backup-lock` re-triggering the loop. Also: `/api/health` now issues **one** gateway `/health` exec shared by readiness + liveness (was a duplicate curl pair per poll) and the **passive** path skips the container + R2 component probe (lerma's frequent probe never reads `components`). Tests: `src/activity/boot-job.test.ts`, `src/gateway/process.test.ts`, `src/gateway/health-probes.test.ts`, `src/routes/public.test.ts`, `src/persistence.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- Slimmer `/api/health` (frequent lerma probe). Two per-poll costs removed. **(1)** Readiness and liveness each ran the same gateway `/health` `curl`, so every poll spawned a **pair** of identical `curl` processes (visible as paired entries in `/debug/processes`); `getHealthSnapshot` now fetches the process scan and the `/health` probe **once**, in parallel, and reuses both across the probes (new `HealthProbeOptions.health`). **(2)** The component probe (an `isRuntimeActive` read + an R2 HEAD of `backup-handle.json`) was built on **every** `/api/health`, including lerma's frequent `?passive=1` poll — which never reads `components`; the passive path now skips `componentsForHealth` entirely (the non-passive path and the dedicated `GET /api/health/components` are unchanged). Tests: `src/gateway/health-probes.test.ts` (single shared exec), `src/routes/public.test.ts` (passive omits components); corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`.
- Restore config guard + recovery-point promotion (observed on `tenant-tarjetero-dev`). After the `v2026.10.13` container rollout, an explicit restore of an older known-good backup did not stick: the redeploy changed the container runtime mid-restore, and a concurrent scheduled snapshot captured the partially-restored state and was auto-promoted to the stable recovery point, so the next cold boot restored the wrong archive and launched the gateway with no config. Three fixes: **(1)** `start-openclaw.sh` only honours `OPENCLAW_RESTORED=true` when `$CONFIG_FILE` exists — otherwise it runs the setup body — so a "restored" boot with no materialized config (interrupted restore, or `restoreIfNeeded`'s "container already holds tenant state" short-circuit) can no longer launch OpenClaw configless. **(2)** `restoreBackupById` now **force-promotes** the restored archive to `stable-backup.json` (`promoteStableBackup(bucket, entry, { force: true })`) so an explicit restore becomes the recovery point even when it predates the current one; the default pointer still only advances forward in time. **(3)** the pre-backup config cleanup used `.${CONFIG_DIR}` with `CONFIG_DIR='.openclaw'`, building `/home/openclaw/..openclaw/…` (a never-existent path, so the cleanup silently no-oped) — it now uses `${CONFIG_DIR}`. Tests: `src/persistence.test.ts` (force-promote; cleanup path); corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `apps/fouria/docs/OPERATOR-GUIDE.md`.
- Cold-start latency + boot legibility (a full non-restored boot on `staging-client-dev` took ~14 min). Two causes, both fixed, plus a loading-page alert. **(1) Non-owner killed the booting gateway.** A non-owner isolate that observed the `start-openclaw.sh start` wrapper waited a _fresh_ `STARTUP_TIMEOUT_MS` (300s) from its own observation, but the wrapper's real boot is `setup (~290s) + gateway bootstrap (~45s) ≈ 335s` — so the waiter timed out ~35s early and `kill()`ed the process just as it had `exec`ed into the gateway, releasing the startup lock and making the waiting second wrapper re-run the whole setup. Confirmed via Workers Observability: `Startup script present but gateway never became reachable … killing and restarting` = 1 while the warm-probe start/catch = 0/0 (ruling out the 30s warm probe). Fixed with `STARTUP_WRAPPER_STUCK_BUDGET_MS` (`STARTUP_TIMEOUT_MS + GATEWAY_START_STUCK_GRACE_MS`): the non-owner's stuck check and readiness wait are measured from the wrapper's `startedAt` (mirroring the owner's watchdog) and a `/health` re-probe runs immediately before any kill. Tests: `src/gateway/process.test.ts`, `src/gateway/process-extra.test.ts`. **(2) Plugin detection read the hanging registry.** `plugin_is_installed` ran `openclaw plugins list --json`, which hangs for its full `CLI_TIMEOUT` and reads as "not installed" on a hung/partial list, so the pre-installed guard fell through and triggered a needless network install (ai-gateway ~57s + otel ~74s). Replaced with `plugin_extension_present` (the `extensions/<id>` dir the gateway actually loads). **(3) Bootup-UI alert.** The loading page now shows a non-fatal slow-step notice (step name + elapsed seconds) once a step overruns `SLOW_STEP_MS` (120s), so a slow-but-progressing boot is distinguishable from a stuck one without opening logs. Tests: `src/assets/loading.test.ts`. Corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `docs/plans/cold-start-latency.md`.
- The tenant **Usage & Billing** widget no longer 403s on Analytics Engine. `GET /api/admin/costs/v2/summary` resolved its AE SQL token as `ANALYTICS_ENGINE_API_TOKEN ?? CLOUDFLARE_API_TOKEN`, but Lerma never provisioned the dedicated token, so the query fell back to `CLOUDFLARE_API_TOKEN` — the AI Gateway credential alias, which lacks Account Analytics:Read — and logged `[cost-summary] Analytics Engine query failed with HTTP 403` on every load (observed on `tenant-client-dev`), zeroing the widget. The token is now a Lerma **platform secret** (`tenant_provisioner.ex` `@platform_secrets`), read from Lerma's environment and deployed per-tenant as a `secret_text` binding; the deploy path wires it through `mix lerma.deploy`, Ansible `app-config`, `docker-compose.prod.yml`/`.pre.yml`, and `deploy-dashboard.yml`, and the secrets manifest. Optional: when unset the worker still degrades to an all-zero summary. Tests: `tenant_provisioner_pipeline_test.exs`, `tenant_provisioner_integration_test.exs`; docs `docs/ENV-VARIABLES.md`, `apps/lerma/README.md`, `apps/fouria/docs/cost-observability.md`, `SECRETS.example.yaml`, `docs/schemas/secrets-manifest.schema.json`.
- Zero-data-loss: a restored container skips the setup body. The R2 restore runs in the Worker (`restoreIfNeeded` → `sandbox.restoreBackup()`) _before_ `start-openclaw.sh`; when that restore actually restored a backup, `ensureGateway` now passes `OPENCLAW_RESTORED=true` in the launch env (gated on `getRestoreStatus().hadBackup && restored`, so a brand-new tenant with no backup still onboards), and the script skips the whole setup body — onboard → plugin installs → `moltlazy patch` → config validate — emitting those phases as skipped and launching the gateway directly from the restored filesystem. The restored state is authoritative: the cold-start budget is spent launching it, not re-mutating it (which could diverge from or clobber the snapshot). Each boot rewrites a durable receipt at `/tmp/openclaw-restored-setup-skipped` (written on the skip path, removed on the setup path). Tests: `src/gateway/process.test.ts`; E2E `test/e2e-backup/zzz_backup_restore_skip_setup.txt` (opt-in `E2E_BACKUP_RESTORE=1`; sync → destroy → restore, asserts the receipt + unchanged restored setup state); corpus `test/business-rules/operational-modes.txt`; docs `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `test/e2e-backup/README.md`, `test/e2e/BACKUP-E2E-DESIGN.md`.
- The dashboard **Usage & Billing** widget now reflects emitted billable actions. `GET /api/admin/costs/v2/summary` was a stub returning `totalCostUsd: 0`; it now aggregates the tenant's `billable-action` Analytics Engine rows via the AE SQL API (`src/cost-summary.ts` → `getCostSummary()`), returning `totalCostUsd`, `totalCredits`, `actionCount`, `byTriggerType`, and `byCostCategory`. Config mirrors Lerma: `ANALYTICS_DATASET` (default `fourai_cost_metrics`), `ANALYTICS_ENGINE_BASE_URL`, and `ANALYTICS_ENGINE_API_TOKEN` (falls back to `CLOUDFLARE_API_TOKEN`; needs Account Analytics:Read). A missing credential/transport error/non-200 degrades to an all-zero summary, so the widget never breaks. The widget surfaces the billable action/credit rollup when data is present. Tests: `src/cost-summary.test.ts`, `src/routes/api/costs.test.ts`, `src/client/pages/__tests__/HomePage.test.tsx`, `src/client/pages/__tests__/AccountPage.test.tsx`, `test/ui/billing-widget.spec.ts`; corpus `test/business-rules/dashboard-api-contract.txt`; docs `apps/fouria/docs/cost-observability.md`, `docs/ENV-VARIABLES.md`, `docs/TESTING.md`.
- The version surfaces now report the Worker's own build identity. `GET /api/admin/version` and `GET /debug/version` return `fouria_version` (component CalVer from `apps/fouria/package.json`) and `fouria_commit` (short git SHA) alongside the container's OpenClaw/Node versions; the DiagnosticDashboard **Version Info** tool surfaces them. Both values are injected at build time by `vite.config.ts` (`__FOURIA_VERSION__` / `__FOURIA_COMMIT__`, consumed via `src/build-info.ts`; `GITHUB_SHA` wins in CI, `git rev-parse --short HEAD` locally). Tests: `src/routes/debug.test.ts`, `src/routes/api/misc.test.ts`; docs `docs/WORKER-DEVELOPMENT.md`, `docs/OPERATOR-GUIDE.md`.
- Expected CLI non-zero exits are no longer logged as failures. `waitForProcess` (like `runShell`) now takes `reportNonZeroExit`; the knowledge `wiki status` probe sets it `false` (operation `knowledge.wiki-status`) because `openclaw wiki status` exits non-zero for an uninitialized vault — which `/api/admin/knowledge/tree` handles as a normal `initialized: false` result — yet every load emitted `Error [SandboxProcessError]: sandbox.waitForProcess failed (non-zero-exit, exit=1)`. Timeouts still log. Tests: `src/gateway/utils.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `docs/WORKER-DEVELOPMENT.md`.
- The prelaunch gateway port probe no longer logs a false failure. `isGatewayPortOpen` ran `curl -sf …/health` through `runShell`, whose `logCommandFailure` emitted a structured `sandbox.process.failed` **error** for the expected closed-port exit (`curl` 7 = connection refused). Every cold start therefore surfaced a scary `Error [SandboxProcessError]: shell failed (non-zero-exit, exit=7)` for the normal "gateway down, start it" outcome. It now passes `reportNonZeroExit: false` with `operation: 'gateway.port-probe'` (matching the existing `gateway.health-probe`), so a closed port is quiet while a probe **timeout** (a hung gateway) still logs. Test: `src/gateway/process.test.ts` (`does not log a structured failure for the expected closed-port exit`); corpus `test/business-rules/operational-modes.txt`; docs `docs/WORKER-DEVELOPMENT.md`.
- OpenClaw is installed globally with **bun**, not npm. `bun install --global --trust openclaw` runs with `BUN_INSTALL=/usr/local`: bun's default global prefix is `$HOME/.bun/bin`, which is not on `PATH` in this image, so `BUN_INSTALL=/usr/local` puts the binary at `/usr/local/bin/openclaw` (on `PATH`) and the package under `/usr/local/install/global/node_modules/openclaw`. `bun pm cache rm` resolves the current project and hard-fails without a `package.json` — the base image's `WORKDIR /container-server` has none — so it runs from `/tmp` (which holds the copied repo `package.json`). The dead-weight trim now targets bun's hoisted global layout (`<prefix>/install/global/node_modules`), and `start-openclaw.sh` resolves the package by following the installed binary (`command -v openclaw` + `readlink -f`) instead of `npm root -g`, so the runtime-version log stays accurate.
- `post-start` no longer blocks readiness. After the port wait succeeds the gateway is already serving `/health` 200, but `ensureGatewayImpl` still `await`ed `start-openclaw.sh post-start` (dev automation + boot-log reset, bounded by `GATEWAY_POST_START_TIMEOUT_MS` = 60s). On dev instances that phase runs `openclaw automations list/create` (30–60s), so every request logged `Gateway start already in progress … (phase=post-start)` and waited on the shared start promise even though the gateway was up. It is now dispatched **detached** (`void runGatewayPostStart(...)`, not awaited) right after `markGatewayReady`; `startShell` dispatches the `sandbox.exec` synchronously before its first `await`, so the work still runs server-side if the isolate is evicted, and `runGatewayPostStart` swallows its own errors. The `post-start` phase is no longer set on the start tracker (the start is complete). Test: `src/gateway/process.test.ts` (`does not extend the shared start promise with the detached post-start phase`); docs `docs/WORKER-DEVELOPMENT.md`.
- ClawHub plugin pre-install is best-effort so the image still builds. The unpinned `clawhub:@openclaw/cloudflare-ai-gateway-provider` resolved to `2026.9.9`, which requires a newer plugin API than the pinned `2026.9.8` runtime, so `openclaw plugins install` failed and aborted the whole Docker build (`Plugin … requires plugin API >=2026.9.9, but this OpenClaw runtime exposes 2026.9.8`). The Dockerfile now wraps each pre-install in a warn-and-continue guard, and `start-openclaw.sh`'s single enablement `config patch` only enables a plugin whose extension directory is present (`CF_AI_GATEWAY_PLUGIN_PRESENT` / `UNIFIED_BILLING_PLUGIN_PRESENT`), so a failed install can neither fail the build nor invalidate the patch. Root-cause follow-up: bump `catalog.openclaw` to the plugin-compatible release (openclaw-migration skill) or pin the specs; until then a build on the pinned runtime ships without the Cloudflare provider plugins. Corpus `test/business-rules/operational-modes.txt`; docs `docs/WORKER-DEVELOPMENT.md`, `docs/OPENCLAW-PLUGINS.md`.
- Gateway start watchdog + process-count hygiene. A cold start is one shared `ensureGateway` promise (restore → supervised `start-openclaw.sh start` → port wait); a wrapper that never `exec`s the gateway, or a control-channel drop, left it "in progress" for ~20 min while every request logged `Gateway start already in progress … waiting...` and waited on it (observed on `tenant-client-dev`: 64 processes, port closed, curl health probes exiting 7 every ~5s, `[PROXY] Failed to start gateway` stack-only). The start is now tracked per sandbox key with a phase (`probe` → `restore` → `launch` → `port-wait`; `post-start` runs detached after readiness, not tracked) and a per-phase budget; `isGatewayStartStuck` trips when a phase overruns its budget + `GATEWAY_START_STUCK_GRACE_MS`. Before resetting `port-wait`, `recoverStuckGatewayStart` rechecks `/health`: a late-positive response emits `gateway.start.late-ready` and is adopted without killing, while a failed/unknown recheck emits `gateway.start.health-recheck-failed` and defers reset. Confirmed stuck starts emit `gateway.start.stuck`, kill the process (`killGateway`), clear startup/readiness caches, and reprovision after `GATEWAY_START_STUCK_MAX_ATTEMPTS` resets (`recoverStaleSandbox`). `findExistingGatewayProcess` also uses the pure `classifyGatewayStart` to kill a `start-openclaw.sh start` wrapper older than `STARTUP_TIMEOUT_MS` with the port closed, instead of waiting out another readiness budget. Because `listProcesses()` includes `exited`/`error` entries the Worker never GCs (the SDK exposes no cleanup), the raw count is append-only and must not gate health: the log now reads `listProcesses returned <total> processes (<running> running)` and every decision considers only `running` entries. Emits `gateway.start.begin`/`phase`/`stuck`/`late-ready`/`health-recheck-failed`. Tests: `src/gateway/classify.test.ts`, `src/gateway/process.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `docs/WORKER-DEVELOPMENT.md`, `docs/OPERATOR-GUIDE.md`.
- Legacy bridge excludes the image-provided shared OpenClaw package. `extensions/<id>/node_modules/openclaw` is a symlink to the image's `/usr/local/lib/node_modules/openclaw`; archiving it is pointless and (when an old dereferencing stage expanded it into a full package copy) made the restore materialization a multi-minute copy that blocked the gateway launch, left `restore-needed` set, and re-ran the restore on every boot (observed on `staging-client-dev`: `cp -a … /root/openclaw/.openclaw/` running 8m45s, 65 Worker probe processes, no `start-openclaw.sh`/gateway). Both `stageLegacyRootsForBackup` and `materializeLegacyRootsAfterRestore` now use a symlink-preserving `tar` pipe with `--exclude='*/node_modules/openclaw'` (a copy-then-`rm -rf` is equally slow — deleting a squashfs-lower tree requires copy-up), so old and new archives restore fast. `restoreIfNeeded` also deletes the `restore-needed` marker immediately after a successful materialization (before the analytics bookkeeping) so an isolate eviction can no longer leave a stale marker. Tests: `src/persistence.test.ts`; corpus `test/business-rules/operational-modes.txt`.
- Legacy restore bridge preserves symlinks (backup/restore fix). The legacy state bridge staged `/root/openclaw/.openclaw` with `cp -aL`, which dereferenced the plugin `extensions/*/node_modules/openclaw` symlinks (→ `/usr/local/lib/node_modules/openclaw`) into full OpenClaw package copies. That bloated every archive (the package three times over), timed out the pre-snapshot stage (600s) — surfacing to the dashboard as a backup-create `Internal Server Error` — and made restore fail because `cp` cannot overwrite the image's symlink with a directory (`cp: cannot overwrite non-directory … with directory`), leaving the tenant looping on a failed restore (`staging-client-dev`). The stage and the restore materialization now use `cp -a` (preserve symlinks); the restore additionally removes any destination symlink the stage replaces with a real directory first, so backups created by the old dereferencing stage still restore. Tests: `src/persistence.test.ts`; corpus `test/business-rules/operational-modes.txt`. Also: the admin API client (`src/client/api.ts`) now reads the response body once and surfaces a plain-text platform error (e.g. `Internal Server Error`) instead of throwing `Unexpected token 'I', "Internal S"... is not valid JSON`; tests `src/client/api.test.ts`.
- Recurring-error hardening: credential redaction, bounded-RPC cleanup, live-gateway logs, and idempotent WS teardown. Investigation of the recurring `ReadableStream received over RPC disconnected prematurely` / `[WS] Client error` telemetry found four concrete causes. (1) **Credential logging:** the catch-all proxy logged the full forwarded header map (`Object.fromEntries(httpHeaders.entries())`), writing CF Access JWT/assertions, cookies, and the tenant metadata blob to Workers Logs; it now logs only an allowlisted subset (`safeHeaderLog`, `src/utils/logging.ts`). (2) **Late container RPCs:** a bounded `containerFetch`/`wsConnect` whose timeout won could still settle later with a live body/socket that was never drained/closed; `withTimeoutAndCleanup` (`src/utils/timeout.ts`) plus `containerFetchWithTimeout`/`wsConnectWithTimeout` (`src/utils/container-rpc.ts`) cancel the late body / close the late socket, applied to the public proxy, diagnostics, and the `gateway-api` probes. (3) **Live gateway logs:** a running gateway's SDK log stream never settles, so reading it opened an RPC stream that was later disconnected; a running `GatewayProcess.getLogs()` now tails `/tmp/gateway-stdout.log`, and `/debug/processes?logs=true` / `/api/admin/logs?id=` refuse a live stream for a still-running process. (4) **WS teardown:** both peer close handlers recorded the same session metric twice, and a client tab disconnect was logged at `error`; a single `settleSession()` now records the metric once and client disconnects log at `warn` with the `sessionId`. Tests: `src/utils/container-rpc.test.ts`, `src/utils/timeout.test.ts`, `src/logging.test.ts`, `src/index.test.ts`, `src/gateway/process.test.ts`, `src/routes/api/misc.test.ts`, `src/routes/debug.test.ts`; local observability canary `test/e2e/ad_observability_canary.txt` + `test/e2e/fixture/observability-query`; corpus `test/business-rules/operational-modes.txt`; docs `docs/TESTING.md`.
- Two local-E2E self-skip fixes. `provision_and_dispatch`'s first scenario (`%require`) echoed a `SKIP:` line and `exit 1` in local mode, so it _failed_ instead of skipping — and because it is `%require`, that failure skipped every later scenario in the file. It now emits the same `PASS: … (skipped - local E2E mode)` line and `exit 0` the other local skips in the file use. `p_otel_billing_queue` asserted `emitted >= 1`, but a dev-environment/staging-platform instance is non-billable by design (`resolveInstanceDimensions`, `src/routes/otel.ts`): the receiver accepts + ignores approved spans (200) and returns `billable: false` without emitting. The local worker (`DEV_MODE=true`, no routable `WORKER_URL`) is always non-billable, so the test now skips on `billable == false` (like its existing missing-secret/`401` skip); the emit path stays asserted by the cloud profile and `src/routes/otel-traces.test.ts`. Corpus: `test/business-rules/operational-modes.txt`, `test/business-rules/dev-expense.txt`; docs `docs/TESTING.md`.
- Local E2E readiness gate accepts a reachable-but-starting Worker. `fixture/server/start-local` waited for `/api/health` with `curl -sf -m 5` (60×2s), but a cold boot legitimately answers **503** while the container/gateway starts (the non-waking-health semantics) and the non-passive probe can block on container provisioning — so the gate misread a live-but-starting Worker as dead. Worse, the non-passive probe on every 2s poll re-triggered `restoreIfNeeded`/`ensureGateway`, launching overlapping R2 restores that abort each other with `OperationInterruptedError: ... runtime changed`. The gate now probes the fast, **non-waking** `GET /api/health?passive=1` and accepts **any** HTTP status (`200`/`503`/…) via the shared `e2e_worker_reachable` helper (`test/lib/e2e-helpers.sh`), failing only on a connection failure (status `000`); the window is overridable with `E2E_STARTUP_TIMEOUT_SECS` (default 120s) and the "local dev server died" check still fails fast. The real gateway wait (`startup.ok` + `readiness.ok`) stays in the `_setup.txt` readiness step. Tests: `src/e2e-helpers.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `docs/TESTING.md`.
- Restore failures are logged as structured `persistence.restore.failed` events with `backupId`, `errorName`, and `errorMessage`; manual route failures also include the Cloudflare Ray ID as `storage.restore.failed`. This keeps the backup identity searchable and correlates async failures to the originating request. Tests: `src/persistence.test.ts`, `src/routes/api/storage.test.ts`.
- Backup retention is enforced on catalog reads, and restores survive a failed pre-restore `umount`. Live `fouria-backup-tarjetero-dev` held 7 archives under `maxVersions: 3`: every snapshot was killed after `sandbox.createBackup()` (260 MB) but before its commit, so a later read adopted all 7 as orphans, and every scheduled snapshot after that failed at the legacy stage before its own prune — the cap was only ever applied on the happy snapshot path. `reconcileOrphanedBackups` now runs `cleanupOldBackups` after merging, so the invariant holds on every read (`getBackupHistory` included) and no write/delete happens when the catalog is already within the cap. Restores failed in a loop on `[persistence] Restore of … failed: … runShell … restoreIfNeeded`: the best-effort `umount` (`|| true` guards the shell, not the exec RPC) threw a cold-start/platform interruption and aborted the restore before `restoreBackup`, keeping `restore-needed` and failing the startup barrier. The umount is now caught + logged in `restoreIfNeeded` and `restoreBackupById`; an empty catalog read no longer deletes `restore-needed` (a transient reconcile `list` failure silently dropped an operator restore) — the marker clears only after a successful restore. Tests: `src/persistence.test.ts` (over-cap prune, adopted-orphan prune, no-delete within cap, marker retention, umount-failure restore); corpus `test/business-rules/operational-modes.txt`; docs `docs/WORKER-DEVELOPMENT.md`, `docs/OPERATOR-GUIDE.md`.
- Async admin backup/restore + non-blocking Backups page. A manual snapshot/restore can wait up to the 120s R2 lease and then transfer a large archive; holding the admin HTTP request open surfaced as a bare `500` after ~132s (restore) with the error dropped. `POST /api/admin/storage/sync?async=1` and `POST /api/admin/storage/restore/:id?async=1` now acknowledge `202 { accepted: true }` and run in `executionCtx.waitUntil` (default stays synchronous for the E2E checkpoint). `BackupsPage` paints every section independently and reads the fast `GET /api/admin/config/raw` instead of the slow structured `/api/admin/config` (which awaits `ensureGateway` + up to two 30s shell waits) — so no single request blocks the page. `BackupHistoryPanel` fires the async endpoints, pushes a "started" bell notification, then polls `GET /storage/backups` (snapshot) / `/api/health` (restore) and pushes success/failure via `NotificationProvider` (survives navigation). Restore failures are reported as structured `storage.restore.failed` events. Tests: `src/routes/api/storage.test.ts`, `src/client/components/__tests__/BackupHistoryPanel.test.tsx`, `src/client/pages/__tests__/BackupsPage.test.tsx`; corpus `test/business-rules/operational-modes.txt`; docs `docs/WORKER-DEVELOPMENT.md`.
- Startup-stream and dashboard polling back-off. `/api/startup/ws` now drops the 2-exec startup-progress read and backs off to `STARTUP_WS_READY_POLL_INTERVAL_MS` (30s) once the gateway is ready — the 5s cadence plus a per-push `listProcesses`/exec probe burst is a large share of the Sandbox DO control-channel churn (`Establishing RPC connection`) during a boot. `useGatewayHealth` now stops its HTTP fallback polling when a `/api/health/ws` snapshot arrives (it previously kept double-polling after the socket recovered). `transformErrorMessage` maps the gateway's `1008 device token mismatch` close to a "re-open `/_admin/` to re-approve this browser" message instead of a reconnect loop. Tests: `src/client/hooks/useGatewayHealth.test.tsx`; corpus `test/business-rules/operational-modes.txt`.
- Backup local-bucket mode is decoupled from `DEV_MODE`. `isLocalBucketDevMode()` keyed `createBackup({ localBucket })` on `process.env.DEV_MODE`, and with `nodejs_compat_populate_process_env` the cloud E2E's `DEV_MODE=true` var (set only to bypass Access on a bare `workers.dev` host) made every E2E snapshot land in the Sandbox **local binding bucket** — invisible to the Worker's `BACKUP_BUCKET` HEAD, so restore logged `Backup <id> missing from R2` and booted without tenant state. It is now `isLocalBucketBackupMode(env)`: an explicit `BACKUP_LOCAL_BUCKET=true` opt-in that falls back to `DEV_MODE` only when unset; the E2E deploy pins `BACKUP_LOCAL_BUCKET=false`. `resolveRestoreEntry` threads `env` through. Tests: `src/persistence.test.ts` (explicit override wins); corpus `test/business-rules/operational-modes.txt`; docs `docs/ENV-VARIABLES.md`, `docs/OPERATOR-GUIDE.md`, `docs/WORKER-DEVELOPMENT.md`, `.dev.vars.example`.
- A dropped port probe no longer spawns a duplicate gateway. When the DO↔container control channel flapped (`Network connection lost.` on every `exec`), `ensureGatewayImpl`'s prelaunch `isGatewayPortOpenResilient` throw was treated as "gateway absent", so it re-ran `start-openclaw.sh start` on each request even though `listProcesses` matched the gateway `running` — piling lock contention and RPC load onto the unhealthy channel. A transport-classified probe failure is now "unknown": the Worker logs `Port probe transport drop …` and returns without spawning, retrying on the next request. Test: `src/gateway/process.test.ts` (`ensureGateway transport-drop resilience`); corpus `test/business-rules/operational-modes.txt`.
- E2E `ac_gateway_startup_concurrency` accepts the 503 cold-boot health response. The concurrent non-passive `/api/health` fan-out used `curl -f`, which fails on any non-2xx, but `/api/health` answers **503 while `starting`** (JSON snapshot present) since the non-waking-health change — so a healthy cold boot always reported `probes_ok=0`. The probes now accept `200`/`503` with a shaped `{startup, readiness}` body; the ready-wait and single-owner assertions are unchanged. Docs/corpus updated.
- Sandbox→R2 bridges resolve the tenant from the per-tenant Worker env, not the fixed sandbox id. The `leads.r2.internal` and `docs.r2.internal` bridges (`src/sandbox-leads-bridge.ts`, `src/sandbox-docs-bridge.ts`) scoped every path to the caller's `<slug>/` using `ctx.containerId`, but production provisions the sandbox under the fixed id `openclaw` (`getSandbox(..., 'openclaw')` in `src/index.ts`, `src/tasks/index.ts`, `src/activity/scheduler.ts`), so listings resolved to `openclaw/` and always returned empty. Added `resolveTenantSlug` / `tenantSlugFromEnv` in `src/sandbox-tenant.ts`: an explicit `customer-<slug>` container id still wins (legacy multi-tenant), otherwise the slug is derived from the per-tenant Worker env — `CUSTOMER_SLUG`, `WORKER_NAME` via `deriveCustomerSlug`, or the always-provisioned `WORKER_URL` hostname (strips `tenant-` / `staging-` prefixes and the `-dev` suffix, so `tenant-tarjetero-dev` / `https://tarjetero-dev.fouria.io` → `tarjetero`). Tests: `src/sandbox-tenant.test.ts`, `src/sandbox-leads-bridge.test.ts`, `src/sandbox-docs-bridge.test.ts`; corpus `test/business-rules/tenant-isolation.txt`; docs `docs/WORKER-DEVELOPMENT.md`.
- Crash-consistent backup cataloging (issue #344). The scheduled `backup` control action runs the full snapshot orchestration inside one tenant request and could exceed the Worker CPU budget after `sandbox.createBackup()` had uploaded `backups/<id>/…` but before the handle was written — leaving an orphaned snapshot, a missing `backup-handle.json` (and empty `configs/`), a live `backup-lock`, and a dispatch `[cron] Failed to trigger backup`. `createSnapshotUnlocked` now commits the catalog entry **immediately after the archive is created** (after encryption, before the config-version capture and retention prune), and `reconcileOrphanedBackups` rebuilds entries from `backups/*/meta.json` on read (`getBackupHistory`, `restoreIfNeeded`, `scheduleBackupIfNeeded`) so pre-existing orphans self-heal and stay restorable. The Dispatch Worker gives the `backup` action a dedicated CPU budget (`BACKUP_CONTROL_CPU_MS`) and logs the underlying error on a thrown control fetch. Tests: `src/persistence.test.ts` (commit ordering, config-save failure, reconciliation), `workers/fouria-dispatch/test/unit/control.test.ts`, `cron-fanout.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `docs/OPERATOR-GUIDE.md`, `docs/WORKER-DEVELOPMENT.md`.
- Transport-resilient backup stage (issue #344 follow-up). The pre-snapshot legacy-stage shell now waits via `runShellResilient` (→ `waitForExitResilient`): a transient Sandbox DO↔container transport drop mid-stage reattaches to the process by id and resumes within the stage deadline, instead of ending the `/control` backup request (observed on `staging-client-dev`: the stage ran ~5 min then the request ended with a `Network connection lost.` drop and no `Creating backup...`). The Dispatch `backup` CPU budget (`BACKUP_CONTROL_CPU_MS`) is raised 300s → 600s to match the 10-minute stage bound. Tests: `src/persistence.test.ts` (transport-drop retry); corpus `test/business-rules/operational-modes.txt`; docs `docs/OPERATOR-GUIDE.md`, `docs/WORKER-DEVELOPMENT.md`.
- Legacy-stage backup robustness + diagnosability (issue #344 follow-up). The scheduled `backup` stage shell (`stageLegacyRootsForBackup` → `runLegacyStageShell`) copies `/root/openclaw/.openclaw` + `/root/clawd` with `cp -aL` and runs a `node:sqlite` online backup; on a large tenant it exceeded the old 300 s bound and killed the backup (`tenant-tarjetero`, observed at exactly 300 s). `LEGACY_STAGE_TIMEOUT_MS` is now env-overridable (default `600000` / 10 min), the idempotent stage is retried **once** on a transient timeout/transport failure, and stage failures log `exit`/`timedOut`/stderr plus a structured `sandbox.process.failed` Worker event with `processId` and a stable operation label. The process id joins Worker Logs to the Containers dataset without exposing the raw command/output. Tests: `src/persistence.test.ts` (retry, no-retry, timeout override, error logging); corpus `test/business-rules/operational-modes.txt`; docs `docs/OPERATOR-GUIDE.md`, `docs/WORKER-DEVELOPMENT.md`, `docs/ENV-VARIABLES.md`.
- Sleeping tenants + non-waking health. `/api/health` now answers **503 unless the gateway is ready**, and a **passive** probe (`?passive=1`) short-circuits on the non-waking `isRuntimeActive()` read and reports `status: "sleeping"` (HTTP 200, healthy-idle) for an idle container instead of running the exec RPCs that auto-start it — so Lerma's `HealthMonitor` (which now probes `?passive=1`) never wakes a tenant. Non-billable tenants (staging 15m, dev 2h) sleep per a **derived** idle timeout (the Sandbox SDK 1.0 `setInactivityTimeout()` platform cap; there is no `SANDBOX_SLEEP_AFTER` binding; the value comes from the instance hostname via `src/dev-mode.ts`), and the `ActivityScheduler` DO alarm wakes/heals ahead of the next automation and runs an interval-gated R2 checkpoint after the wake. **Billable production** (`platform === 'production' && environment === 'production'`) does **not** auto-sleep; the container stays awake and is stopped only by an explicit lerma freeze (`POST /control` → `sandbox.destroy()`), so a warmed gateway never incurs a cold-start/downtime window after an idle period. Runtime cutover is gated by `SANDBOX_RUNTIME` (`legacy` default | `v1`); `SandboxV1` is still scaffold-only, so `resolveSandboxRuntime()` reports `v1` as `pending` and the legacy class keeps serving (`src/sandbox-runtime.ts`). Tests: `src/routes/public.test.ts`, `src/gateway/health-probes.test.ts`, `src/activity/alarm.test.ts`, `src/sandbox-runtime.test.ts`, `src/sandbox-do.test.ts`, `src/build-sandbox-options.test.ts`; corpus `test/business-rules/operational-modes.txt`; lerma `health_monitor_test.exs`, `health_poller_test.exs`, `tenant_provisioner_pipeline_test.exs`. See [docs/WORKER-DEVELOPMENT.md](/llms/platform/apps/fouria/worker-development/index.md).
- Single-owner gateway launch (`start-openclaw.sh start`): the production cold-start mode acquires the container-wide startup lock **before** config setup and holds fd 9 through the Gateway `exec`, so one owner covers setup **and** launch. The lock is inherited by the gateway process and released exactly when it exits, so only **one** owner can run — the OpenClaw 2026.9.8 state-DB single-owner migration otherwise caused concurrent launches (from different Worker isolates) to lose the migration, exit before binding the port, and leave the boot forever unhealthy (`state.sqlite is undergoing offline maintenance`). Losers poll the local `/health` (bounded) and exit 0 without a duplicate launch, or take over if the owner dies and releases the lock; the log-mirror `tail -f` closes fd 9 (`9>&-`) so it cannot hold the lock. The `launch` mode remains a standalone smoke/manual entry point with its dedicated `/tmp/openclaw-gateway-launch.lock`; `full` routes through the same lock in `background` mode. Regression: the serial `test/e2e/ac_gateway_startup_concurrency.txt` (`phaseD_startup`) asserts exactly one owner and the absence of both the migration-input-change error and the SQLite offline-maintenance error. See [docs/WORKER-DEVELOPMENT.md](/llms/platform/apps/fouria/worker-development/index.md).
- Fixed the develop cold-boot stall introduced by #318 (`startup.ok=false / readiness.ok=false` for the whole E2E readiness budget; 68 processes, `start-openclaw.sh setup` exited 0, `curl /health` always exit 7, **no** `openclaw gateway run` ever present). Root cause: #318 had split the boot so the **Worker** launched the gateway _after_ `start-openclaw.sh setup` exited (supervised exec) — this was the former behavior; current production collapses setup and launch into the atomic `start-openclaw.sh start` described above (startup lock acquired before setup, held through the Gateway `exec`). `/api/health` is bounded to a 5s snapshot, so the request returned long before the ~24s setup finished; when the response was sent, workerd cancelled the unregistered `ensureGateway` continuation, so the container-side setup completed but the Worker never resumed to launch the gateway. Fix: `/api/health` (and `/api/warmup`) now register the in-flight boot with `c.executionCtx.waitUntil(ensureGateway(sandbox, env, { waitForReady: true }))` so the setup → gateway-launch sequence survives past the response; `waitForReady: true` returns the shared start promise unwrapped and dedupes onto the one the snapshot already started. Also hardened the RPC-first health probe (`getRpcHealthSnapshot`, `src/gateway/health-probes.ts`) to run only once a gateway process is detected — `sandbox.containerFetch` auto-starts the container and blocks on the closed gateway port during a cold boot, piling port-waits on the Sandbox DO control channel. `buildHealthSnapshot` (`src/routes/public.ts`) passes its already-fetched `findGatewayProcess` result as `{ proc }`. Bisected: production E2E boot green at `ad9ad4a5` (#317), red at `9ae5f66c` (#318). See `apps/fouria/docs/WORKER-DEVELOPMENT.md` §Admin RPC.
- Startup resilience to Sandbox DO code updates: a Worker/DO redeploy (e.g. a Lerma redeploy of a tenant script) resets the DO↔container control session while the container keeps running, so the supervised `waitForPort` on the atomic `start-openclaw.sh start` process rejected with `Network connection lost.` and the cold start was aborted — killing a gateway that would have completed. The atomic `start` path has **no separate setup-process wait** (one supervised process spans config setup through the gateway's whole life), so only the port wait is transport-resilient: `ensureGateway` waits with `waitForPortResilient` (`src/gateway/sandbox-exec.ts`), which reattaches to the process by id via `getProcess(id)` and resumes the wait within the shared `STARTUP_TIMEOUT_MS` deadline; only a non-transport error, an un-reattachable process, or an exhausted deadline fails. The generic `waitForExitResilient` helper remains in `sandbox-exec.ts` for the legacy/boot-smoke supervised waits. The loading page's `GET /api/startup/ws` stream also treats a transport-drop snapshot as transient: it re-sends the last known state (or a synthetic `starting` view) and backs off exponentially (`STARTUP_WS_POLL_INTERVAL_MS` → `STARTUP_WS_MAX_BACKOFF_MS`, 30s) instead of re-probing and re-logging the same error every 5s.
- Backups run only on running instances; health is observability-only. `GET /api/health` no longer schedules snapshots (passive or active) — the lerma `HealthMonitor` is a pure probe and MUST NOT alter the R2 catalog. The **Dispatch Worker cron** (`workers/fouria-dispatch`, 15 min prod / hourly staging) fans a `backup` control action (`POST /control`, `X-FourIA-Control: backup`) out to every non-frozen tenant; the tenant asserts `sandbox.isRuntimeActive()` (a **non-waking** bridge read) and skips a sleeping container rather than waking it (an idle instance has no workload or filesystem changes, and waking would restart billing), then runs the interval/lock-gated `scheduleBackupIfNeeded`. `POST /debug/reprovision` now checkpoints only a live runtime, bounded by `REPROVISION_CHECKPOINT_TIMEOUT_MS`, verifies restorability by HEADing the newest catalog entries' `meta.json`, and **fails closed** (`409`) unless a restorable backup exists; `{"force":true}` overrides for an already-unrecoverable container. Root cause of the empty buckets: the old health-triggered path scheduled a snapshot regardless of `readiness.ok`, so a booting/stopped container could start `sandbox.createBackup()`, hang, and leave a stale `backup-lock` (observed in `fouria-backup-tarjetero`); dev instances the HealthMonitor does not monitor had no trigger at all. Tests: `src/routes/public.test.ts`, `src/routes/debug.test.ts`, `src/index.test.ts`, `workers/fouria-dispatch/test/unit/control.test.ts`; corpus `test/business-rules/operational-modes.txt`; docs `docs/OPERATOR-GUIDE.md`, `apps/fouria/docs/WORKER-DEVELOPMENT.md`, `docs/CLOUD-ARCHITECTURE.md`, `workers/fouria-dispatch/README.md`. E2E: `test/e2e-backup/zz_reprovision_guard.txt` + `test/e2e-backup/zzz_backup_cold_restore.txt` + `test/e2e-backup/zzz_backup_semantics.txt` (separate `test/e2e-backup/` corpus, excluded from the default `test/e2e` run; run locally with `bun run test:e2e:backup`; registered in `phaseE_restore`, reprovision guard first), the dispatch fan-out unit test `workers/fouria-dispatch/test/unit/cron-fanout.test.ts`, deterministic E2E backup env in `test/e2e/fixture/server/deploy`, and `POST /_control/:slug/backup` on the dispatch worker; design in `test/e2e/BACKUP-E2E-DESIGN.md`. Dispatch note: `vitest.config.ts` now has `unit` (node) + `integration` (workerd) projects so the fan-out logic is testable without Miniflare.
- PR E2E scope: the `e2e` job now sets `E2E_BOOT_ONLY=1` for **pull requests**, so `run-parallel.sh` provisions the worker and only asserts the gateway reaches `startup.ok + readiness.ok`, then tears down without running any cctr group. `develop` and a manual `workflow_dispatch` (new `full_tests` input, default checked) run the full corpus. See `apps/fouria/test/e2e-parallel/run-parallel.sh` and `docs/TESTING.md`.
- Setup serialization (`start-openclaw.sh`): `start`, `setup`, and `full` take an exclusive `flock` on `/tmp/openclaw-setup.lock` (~900s wait; `OPENCLAW_SETUP_LOCK_TIMEOUT` override) right after the gateway fast-path, then re-check the fast-path and exit 0 if a concurrent invocation brought the gateway up while they waited. In production, `start` acquires this lock **before** the setup body and holds it through the Gateway `exec`, so one owner serializes setup and launch. The Worker's `gatewayStartPromises` guard is per-isolate, so an E2E cold-start burst lands several runs in different isolates at once; without the lock they ran the slow setup body concurrently, contended on OpenClaw's native config/state locks, and stalled cold start for ~18 min (run 36692775633 — `[SandboxRecovery]` never fired because the admission-failure streak is also per-isolate). `full` releases fd 9 before backgrounding the gateway so the child does not inherit the lock; `post-start` is outside it. Degrades to the previous unserialized behaviour when `flock` is absent.
- Native supervised gateway launch (`src/gateway/process.ts`): the Worker's production cold start is one atomic supervised `sandbox.exec` of `start-openclaw.sh start`, which acquires the container-wide startup lock **before** the setup body (R2 restore → onboard → plugin installs → `moltlazy patch`) and holds it through the gateway `exec` — `/bin/bash -lc 'touch /tmp/gateway-stdout.log; pgrep … || ( tail -f … & ); exec openclaw gateway run … >> /tmp/gateway-stdout.log 2>&1'` — instead of the script backgrounding it with `&`. `exec` replaces the bash wrapper so the supervised process _is_ the gateway (`listProcesses`/`getProcess` track it for the life of the container; `kill()` targets it directly); the guarded `tail` starts the container-stdout log mirror as a sibling (safe because the Worker never `waitForExit`s on the gateway handle), and the redirect preserves the ops-readable log file. After the port is ready the Worker dispatches `start-openclaw.sh post-start` (dev automation + boot-log reset, best-effort) **detached** (`void`, not awaited) on every cold start — the gateway is already serving, so post-start must not extend the shared start promise (dev instances run `openclaw automations list/create` there, 30–60s, and awaiting it left every request logging `Gateway start already in progress … waiting...` while `/health` already returned 200); because it is dispatched unconditionally, a `/api/health`-triggered boot (`waitForReady:false`) still creates the dev automation even though its caller returned early. `start-openclaw.sh` gains `setup`/`post-start` modes; the default `full` mode keeps the legacy background launch for manual runs and `test:gateway:boot`. A shared `STARTUP_TIMEOUT_MS` deadline bounds setup + port wait; `launchWithRecovery` centralizes the stale/orphan/transport retry policy shared by both phases. **The `setup`/`post-start` phases must stay child-free**: the SDK's `waitForExit` only settles once the supervised terminal exits, which never happens while the script leaves a live background child (a `tail -f` log mirror or a background `cloudflared` there stalled every cold start for the full `STARTUP_TIMEOUT_MS`, re-running setup until the ~11-minute mark). The mirror is started by the native launch; cloudflared is detached with `setsid`. **Never call `output()`/`runShell` on the live gateway handle** (it never exits): read diagnostics from the `/tmp/gateway-stdout.log` / `/tmp/startup.log` tails instead (`readLogTail`).
- All three ClawHub plugins (`cloudflare-ai-gateway` provider, `diagnostics-otel`, `cloudflare-unified-billing`) are **pre-installed into the container image at Docker build time** (unpinned → latest ClawHub release; capability consent recorded in the image). This moves the per-boot network installs off the latency-critical startup path: `start-openclaw.sh` detects them via the extension dir + `plugins list` and only falls back to a ClawHub install when that state is missing (e.g. a warm R2 restore replaced it). The Dockerfile does **not** copy the Paso4 plugin source — it downloads the published ClawHub artifact — so the image stays source-free. The plugin's source and tests now live in [Paso4/openclaw-plugins](https://github.com/Paso4/openclaw-plugins) (removed from this repo); see [docs/OPENCLAW-PLUGINS.md](/llms/platform/libraries/openclaw-plugins/index.md). Runtime enablement is written by a single working `config patch` that also enables `admin-http-rpc` and disables `diagnostics-otel` unless its endpoint is set (the patch must not carry `plugins.bundledDiscovery` — removed in 2026.9.x, it fails validation and drops the whole patch — nor `plugins.allow`, which disables every bundled provider not listed). Memory search is disabled via the top-level `memory.search.enabled=false` (the old `agents.defaults.memorySearch` key is gone in 2026.9.x). Measured local cold boot: 35s vs 56s before
- Gateway restart split: `POST /api/admin/gateway/restart` is the **soft reload** — sends `SIGUSR2` to the live `openclaw gateway run` process (in-process, no connection drop, no port rebind) via `softReloadGateway` and falls back to `signalRestoreNeeded` only when no live gateway is found; `POST /debug/restart-openclaw` (DEBUG_ROUTES=true, ROOT) is the **hard restart** — `killGateway` + `resetStartupCache` + `signalRestoreNeeded` + `start-openclaw.sh` reprovision (in-place, no `startGatewayDirect`) so existing client tokens keep working. `SIGUSR1` is **not** a restart signal (it opens Node's inspector on `127.0.0.1:9229`); the E2E `n_gateway_soft_reload.txt` asserts the inspector stays closed after a soft reload. The reload targets the **server**, not the `openclaw gateway run` launcher: `buildSoftReloadScript` signals the exact `comm` (`pgrep -x 'openclaw-gatewa'`, Linux 15-char truncation, plus a defensive `openclaw-gateway`), because signalling the launcher kills it and orphans the server without restarting it (readiness never dips while the endpoint still reports `reloaded: true`). The E2E therefore requires both a stable `/api/status.processId` **and** an increased `http server listening` count, since `processId` alone survives a broken reload. Issue #463: `OPENCLAW_NO_RESPAWN` stays — removing it cannot enable process-level restarts inside a container (see the decision record in `docs/plans/sandbox-memory-usage.md`)
- `OPENCLAW_NO_RESPAWN=1` is set in the Dockerfile as explicit intent only: inside a container OpenClaw already resolves restart mode to in-process (`isContainerEnvironment()` → `disabled` / `unmanaged`), so removing the flag cannot enable process-level restarts or reclaim memory (#463); `softReloadGateway` only signals live processes and skips zombies
- Sandbox SDK 1.0 preview (`@cloudflare/sandbox@0.13.0-next.776.1` + `cloudflare/sandbox:0.13.0-next.776.1` image): `startProcess`/`execStream`/`getLogs()` and `SANDBOX_TRANSPORT` are gone. All command execution goes through `sandbox.exec(argv)` (which resolves on **launch**, not completion); `src/gateway/sandbox-exec.ts` centralizes the shapes (`startShell` for a handle, `runShell` for buffered output, `outputOf`, `collectLogs`). `listProcesses()` now returns `ProcessStatus[]` (`.state`, argv `.command`) and a live handle is fetched via `getProcess(id)`; `findExistingGatewayProcess` returns a `GatewayProcess` wrapper. `waitForProcess` delegates to `waitForExit({timeout})` and kills on timeout. `SandboxOptions.labels` was removed upstream. Deploy cutover requires `wrangler deploy --containers-rollout=immediate` (stable/1.0 control protocols are incompatible)
- Stale sandbox recovery (`src/gateway/sandbox-recovery.ts`): when the platform terminates the container behind a live Sandbox DO, every RPC fails with `sandbox.exec error … HTTP error! status: 500` — `isStaleContainerError` classifies these and `recoverStaleSandbox` destroys the dead instance + resets startup/persistence caches so the next RPC auto-provisions a fresh container; wired into `findExistingGatewayProcess`, `checkGatewayHealth` and the `ensureGateway` `startShell` retry; cooldown-guarded (60s), never fires for deploy `OperationInterruptedError` or transient provisioning errors. It also matches the **control-protocol mismatch** (`'utils.getRuntimeMetadata' is not a function`, `activateControlSession`, `RuntimeControlProtocolError`) seen during a deploy cutover while the still-running container predates the new SDK's control handshake — the SDK logs a `sandbox.exec error …` line but re-throws the bare `TypeError`, so the message patterns are matched directly and the stale instance is reprovisioned instead of failing until the platform rollout replaces it
- Orphaned-instance recovery (`src/gateway/sandbox-recovery.ts`): a container destroyed out-of-band (Cloudflare Containers dashboard "Destroy", failed force-stop) leaves the DO alive pointing at a dead placement and surfaces the SDK's _retryable_ admission errors (`no container instance available`, `max container instances exceeded`) instead of the 500 signature. `isContainerUnavailableError` + `trackProvisioningFailure` escalate to a recovery only once the streak reaches `ORPHANED_INSTANCE_MIN_FAILURES` **and** spans `ORPHANED_INSTANCE_GRACE_MS` (120s, > the SDK's ≈110s provisioning budget, so slow-but-healthy cold starts are never cancelled); `maybeRecoverSandbox` unifies both failure classes and clears the streak whenever a sandbox RPC succeeds. Operator lever: `POST /debug/reprovision` (DEBUG_ROUTES, ROOT) destroys the instance + clears caches + resets the streak **without** touching the gateway, so it works when the container is entirely gone
- Activity-driven wake (`src/activity/`): automations run inside the Gateway, so it must be running for schedules to fire. A per-tenant `ActivityScheduler` Durable Object (binding `ACTIVITY`, SQLite-backed, **separate** from the SDK `Sandbox` DO whose alarm drives `sleepAfter`) mirrors the tenant's schedule (`openclaw automations list --json`, refreshed on warm `/api/health` when `readiness.ok`, throttled to 5 min) and arms one alarm `CRON_WAKE_AHEAD_MINUTES` (default 10) before the next enabled job. On fire it wakes + heals via `ensureGateway` (recovering an orphaned instance) then re-arms; no enabled jobs → alarm cleared → container sleeps. Healing/wakeup is independent of user traffic, gated on job activity, and free when idle (`src/activity/alarm.ts` state machine is unit-tested; the DO is a thin adapter). Requires the `ACTIVITY` binding in `wrangler.jsonc` + lerma (`tenant_provisioner.ex` `resource_bindings`, `dispatch_namespace.ex` `@do_sqlite_classes`)
- Container image build profiles: the Dockerfile takes a `BUILD_PROFILE` build arg (default `production` → slim, only `xz-utils`; `dev` → adds the optional `pandoc` + `weasyprint` PDF/document stack, ~760 MB, used by the dev-only Fouria Builder `fouria-leads` skill). The profile is selected per wrangler environment via `image_vars` (equivalent to `docker build --build-arg`): `env.staging` (dev/test) sets `dev`; `env.production` and the default/local config keep the slim default. The release CD builds the slim default, and the E2E `basic` (PR) and `production` (develop) profiles both provision the slim production image on the same `standard-2` instance (differ by env only)
- Sandbox → leads R2 bridge (`src/sandbox-leads-bridge.ts`, `leads.r2.internal`): the `fouria-leads` skill now reaches the global `paso4-leads` bucket through the Sandbox `outboundByHost` bridge (plain HTTP resolved in the Workers runtime with the `LEADS_BUCKET` binding — no R2 creds in the container, no public route) instead of the public `wss://<worker>/api/leads/ws` proxy. The proxy is intercepted by Cloudflare Access on tenant hostnames (a per-tenant Access app shadows the `*.fouria.io/<path>` apps — confirmed via the login redirect's `kid`) and is retained only for the host-side E2E fixture. Related: lerma's `HealthMonitor` now probes with `redirect: false`, so a shadowed `/api/health` 302 (Access login page) is surfaced instead of being followed and misreported as healthy
- Client docs hardening + operations (dev-only): tenant-isolated the `leads.r2.internal` bridge (`src/sandbox-leads-bridge.ts` resolves the caller slug via the shared `src/sandbox-tenant.ts` resolver and treats every agent path as relative to `<slug>/`, so a sandboxed agent can only reach its own proposal; the shared `build-pdf.sh` stays read-only). Added the Admin **Client Docs** page (`/_admin/docs`, ADMIN, dev-only) + the extended `src/routes/api/fouria-docs.ts` managing docs (`customers/<slug>/docs/`) and leads **directly from the Worker** (no CLI) with markdown rendering (`marked` + `DOMPurify`), native-browser PDF opening via `/raw`, and multipart uploads; HTML/SVG raw downloads are forced to `attachment` + sandbox CSP. Added Worker-side git-like versioning (`DocsGitStore`: content-addressed blobs/commits/refs, `GET /commits`, `POST /rollback/:sha` as a non-destructive revert) and backups (`DocsBackupStore`: `POST /backup`, `GET /backups`, `POST /backups/:id/restore`, retention 10). See `docs/WORKER-DEVELOPMENT.md` §"Client Docs (dev-only admin surface)"
- `ECONNRESET` from the sandbox R2 bridges fixed: the tenant's `LEADS_BUCKET` was provisioned as a **plain-text var** (the bucket name) instead of an `r2_bucket` binding, so `env.LEADS_BUCKET` was a string, `bucket.list()` threw inside the `outboundByHost` handler, and the escaping exception surfaced to the container as an opaque `ECONNRESET`. Two fixes: (1) the bridges (`leads`/`docs`/`releases`) now validate the binding with `isR2Bucket` (`src/r2-binding.ts`) and wrap their bodies in try/catch, returning a structured JSON `503`/`500` instead of resetting the socket; (2) lerma (`instances.ex` `instance_bindings/2`, config `LEADS_R2_BUCKET` default `paso4-leads`) now provisions `LEADS_BUCKET` as an `r2_bucket` binding on every tenant worker (redeploy existing tenants to replace the stray var)
- Admin RPC-first read path (`src/gateway/admin-rpc.ts`): the hot dashboard reads/writes now go through OpenClaw's bundled `admin-http-rpc` plugin (`POST /api/v1/admin/rpc`) over `sandbox.containerFetch`, reusing the typed `moltlazy/sdk` HTTP `RpcClient` (which gained an injectable `fetcher`). This removes the ~10–15s per-call CLI spawn/WS-reconnect on `GET /api/admin/devices`, `POST .../approve`, `GET /api/admin/models`, and `GET /api/admin/channels/probe` (measured locally: ~1.3–2.5s CLI → ~30–40ms RPC), each of which keeps its CLI path as a fallback (`createAdminRpcClient` returns null / throws → existing parser runs). `start-openclaw.sh` enables the plugin with a dedicated `plugins.entries["admin-http-rpc"].enabled=true` patch — NOT `plugins.allow`, which disables every bundled provider not listed (and the nearby batched plugin patch is rejected upstream for the removed `plugins.bundledDiscovery` key). `src/index.ts` shadows `/api/v1/admin/rpc` with a 404 before the public catch-all proxy so the operator surface stays container-local. The home-grown `moltlazy/sdk` WebSocket client is intentionally not used for admin RPC
- Gateway WebSocket RPC (`src/gateway/gateway-rpc.ts`): methods outside the `admin-http-rpc` allowlist (notably `audit.activity.list`) are reached with OpenClaw's official `@openclaw/gateway-client/browser` reference client + `@openclaw/gateway-protocol` types, bridged over `sandbox.wsConnect` and authenticated with the shared gateway token as an `operator`. The wire-v4 handshake (`connect.challenge` → `connect` → `hello-ok`) is owned by the official client; the connect frame's `client.version` comes from `__OPENCLAW_VERSION__`, injected by `vite.config.ts` from `catalog.openclaw`. RPC-first with CLI fallback: `GET /api/admin/audit-logs` (`audit.activity.list`) and the agent board (`agents.create`/`agents.delete`/`agents.update`, model via `config.patch`). `GET /api/admin/integrations` is deliberately **WebSocket-free** (the gateway drops these sockets): channels/agents over the admin HTTP RPC, plugin enabled-state from `config.get`, toggles/config via `config.patch`, secrets reload via the CLI. `theme`, agent bindings, `secrets.apply`, and `channels list --all` have no RPC method and stay on the CLI. The two packages are pinned in `catalog.@openclaw/gateway-client` / `catalog.@openclaw/gateway-protocol` and must track `catalog.openclaw`
- Realtime device-pairing notifications (`src/gateway/gateway-rpc.ts` `openGatewayEventStream`, `src/routes/api/devices.ts` `GET /ws`): the dashboard `NotificationBell` now subscribes to `GET /api/admin/devices/ws`, which pushes an initial `{ pending, paired }` snapshot then relays Gateway `device.pair.*` events (reconnecting via the official gateway client over `sandbox.wsConnect`). Admins approve/reject pending devices inline from the bell via `POST /api/admin/devices/:requestId/approve` / `.../reject` (RPC-first; the reject route was added). Also fixed `secrets apply` invocations (`integrations`/`dreaming`): current OpenClaw removed `--yes` (only `--from`, `--dry-run`, `--allow-exec`, `--json`)
- CLI→RPC sweep (finishing the RPC-first migration): `config` set/unset use `config.patch`; `dreaming` config writes use `config.patch` (shared `patchConfigViaRpc` in `src/routes/api/shared.ts`); `secrets.reload` uses the WS `secrets.reload`; and `resolveAgentFlag` (used by `models`/`knowledge`) resolves the roster via `agents.list` instead of `openclaw agents list`. CLI fallbacks are retained everywhere. Remaining CLI-only paths have no Gateway RPC method: `secrets apply/audit`, `openclaw wiki *`, agent `bind`/`unbind`/`bindings`, and `openclaw --version`

Release notes for the fouria tenant Worker. Entries are generated by
`scripts/release.mjs` from the conventional commits that touched `apps/fouria`
since the previous CalVer tag.
