Sandbox v1 — staging observation record
On this page
- How to reproduce
- S1 — Authority, identity and platform gate (unstable-a90015f5...)
- S2 — Conversion intent, lease and DO conversion (unstable-486b2938...)
- S3 — Serving parity: proxy, health and admin RPC (unstable-41be4409...)
- Next observation
- OCM-4a — staging provisioning of the v1 surface (code landed, not yet deployed)
- OCM-4a live validation on staging (2026-11-05)
- S5 / PER-3 — legacy → v1 conversion, live (2026-11-05)
- PER-2 — v1 snapshots, recovery point and restore drill (2026-11-05)
- OCM-2 — durable lanes, boot guards and repair (2026-11-05)
- SEC-3 — v1 rollout park and resume (2026-11-06)
- SEC-3 (Lerma side) — the direct bundle upgrade parks and resumes (2026-11-06)
- Rollout park through the real promotion lane (2026-11-06)
- PER-2 — restore by record id on v1 (2026-11-06)
- Promotion gate fix verified live (2026-11-06)
- Combined promotion: promotion-gate repair + v1 idle snapshot cadence (2026-11-07)
- CDP bridge merge fix + HEAD redeploy (2026-11-07)
- Operator surface: Lerma migration + ROOT debug tools, and the already_converted signal (2026-11-07)
- Durable migration job + read-only status, live-validated (2026-11-08)
- Provisioning a tenant directly on Sandbox v1 (“born v1”), live-validated (2026-11-08)
- Wake-lifecycle merge into v1-sandbox and the combined deployment (2026-11-08)
- Rollback coverage + upgrade-pipeline repair (2026-11-09)
- Gateway proxy attribution fix, validated live (2026-11-09)
- Preflight bundle verification (2026-11-09)
- Born-v1 first-provision self-heal (2026-11-11)
- Born-v1 self-heal: live drill attempt and the active fix (2026-11-11)
- Develop integration sync, production-surface removal and the local gate (2026-11-12)
- Staging validation after the cec359fa promotion (2026-11-12)
Append-only record of the step 5 observation for each milestone on the
v1-sandbox track. The target is Lerma client client-sandbox-v1 (id 20) and
its two instances (tenant-client-sandbox-v1-dev, tenant-client-sandbox-v1)
in the Paso4 staging account, dispatch namespace fouria-tenants-staging.
Nothing here is a production observation: no production tenant has been
converted, and the migration authority (WORKER_PLATFORM === "staging") is not
present in production.
How to reproduce
# Instance state + health (staging Lerma)
# docker exec -i lerma-pre sh -s <<'EOF'
# ... Application.get_env(:lerma, :provisioning)[:dispatch_namespace] ...
/app/bin/lerma rpc 'Lerma.Instances.list_all_for_client(20)'
# Deployed script identity, bindings and bundle content (Cloudflare API)
# GET /accounts/<acct>/workers/dispatch/namespaces/fouria-tenants-staging/scripts
# GET .../scripts/<name>/settings
# GET .../scripts/<name>/content # grep for SANDBOX_V1_* markers
# GET /accounts/<acct>/containers/applications
# Worker telemetry (Lerma ingest of the Observability destinations)
# from(e in "observability_events", where: e.script_name in ^names, ...) |> Repo.all()S1 — Authority, identity and platform gate (unstable-a90015f5...)
- Both instances report the version; Lerma health
:ok. - Health events for production:
down(503, 7.5 s) at 20:00:52 →upat 20:03:37 during the rollout window. - No migration opportunity was exercised (there is no conversion yet).
S2 — Conversion intent, lease and DO conversion (unstable-486b2938...)
- Both instances report the version, status
sleeping, health:ok. Dockerfile.v1includessquashfs-tools.- No intent exists, so both tenants remain legacy. No conversion was triggered.
S3 — Serving parity: proxy, health and admin RPC (unstable-41be4409...)
Verified
| Check | Result |
|---|---|
| Tenant scripts modified at the promotion times | ✅ tenant-client-sandbox-v1 08:20:47Z, …-dev 08:22:41Z |
| Bundle contains the S3 code | ✅ SANDBOX_V1_RUNTIME_UNKNOWN, SANDBOX_V1_CONTROL_UNSUPPORTED, SANDBOX_V1_SLEEPING, sandbox.runtime.selected, __fouriaSandboxV1, v1EnsureGateway |
| Same bundle on both instances | ✅ identical script etag |
| Container image | ✅ fouria:unstable-41be4409… on both legacy tenant apps |
| Container health | ✅ 3 healthy / 0 active per app (idle-slept) |
| Lerma instance state | ✅ status sleeping, deployed_version = unstable-41be4409…, HealthMonitor.check_instance_now/1 → :ok |
| Worker telemetry after the deploy | ✅ spans 200 (80 / 67), no error outcome, no exception class after 2026-11-04T20:00Z |
503 spans | all 13 + 13 from 2026-11-04 20:02–20:03Z (first cold boot), none after the S3 deploy |
| Latest activity | ✅ 2026-11-05T08:57Z |
Hosted /healthz | ✅ 302 (Cloudflare Access), no 5xx |
Blocking finding: the v1 path is deployed but cannot be selected
The live tenant script bindings (read from the dispatch-namespace settings) are:
ACTIVITY, ANALYTICS, BACKUP_BUCKET, BACKUP_BUCKET_NAME,
BACKUP_CHANGE_DETECTION, BACKUP_CHANGE_WATCH,
BACKUP_ENCRYPTION_KEY, BACKUP_ENCRYPTION_REQUIRED, BROWSER, CDP_SECRET,
CF_ACCESS_*, CF_AI_GATEWAY_*, CLOUDFLARE_*, DEBUG_ROUTES, LEADS_BUCKET,
MOLTBOT_GATEWAY_TOKEN, OPENCODE_API_KEY, PASO4_TELEGRAM_GROUP_ID,
R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, Sandbox,
TELEGRAM_INGRESS, TELEGRAM_INGRESS_HOST, TELEGRAM_INGRESS_QUEUE,
WORKER_URL.
Missing for the v1 track:
WORKER_PLATFORM(andSANDBOX_V1_CUTOVER).resolveRuntimeIntentrequires the exact trusted valuestaging; without the binding it always returnslegacy. The S3 runtime selection therefore never activates.SandboxV1Durable Object binding (and its class migration). The bundle exportsSandboxV1, but with no namespace bindingacquireSandboxV1()would throwSandboxV1 binding is not configuredif selection ever reached it.- A v1 container application (
scheduling_policy: "durable_object", v1 image). The account has only the per-tenant legacy applications (tenant-<slug>-sandbox-staging, imagefouria:unstable-…). Thefouria-sandbox-v1-stagingapplication declared inwrangler.jsoncis created by a standalonewrangler deploy, not by Lerma’s dispatch-namespace upload, so it does not exist for tenants.
Root cause
Tenant scripts are uploaded by Lerma into fouria-tenants-staging, not by
wrangler deploy with this repo’s wrangler.jsonc environment. The two Lerma
paths differ:
Lerma.Provisioning.TenantProvisioner.required_var_bindings/2addsWORKER_PLATFORM/SANDBOX_V1_CUTOVER(S1 change).Lerma.Instances.instance_bindings/2, used by the upgrade/promotion redeploy, does not add them (and neither path addsSandboxV1). The promotion therefore re-uploads the tenant with the legacy binding set.- The deployed staging Lerma runs
develop, which does not contain the S1 provisioning change at all, so instances created before it also lack the bindings.
Verdict
- S3 is deployed (bundle identity proven) and inert (runtime selection cannot choose v1; the v1 runtime is not provisioned). This is a truthful “deployed, not activatable” state, not a runtime failure: legacy serving is unchanged, with no post-deploy errors.
- Activation belongs to OCM-4 (provisioning, rollout, promotion) with the
conversion orchestration in S5 / PER-3:
- add
WORKER_PLATFORM,SANDBOX_V1_CUTOVERand theSandboxV1DO binding (plus its class migration) toInstances.instance_bindings/2; - provision/declare a per-tenant v1 container application with the v1 image
and
scheduling_policy: "durable_object"(naming + rollout policy); - redeploy the staging Lerma with the v1-branch changes so an upgrade carries the bindings;
- then run the conversion job and repeat this observation against the v1 runtime (HTTP/WS proxy, passive health, single owner).
- add
Next observation
Repeat S3 once the binding/application gap above is closed: the same API reads
must show SandboxV1 in the tenant bindings and a per-tenant v1 container
application, and the observability stream must show v1-path traffic
(sandbox.runtime.selected: v1) with no proxy/WS errors.
OCM-4a — staging provisioning of the v1 surface (code landed, not yet deployed)
Closing the gap above is provisioning work, implemented on v1-sandbox:
Lerma.Cloudflare.SandboxV1is the single selection point:enabled?/0is true only forcf_environment == "staging";image_enabled?/1additionally requires anunstable-*validation image (the publisher pushes the matchingv1-unstable-*tag);v1_image/1,binding/0,container_config/3anddrop_binding/1keep the binding, the class migration and the container application keyed on the same predicate.- Provisioning and upgrade bindings both add
WORKER_PLATFORMandSANDBOX_V1_CUTOVER; the staging platform also gets theSandboxV1DO binding. The upgrade path (Instances.instance_bindings/2) previously lacked the platform bindings entirely — that was the root cause above. DispatchNamespacederives thenew_sqlite_classesset from the declared container entries, soSandboxV1is migrated only when it is also bound (production stays legacy-only).namespace_id_for_class/3resolves the per-class namespace.Containers.deploy_script_and_container/1declares both container entries on staging/unstable images, prepares the v1 image artifact, and creates the per-tenant…-sandbox-v1-<platform>application withscheduling_policy: "durable_object". It drops theSandboxV1binding when the image has no v1 tag, so an upload can never bind an unmigrated class.- Deprovisioning removes both applications.
Verified locally (unit + full Lerma suite); the staging effect is not yet
observed because the running staging Lerma predates this change. The next
step is to deploy the branch Lerma to staging (:latest-pre lane), promote a
tenant, and repeat the checks above. Note the manual fallback: a promotion from
develop Lerma would re-upload the legacy binding set, so the branch Lerma
must be the one driving the upgrade.
OCM-4a live validation on staging (2026-11-05)
The branch Lerma was temporarily deployed to the staging host under a unique
local image tag (transferred with docker save/load, compose repointed, then
restored to :latest-pre afterwards), a fresh validation build was published
(unstable-8ce99552…), and both tenants were upgraded through that Lerma.
Verified live
| Check | Result |
|---|---|
| Worker bundle | both tenant scripts at unstable-8ce99552…, identical script etag |
| Platform bindings | WORKER_PLATFORM = staging and SANDBOX_V1_CUTOVER = false on both |
| v1 DO binding | SandboxV1 bound (prod namespace c1b83745…, dev f50564f8…) |
| v1 container apps | tenant-client-sandbox-v1{,-dev}-sandbox-v1-staging, scheduling_policy: "durable_object", empty application config (image comes from ctx.container.images) |
| Named image | script metadata carries images.sandbox as a digest-pinned Cloudflare Registry reference (…/fouria@sha256:35b931…) |
| Legacy apps | both on fouria:unstable-8ce99552…, 3/3 healthy, no errors |
| Instance state | both running, HealthMonitor.check_instance_now/1 → :ok; /healthz → 302 (Access) |
| Telemetry after recovery | no non-ok spans or exception classes; latest activity current |
| Runtime selection | still legacy — no conversion intent exists, so the v1 surface is inert as designed |
Incident during the first validation attempt (both scripts deleted)
The first two upgrade attempts used an earlier branch image whose v1
container-application body still contained configuration.image /
instance_type (a durable_object application rejects them). The API
rejected the app creation after the script upload, and the atomic-deploy
rollback deleted the just-uploaded script — for a live tenant that also removed
its routing, leaving both instances in error (the containers themselves stayed
healthy). Recovery was a re-run of upgrade_instance once the code was fixed.
Two code changes came out of it:
preserve_do_namespacesnow covers the script. An upgrade rollback must never delete a live tenant’s script (it is what routes the tenant); the newly uploaded script and its DO namespaces are kept and the failure is reported for a retry. Fresh provisions still clean up.- The upgrade binding set is self-sufficient.
Instances.instance_bindings/2previously relied onkeep_bindingsfor the per-instance/platform bindings, so a rebuild after a deleted script left the tenant without the gateway token, CDP secret, AI Gateway credential, Access AUD, agent roster, Telegram queue, and other vars. It now re-asserts the full set:MOLTBOT_GATEWAY_TOKEN(stored), a deterministically derivedCDP_SECRET, AI Gateway/API token,OPENCODE_API_KEY(dev), the per-instance Access AUD (resolved by the tenant’s own hostname), agent/scope flags, account/team vars, Telegram ingress queue/host, change-detection vars andDEBUG_ROUTES. Verified after recovery: 32 bindings on dev, 31 on production (onlyDEBUG_ROUTESis dev-only).
Operational notes
- The staging Lerma is back on
:latest-pre; the branch image was a temporary validation deployment. A later promotion fromdevelopLerma would again drop the v1 bindings, so the branch Lerma must drive the upgrade until this lands. - The v1 container applications are idle (no instances) and start only when a
converted tenant’s DO calls
container.start(). - Next: S5 / PER-3 conversion orchestration (intent +
convertToV1through the control plane) so a tenant can be converted and served onSandboxV1.
S5 / PER-3 — legacy → v1 conversion, live (2026-11-05)
Bundle unstable-e7d193fd… promoted to client-sandbox-v1 (id 20) from the
branch Lerma (ghcr.io/0xcab0/lerma:v1-sandbox-02327309, deployed temporarily).
| Check | Result |
|---|---|
POST /_control/:slug/migrate | 200 { state: "committed", recordId: "77cadb66-cc11-4561-a6b5-edc1f1a52248" } on client-sandbox-v1-dev |
| Conversion record | sandbox-migration/client-sandbox-v1/conversion.json — v1 DirectoryBackup, dir=/home/openclaw, 22.5 MB tar+zstd/1, source f19e007a… |
| C8 barrier | sandbox-migration/client-sandbox-v1/stable.json points at the record (committed 12:04:26) |
| Runtime gate | backup on the converted instance → 503 SANDBOX_V1_CONTROL_UNSUPPORTED (v1) |
| Gate is per-tenant | backup on client-sandbox-v1 (still legacy) → 200 (no conversion intent) |
| v1 container | tenant-client-sandbox-v1-dev-sandbox-v1-staging / instance openclaw running, image digest sha256:bd925b80… |
| Served end-to-end | GET /api/health (health-monitor service token) → 200, status: "running", startup.ok: true, readiness.ok: true, aiGateway/container/durableObject/r2 all up |
| Instance records | both instances running on unstable-e7d193fd… |
Defects found and fixed while making the conversion work
- A queued conversion stayed
queued— at the time this read as “the DO alarm never fires in the tenant runtime” (no alarm span, no error), so the control plane briefly awaitedv1ConvertNow()inline (the legacy checkpoint’s long-request pattern). Corrected: alarms do fire on this lane — the idle-snapshot cadence alarm later committed record4437c278-…with no operator action — so the inline path was removed.migrateis now queue-only (202+ poll) and the conversion runs on the DO alarm, with the durable marker re-arm (ensureConversionArmed) covering a lost alarm. SeeWORKER-DEVELOPMENT.md§ Sandbox v1 conversion orchestration (S5). execstdin silently truncated the upload. Passing aReadableStreamasexecstdin is accepted but did not deliver every byte (unsquashfsread a partial file and exited 2, while the R2 object was a valid SQUASHFS 4.0 lz4 image, verified locally and in the sandbox base image). Uploads now usestdin: 'pipe', close the pipe before awaiting the exit code, and surface the extractor’s stderr.- The archive is not a copy of
/home/openclaw. The legacy bridge stages the roots under.fouria-legacy-stage/{openclaw,clawd}and leaves/home/openclaw/.openclaw+/home/openclaw/clawdas symlinks into/root. The v1 image has real directories (aDirectoryBackupdoes not follow symlinks), so extraction now materializes the staged roots as real v1 directories and relocates the absolute legacy paths (/root/openclaw→/home/openclaw, length preserving and applied in place to the state SQLite with anode:sqlitequick_check;/root/clawd→/home/openclaw/clawdin text files only). A snapshot without a complete legacy stage fails closed. - Serving before the barrier committed. A present intent used to resolve as
v1immediately, so a failed conversion could wake a fresh empty v1 container and onboard over the legacy recovery point. The gate now requires the record + pointer (C8) and answers503 SANDBOX_V1_CONVERSION_PENDINGuntil then. - A gateway launch failure retracted a committed conversion.
convertToV1no longer launches: the commit is the data artifact and a launch failure is logged (sandbox.v1.conversion.launch_failed) and retried by the next request’s v1 boot, instead of reportingfailedand re-running the destructive conversion on retry. - Lerma marked a healthy rollout as
error.await_container_healthyrequiredactive_rollout_idto clear, but the platform keeps reporting it after every instance serves the new image (both tenants: 3/3 healthy for the whole 10-minute budget). Completion is now “no active rollout or the application image carries the target version”.
Pre-flight proof of the materialization
materializeLegacyArchive was driven against the real stable archive inside the
published v1 image (a docker exec-backed fake container): the extracted state
boots start-openclaw.sh start with OPENCLAW_RESTORED=true, /health and the
Control UI answer 200, the agent workspace is present, and the state database
reports quick_check: ok with zero /root/openclaw references.
Known gaps (next slices)
- The Control UI (
GET /) sits behind the tenant host Access application, which rejects the health-monitor service token (service_token_status: falsein the login meta) and needs an interactive identity, so the browser surface could not be exercised headlessly. The health path is a separate, token- accepting Access app and is what proved readiness (status: running,startup/readinessok, all components up). - v1 control actions other than the idempotent
migratestill answer503 SANDBOX_V1_CONTROL_UNSUPPORTED(checkpoint/backup/resume on v1). - Only the dev-env instance was converted; the production-env staging instance remains legacy until the S6 fleet decision.
Deployment state left behind
- Staging Lerma (
root@46.225.138.152, containerlerma-pre) runs the branch imageghcr.io/0xcab0/lerma:v1-sandbox-02327309; the original compose file is kept at/opt/lerma/docker-compose.yml.bak-v1. Roll back with/tmp/rollback-lerma.sh(or restore that backup anddocker compose up -d --no-deps --force-recreate app). - The staging Dispatch (
dev.fouria.io) is the branch deploy, so it still forwardsmigrateand echoes the conversion state. A develop deploy replaces it and drops that forwarding. - Both are deliberately left in place while the v1 lane continues (the converted tenant needs the v1-aware upgrade path); they are staging-only and must be reverted before any production promotion.
PER-2 — v1 snapshots, recovery point and restore drill (2026-11-05)
Bundles unstable-6795827b… (catalog + snapshots + freeze) and
unstable-473c46b0… (pre-catalog adoption) promoted to client-sandbox-v1
(id 20); the drills ran on client-sandbox-v1-dev.
| Check | Result |
|---|---|
POST /_control/:slug/backup | 200; sandbox.v1.snapshot.committed (record 79165e52…, 22.5 MB) — served by the DO, no .sqsh written |
| Catalog + pointer | sandbox-migration/client-sandbox-v1/records/<id>.json; stable.json advanced to the new record |
| Encryption at rest | sandbox-v1/openclaw/<id>.tar.zst.enc (22.5 MB) + <id>.meta.json (encrypted: true, plaintext size/sha256) — no plaintext object |
POST /_control/:slug/freeze | 200; snapshot committed then the container was destroyed (status: stopped) |
| Cold boot restore | one non-passive GET /api/health (health-monitor token) → new container instance started 17:18:17 → 200, running, startup.ok/readiness.ok within ~15 s |
| Retention | 4 records → 3; the oldest catalogued record’s entry and 22.5 MB ciphertext removed, the new record protected |
| Pre-catalog adoption | a conversion record written before the catalog existed was adopted on the next snapshot (fix 473c46b0) |
Notes:
- A
durable_objectcontainer application is an instance application: the rollout API rejects it (Instance applications do not support rollouts), so a container replacement is either the platform’s own instance churn or the DO’sfreeze→ next-request boot.freezetherefore doubles as the restore drill lever. - A non-passive
/api/healthboot is detached: the first probe answers503 stoppedin ~0.5 s while the registered boot runs, and the next probe is200 running. The loading page’s startup WebSocket drives the same boot. - Retention only manages catalogued records; a record committed before the
catalog existed needs adoption, which is why a snapshot now adopts
conversion.jsonbefore listing (found live: the pruned record’s payload was deleted correctly, the un-catalogued conversion archive was not).
OCM-2 — durable lanes, boot guards and repair (2026-11-05)
Bundle unstable-889ec3a2… promoted to client-sandbox-v1 (id 20); drills on
client-sandbox-v1-dev.
| Check | Result |
|---|---|
| Scheduler runtime resolution | the alarm snapshot after the promotion wrote v1 record 2c97395d… (catalog + pointer), not a legacy archive |
| Legacy lane untouched | legacy app instance stays inactive; backups/f19e007a… and backup-handle.json still date from 2026-11-04 |
| Freeze on the new bundle | 200, backupId: 8910f736…; pointer advanced to it |
| Cold boot after the freeze | instance started 17:50:25; 503 stopped ×3 (~36 s, detached boot) → 200 running with startup.ok/readiness.ok |
| Launch guards (unit) | non-zero early exit fails in seconds (not after the 300 s budget); a corruption signature queues doctor-repair (gateway-launch-state-database) |
Notes:
- A v1 upgrade redeploy runs
checkpoint(v1 snapshot), notcheckpoint-park: park is still legacy-shaped and belongs to SEC-3, so the rollout is not write-blocked on v1 yet. - The scale-to-zero path is the platform inactivity stop (no DO idle callback), so “stop without re-archiving on idle expiry” remains open (PER-2 note).
- The corrupt-state-DB repair is covered by unit tests only: forcing real SQLite corruption on the staging tenant would risk the tenant for no extra signal.
SEC-3 — v1 rollout park and resume (2026-11-06)
Bundle unstable-548cb408… promoted to client-sandbox-v1 (id 20); the drill ran
on client-sandbox-v1-dev.
| Check | Result |
|---|---|
checkpoint-park | 200 { parked: true, backupId: 2455d8c8…, sessionsPatched: 0, sessionsDrained: false } (no live sessions) |
| Recovery point committed | the pointer advanced to the park record before the write lock armed |
| Write while parked | POST /_control/:slug/backup → 409 TENANT_PARKED (echoed by the dispatch as code) |
resume | 200 { resumed: true }; the park record was cleared |
| Readiness after resume | health probe 200 running on the first attempt (park never destroys the container, so no restore is needed) |
| Unparked resume | no-op that never wakes the tenant (unit-tested) |
Notes:
- The Lerma promotion of this bundle used
checkpoint(the direct-upgrade path), notcheckpoint-park, and loggedPre-replacement checkpoint unconfirmedplus a skipped legacy job: the v1 payload carriedrecordIdbut the control contract names the committed archivebackupId, so the dispatch did not echo the evidence.fdb1f639returns both. Follow-up (closed): right after the promotion the runtime still answered a plaincheckpointwithoutbackupId(thecheckpoint-parkbranch from548cb408was already serving), so a bundle-blackend question was recorded. A re-deploy of the same bundle settled it:POST …/checkpointnow returns{ ok, action: "checkpoint", backupId: "dd0b1be3-7116-4a7d-b970-21f54b08cc5d" }and the deployed content carries the alias. The transferable finding is the preflight asymmetry:upgrade_all_instances_asyncskips an instance when the live container image already matches the requested version, so a script upload that did not take effect cannot be repaired by re-promoting. Recorded as a follow-up (verify the deployed bundle, not only the image) rather than fixed in this slice. - Park is intentionally a write-block only: the container is replaced by the
rollout, not by the park, so
resumeneeds no restore.killGatewaystays process-scoped. - The session drain found nothing to patch on this tenant; the read-only drain and
the durable mode map are exercised by unit tests (
session-drain.test.ts,session-mode-restore.test.ts).
SEC-3 (Lerma side) — the direct bundle upgrade parks and resumes (2026-11-06)
Instances.upgrade_instance/3 (the path the staging validation promotion API
uses) re-uploaded the bundle and rolled the container without write-blocking the
tenant. It now mirrors @upgrade_pipeline: parked checkpoint first (fail-closed
on a failed or non-parked response), released after the rollout settles and on
every error path, sleeping instances skipped without a resume request.
Verified live against the branch Lerma image v1-sandbox-e5dd5278 on
client-sandbox-v1-dev:
| Check | Result |
|---|---|
Containers.checkpoint/2 | {:ok, %{backup_id: "545f7844-…", parked: true, checkpointed: true}} (direct probe) |
Containers.resume/1 | {:ok, %{resumed: true}} |
| Upgrade parks | a full upgrade_instance moved the v1 recovery point to record d1c9f143-1aba-4bc1-b8e0-31522497eb06 (committed inside the upgrade) |
| Upgrade releases | POST …/backup → 200 (not 409 TENANT_PARKED) and /api/health 200 running right after |
| Missing park (not running) | the first drill ran while the container was idle: skipped: not-running was accepted, no resume was sent, and the pointer did not move |
Note: logs emitted inside lerna rpc probes are not visible in docker logs
(the RPC process does not own the console), so this drill is evidenced by the
durable v1 pointer + the post-upgrade write acceptance rather than by Lerma log
lines. A promotion-driven run (Task in the app) is the way to read those lines.
Rollout park through the real promotion lane (2026-11-06)
Bundle unstable-fd05f0c3… (the v1 outbound-bridge fix) was promoted through
Lerma’s promotion API, i.e. the same Task-driven upgrade_instance/3 path a
release uses. This is the first end-to-end park → replacement → resume run:
| Phase | Evidence |
|---|---|
| Park | 16:06:41 Pre-replacement checkpoint parked for tenant-client-sandbox-v1-dev: backup_id="d1c02116-…" parked=true |
| Recovery point | sandbox-migration/client-sandbox-v1/stable.json → d1c02116-5363-4ab8-bb3e-388a776ce03d (committed before the upload) |
| Script + rollout | script PUT …/scripts/tenant-client-sandbox-v1-dev, then the container application rolled to unstable-fd05f0c3… |
| Settle | Waiting for … container health … healthy 3, rollout_active: false |
| Resume | 16:09:21 Resume after container replacement for tenant-client-sandbox-v1-dev: resumed=true |
| After | POST …/backup → 200 (park cleared); /api/health → 200 running, readiness.ok: true; instance running on fd05f0c3… |
Defects found by the drill
- Premature convergence. The promotion gate read the DB only:
upgrade_all_instances_asyncsetsdeployed_versionto the requested version before deploying, andHealthMonitor.mark_healthy/3flipped the in-flightupgradinginstance back torunning(the container being replaced still answers200). The gate therefore declared success ~30 s into a rollout that took ~2m40s. Fixed: the status endpoint reportsserving_version(the live container-application image tag),promote-validation.shrequires it to match, and the health monitor no longer transitionsupgradinginstances. - Stuck rollout carried across promotions.
tenant-client-sandbox-v1(prod-env) still hadrollout_active: trueon the previous image (unstable-548cb408…) from the earlier promotion, so the fd05 promotion timed out settling it and marked the instanceerrorfor hours; because the gate ignoreserrorinstances, the promotion still reported success. Re-runningupgrade_instancesuperseded the stuck rollout and the instance is nowrunning/healthy onfd05f0c3…. Follow-up worth doing: for a converted tenant, gate on theSandboxV1application (or skip the legacy app), since the legacy app is unused by the v1 runtime and can wedge independently. upgrade_instancereliability verified for the Lerma-side park: sleeping containers reportskipped: not-running(no park, no resume), a failed or plain (parked: false) checkpoint aborts before the upload, and the park is released on the error path too (all covered bytest/lerma/instances_upgrade_test.exs).
Still open for this bundle
The v1 outbound virtual hosts fix (.github-published src/sandbox-outbound.ts
SandboxV1intercept registration) is unit-verified and live on both instances (fd05f0c3…), but its runtime effect (the container reachingotel.internal) has no operator-visible signal in the v1 lane yet: the tenant was idle, so no model spans were exported and no[otel] Received …log line was produced. Confirm on the next agent turn (or when an operator checks the tenant’sdiagnostics-otelhealth).
PER-2 — restore by record id on v1 (2026-11-06)
POST /api/admin/storage/restore/:id on a converted tenant used to fall into the
legacy .sqsh path (unsupported on v1), so an operator could not apply an older
v1 recovery point. The route now branches on the runtime:
| Step | Behaviour |
|---|---|
| Maintenance window | opened (restore) before the DO touches the container; the v1 ensureGateway branch honors it, so no request can reboot under the swap |
| Stop writers | DO kills the single gateway owner, then waits (≤5 s) for /health to fail so the SDK swap does not hit EBUSY |
| Apply the record | DirectoryBackup.restore(record, { dir: HOME_DIR }) — the SDK swaps the directory atomically |
| Recovery point | by default the stable pointer advances to the restored record (?setCurrent=0 applies it without promoting) |
| Fail-closed | unknown/unreadable record → 404, container untouched; a restore already in progress is refused by an in-DO flag |
| Boot | Worker boots the restored state and reports gatewayReady; a boot failure never undoes the restore (next request retries) |
v1 restores are synchronous (there is no legacy job queue for the record format).
Live drill status: the code path is unit- and corpus-verified
(sandbox-v1.test.ts, storage.test.ts, business-rules/sandbox-v1-persistence.txt),
but the live drill is blocked by Access: /api/admin/* on a tenant host is
behind the host Cloudflare Access app, and the only service token available to
this lane (the health monitor) is accepted by the health app, not the host app.
An operator session (or a browser login) is required to exercise the route on
client-sandbox-v1-dev; the bundle unstable-9952203a… ships the code.
Promotion gate fix verified live (2026-11-06)
unstable-9952203a… (v1 restore-by-record) was promoted through the validation
lane after redeploying the branch Lerma that carries the new gate:
- the branch Lerma build
v1-sandbox-9952203aexposesserving_versioninGET /api/provisioning/upgrade/:slug; - the promotion waited for the real rollout:
Lerma accepted …at the start, then convergence only after both instances reportedserving_version == unstable-9952203a…(rollout_active: false, healthy 3); the pre-fix run had declared success ~30 s in while the rollout was mid-flight; - park/resume evidence on the dev instance: recovery point advanced to
26c78dac-ffcb-4574-a4b8-5b82c4be792b(committed at 21:57:22, before the image rollout), andPOST /_control/…/backupafterwards returns200(the park was released byresume); - final state: both instances
deployed_version = serving_version = unstable-9952203a…, container health 3/3,tenant-client-sandbox-v1-devrunning,tenant-client-sandbox-v1sleeping(idle container).
Operational note: the promotion gate requires serving_version, so the Lerma
image running on staging must include that change; an older Lerma image makes a
promotion hang until its timeout (it never sees the field). The staging Lerma was
redeployed from v1-sandbox-9952203a (compose backup:
/opt/lerma/docker-compose.yml.bak-latest-pre, which still points at
latest-pre).
Combined promotion: promotion-gate repair + v1 idle snapshot cadence (2026-11-07)
Two slices landed together and shipped as one validation bundle, so the staging lane picked up the cadence and the promotion-gate repair in a single rollout:
| Commit | Slice |
|---|---|
4389a6e4 | Lerma promotion repair + truthful gate. POST /api/provisioning/upgrade/:slug also retries error instances whose deployed_version is not the requested version (an instance already declaring the target is excluded, since re-driving a broken container at the target only loops). promote-validation.sh is_converged now requires zero error instances, so a promotion can no longer report success while a tenant stays down at the old image. |
855810b9 | v1 idle snapshot cadence (PER-2). A converted tenant is owned by SandboxV1, and raw ctx.container has no eviction hook, so the platform’s inactivity stop silently dropped writes since the last committed record. The DO now arms a durable marker + ctx.storage.setAlarm on startContainer() and, on alarm(), commits a backup snapshot then re-arms. It never wakes a stopped container, skips while a restore is in flight, and skips when stable.json is newer than the interval (deduping with the ActivityScheduler). resolveSnapshotIntervalMs resolves SANDBOX_V1_SNAPSHOT_INTERVAL_MS → BACKUP_INTERVAL_MINUTES → 1 h, capped at half the inactivity timeout (staging 7.5 m, dev 1 h). selectRecordsToPrune breaks a same-millisecond createdAt tie by record id, so retention is deterministic. |
795f1dab | Test hygiene (unrelated to the track). The ensure_container_application/5 keyless dev-SSH test now pins its own :provisioning[:dev_instance_ssh_public_key]; config/runtime.exs loads the developer-local apps/lerma/.env (which sets DEV_INSTANCE_SSH_PUBLIC_KEY), so the test only passed on a machine without that entry. |
Autogenerated-types covenant (checked, no fix needed)
855810b9 adds SANDBOX_V1_SNAPSHOT_INTERVAL_MS?: string to
apps/fouria/src/types.ts. apps/fouria/AGENTS.md calls that file
autogenerated, but bun run types (wrangler types) writes
apps/fouria/worker-configuration.d.ts only — src/types.ts is the
hand-maintained OpenClawEnv, and the call site already narrows it with its own
SnapshotCadenceEnv ({ SANDBOX_V1_SNAPSHOT_INTERVAL_MS?, BACKUP_INTERVAL_MINUTES? }).
The addition therefore stands, no call-site cast was needed, and the regenerated
worker-configuration.d.ts (unrelated workerd/binding churn) was reverted rather
than committed.
Deployment
- Bundle + both runtime images:
unstable-795f1dab0d6b2d4bce7b6a98c9b5afb3be089e1f(fouria:unstable-…+fouria:v1-unstable-…, R2fouria-bundles/unstable-…/fouria.mjsandcontainer-manifest.json). - Staging Lerma image:
ghcr.io/0xcab0/lerma:v1-sandbox-795f1dab, built from the repo root and shipped bydocker save | ssh … docker load(no ghcr push), compose backup/opt/lerma/docker-compose.yml.bak-pre-795f1dab, containerlerma-prehealthyand/health200. - One promotion for everything:
Lerma accepted unstable-795f1dab…: 2 running instance(s)→client-sandbox-v1 is running unstable-795f1dab…. The direct bundle-only path (Instances.upgrade_instance/3) completes park → script upload → container rollout → resume inside the POST, so the gate converged on its first poll (park/resume visible in the Lerma log below).
Live drills (2026-11-07, unstable-795f1dab…)
| Drill | Result |
|---|---|
| Status gate | both instances deployed_version = serving_version = unstable-795f1dab…, status running / sleeping, container_health.errors: 0, healthy: 3 / 2 (rollout_active: false) |
checkpoint-park (dev) | 200 parked: true, backupId: 63f79239-9d43-4824-96b0-83f37e34b71e, sessionsPatched: 0 (no live sessions) |
resume (dev) | 200 resumed: true |
backup after resume | 200 (ok: true) — proves the park guard was released (a parked tenant answers 409 TENANT_PARKED); the dispatch projection only echoes backupId for park/freeze |
| Recovery point (R2) | sandbox-migration/client-sandbox-v1/stable.json → recordId 1a169e07-cb50-444c-8ae5-5976bfe72a20, committedAt 2026-11-07T08:50:12Z (written by the drills) |
| Lerma park/resume evidence | Pre-replacement checkpoint parked for tenant-client-sandbox-v1-dev … parked=true (08:46:35Z) and Resume after container replacement … resumed=true (08:48:39Z) |
| Idle cadence | unit-verified, live observation pending the next hourly tick (dev cadence is capped at 1 h); it is not waited on |
Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (3974 passed, 8 skipped), cctr apps/fouria/test/business-rules/ (582/582), scripts/staging-validation.test.ts
(83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, and
the full Lerma suite (1242 passed, 4 excluded).
CDP bridge merge fix + HEAD redeploy (2026-11-07)
The develop merge (806a2580) brought the CDP bridge (#563) in, but the merge
resolution left cdp.internal out of the shared outbound host map: the
v1-side refactor had moved TenantSandbox.outboundByHost into
src/sandbox-outbound.ts, so handleCdpRequest existed while no route
referenced it. Two apps/fouria/src/index.test.ts node-pool suites failed on the
merged tree (4020 passed, 8 skipped, 2 failed) — format:check, lint and
typecheck stayed green because the map is generic, so this was a
merge-resolution regression rather than a type error, and because both runtimes
consume the same map v1 was missing the intercept too.
| Commit | Slice |
|---|---|
e56ac9c8 | cdp.internal joins the shared host map. fix(fouria): register the cdp.internal bridge on the shared outbound host map adds [CDP_INTERNAL_HOST]: handleCdpRequest to SANDBOX_OUTBOUND_ROUTES (6 hosts: releases/docs/leads/software.r2.internal, otel.internal, cdp.internal), drops the bridge imports left dead in src/index.ts, and pins the host in sandbox-outbound.test.ts / sandbox-v1.test.ts; the two node-pool suites now stub @cloudflare/sandbox-legacy, which the CDP bridge pulls in transitively. |
Deployment
- Bundle + both runtime images:
unstable-e56ac9c8d79f062ee6b66bfed1f221e1e82f102f(fouria:unstable-…+fouria:v1-unstable-…onregistry.cloudflare.com, R2fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}). - Staging Lerma image:
ghcr.io/0xcab0/lerma:v1-sandbox-e56ac9c8, built from the repo root and shipped bydocker save | ssh … docker load(no ghcr push), compose backup/opt/lerma/docker-compose.yml.bak-pre-e56ac9c8, containerlerma-prehealthyand/health200. - One promotion for everything:
Lerma accepted unstable-e56ac9c8…: 2 running instance(s)→client-sandbox-v1 is running unstable-e56ac9c8…. The bundle-only path parked both instances (v1-dev20:02:50Z,v120:07:39Z), replaced the container and resumed each (v1-dev20:07:16Z,v120:11:01Z).
Live drills (2026-11-07, unstable-e56ac9c8…)
| Drill | Result |
|---|---|
| Status gate | both instances deployed_version = serving_version = unstable-e56ac9c8…, status running, container_health.errors: 0, healthy: 2 / 3, rollout_active: false |
checkpoint-park (dev) | 200 parked: true, backupId 8f4e88bc-368a-48c5-888e-30317895f6f8, sessionsPatched: 0 (no live sessions) |
resume (dev) | 200 resumed: true |
backup after resume | 200 (ok: true) — the park guard is released (a parked tenant answers 409 TENANT_PARKED) |
| Recovery point (R2) | sandbox-migration/client-sandbox-v1/stable.json → recordId 0e426f0e-577b-40a1-a4a6-c0abcafaa20e, committedAt 2026-11-07T20:12:55Z, trigger: "backup" (the park’s checkpoint record is 8f4e88bc-…) |
cdp.internal (deployed) | both tenant scripts in fouria-tenants-staging (10,767,930 bytes, same content) contain cdp.internal, and SANDBOX_OUTBOUND_ROUTES lists all 6 hosts including [CDP_INTERNAL_HOST]: handleCdpRequest |
| Lerma park/resume evidence | Pre-replacement checkpoint parked for tenant-client-sandbox-v1-dev … parked=true (20:02:50Z) / Resume … resumed=true (20:07:16Z); same pair for tenant-client-sandbox-v1 (20:07:39Z / 20:11:01Z) |
| Idle cadence | still [suite] only — not waited on for this deploy (dev cadence is capped at 1 h) |
What is still unmeasured. The cdp.internal registration is now observed in
the deployed bundle on both runtimes; the end-to-end browser round-trip (a
converted tenant driving Browser Rendering through the container’s
cdp-proxy.mjs) was not exercised — test/e2e/r_browser_remote_profile.txt
is cloud-profile only and was deliberately not run.
Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4022 passed, 8 skipped, 199 files), cctr apps/fouria/test/business-rules/ (593/593), scripts/staging-validation.test.ts
(83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, and
the full Lerma suite (1259 passed, 4 excluded).
Operator surface: Lerma migration + ROOT debug tools, and the already_converted signal (2026-11-07)
Two slices that make a conversion reachable by an operator instead of by a hand-rolled control call, plus the correctness fix that came out of validating them live. Nothing here converts a tenant.
| Commit | Slice |
|---|---|
c01d00b1 | ROOT debug tools work on Sandbox v1. Every /debug/* route was audited against the client a converted tenant actually serves with (V1SandboxClient): a shared readFinishedProcessLogs() (legacy logs() → v1 output()) fixes /processes?logs=true and /logs?id= for a finished process; /gateway-logs returns path + runtime; /restart-openclaw gained a v1 branch (no legacy checkpoint, no restore-needed); /reprovision checkpoints with v1CreateSnapshot and validates restorability with the C8 verifyRecoveryPoint; /backup/:mode snapshots through the DO; /destroy-container clears the startup cache and the isolate start dedupe; /env reports sandbox_runtime_actual; /debug/sleep answers 503 SANDBOX_V1_SLEEP_UNSUPPORTED (SDK 1.0 has no stop primitive). Suite-verified only (see the drill table). |
804c512e | Lerma can migrate a legacy tenant. Containers.migrate/2 posts the migrate control action with a 30-minute budget and normalises {state, record_id, duration_ms}; SandboxV1Migration computes eligibility (staging platform, never the reserved client slug, a client’s dev environment only, active instance, committed legacy recovery point) and the dry-run report; POST /api/provisioning/migrate/:slug serves ?dry_run=1 plus real runs (200/403/409/502); mix lerma.sandbox_v1.migrate prints the same table and requires --yes off a TTY. |
6602f1fc | The dry-run stops calling an already-converted tenant eligible. A conversion leaves the legacy archive in R2 (it is the conversion’s source), so the legacy recovery point alone cannot tell: Containers.v1_recovery_point/1 reads the v1 stable pointer (sandbox-migration/<canonical tenant>/stable.json, in the same per-instance bucket the legacy recovery point resolves), R2.object_body/2 supplies the record id, and SandboxV1Migration emits already_converted (true/false/nil when unreadable) + v1_record_id per instance and at the top level, as both the blocking reason and a warning. A real run on a converted tenant answers 200 {state: "committed", already: true, record_id} without a control call, so a no-op never occupies the 30-minute request path. |
Deployment (2026-11-07)
- Bundle + both runtime images:
unstable-6602f1fc98fc41692678ea05718adde8505f6b56(fouria:unstable-…+fouria:v1-unstable-…onregistry.cloudflare.com, R2fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}). - Staging Lerma image:
ghcr.io/0xcab0/lerma:v1-sandbox-6602f1fc, built from the repo root and shipped bydocker save | ssh … docker load(no ghcr push); compose backup/opt/lerma/docker-compose.yml.bak-pre-6602f1fc;lerma-prehealthyand/health200. - One promotion for everything:
Lerma accepted unstable-6602f1fc…: 2 running instance(s)→client-sandbox-v1 is running unstable-6602f1fc…. The park checkpoints were556d5773-…(dev, parked 21:34:57Z → resumed 21:40:02Z) and5eb28f82-…(production-env, parked 21:40:31Z → resumed 21:44:10Z). - Status gate afterwards: both instances
status: running,deployed_version = serving_version = unstable-6602f1fc…,container_health.healthy3 (dev) / 2, zero errors.
Idle cadence — first live tick (observed, not injected)
sandbox-migration/client-sandbox-v1/stable.json advanced to
4437c278-baf2-4ec2-af7f-fa9598f5eef7 at 2026-11-07T21:18:55.381Z with
records/4437c278-….json recording trigger: "backup" — a scheduled v1
snapshot committed with no operator action and no promotion in flight (the
trigger: "checkpoint" records are the promotion parks). This is the live tick
the earlier entries listed as pending; the ~1 h dev interval matches
resolveSnapshotIntervalMs.
Live drills (2026-11-07, unstable-6602f1fc… + v1-sandbox-6602f1fc)
All requests used the provisioning application key with
X-FourIA-Application: fouria; the key is never printed.
| Drill | Result |
|---|---|
Dry-run, converted tenant (client-sandbox-v1) | 200, eligible: false, reasons: ["already_converted"], warnings: ["already_converted"], top-level already_converted: true, v1_record_id 556d5773-5ea2-4fff-b777-3221234dcaf1. Per instance: tenant-client-sandbox-v1-dev already_converted: true; tenant-client-sandbox-v1 already_converted: false + production_environment |
| Dry-run, production-environment instance | ?instance=tenant-client-sandbox-v1 → eligible: false, reasons: ["production_environment"], already_converted: false (its own bucket has no pointer — it is still legacy), warning checkpoint_recommended_before_conversion |
Dry-run, reserved slug (client) | 409, code: MIGRATION_NOT_ELIGIBLE, reasons: ["reserved_client"], no control call |
Dry-run, non-existent client (client-sandbox) | 404 — the slug briefed for the negative gate does not exist on staging; the production-environment refusal was recorded on client-sandbox-v1’s own production instance instead |
Idempotency drill (real run, no dry_run) | 200 in 1.16 s: {state: "committed", already: true, record_id: 556d5773-…, duration_ms: 0} — no control call, so no 30-minute no-op path |
| Recovery point unchanged by that run | stable.json 556d5773-… @ 21:34:56.087Z before and after; conversion.json still 77cadb66-cc11-4561-a6b5-edc1f1a52248 @ 2026-11-05T12:04:26Z (no new conversion record) |
| Tenant serving after the promotion | GET /api/health on staging-client-sandbox-v1.fouria.io and …-dev.fouria.io (health-monitor service token): 200, status: running, startup.ok + readiness.ok true, container/durableObject/r2/aiGateway up |
| ROOT debug tools | Not verified live: GET /debug/version and /debug/gateway-logs on the tenant host answer 302 to paso4.cloudflareaccess.com (service_token_status: false), so a headless call cannot reach them. Unchanged from the earlier entries — suite-verified only |
What this bundle did not do. No tenant was converted. The production-env
instance of client-sandbox-v1 is still legacy (already_converted: false), and
policy keeps it that way: only a client’s dev environment tenant is eligible
(contract C2/S7). No migrate control action was sent during these drills.
Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4037 passed, 8 skipped, 199 files), cctr apps/fouria/test/business-rules/ (596/596), scripts/staging-validation.test.ts
(83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, the
full Lerma suite (1305 passed, 4 excluded) and workers/fouria-dispatch
--project unit (127 passed).
Durable migration job + read-only status, live-validated (2026-11-08)
The migrate control action stopped being a synchronous conversion. It is now a
queue-only enqueue of a durable job on the tenant’s SandboxV1 Durable
Object, and polling moved to a separate read-only action so that asking for
the state can never convert a tenant. Nothing here converts a tenant: the
converted instance in this drill was converted on 2026-11-05.
| Commit | Slice |
|---|---|
be255fdd | The conversion runs as a durable Durable Object job. migrate calls v1RequestConversion() and answers immediately (202 queued/running, 200 committed + recordId, 502 failed, 503 SANDBOX_V1_UNAVAILABLE when the binding/DO is unreachable — nothing recorded, safe to retry). The conversion itself streams the legacy squashfs, extracts it in the v1 container and uploads a DirectoryBackup on the DO alarm() (conversion first, cadence second, each wrapped), is durable (migration:convert-request marker + migration:convert-status + 1-based attempt) and idempotent, alarm() orders running → committed before dropping the marker and before the launch (a launch failure never retracts the commit), a failure records failed with a bounded message instead of throwing, ensureConversionArmed() re-arms a lost alarm from conversionState()/requestConversion()/status()/ensureGateway() without waking the container, and #conversionRun is the single owner so overlapping alarm turns cannot start a second destructive conversion. v1ConvertNow was removed from the DO, the RPC surface and V1SandboxClient. |
92a21080 | migration-status is a read-only action. X-FourIA-Control: migration-status calls only v1ConversionState() — never v1RequestConversion() — so it cannot enqueue, start work or wake the container; a legacy tenant answers 200 {state: "absent"} without addressing a v1 DO. It never answers 202. The Dispatch Worker forwards it (CONTROL_ACTIONS, control echo of state/recordId, no snapshot CPU budget). Lerma’s Containers.migration_status/2 normalises absent/queued/running/committed/failed. a3b3e162 is the Lerma half (enqueue + poll: GET /api/provisioning/migrate/:slug reads status, the Mix task enqueues and then polls through migration-status, and the old edge/524 caveat no longer applies to the trigger). Its commit message reads feat(fouria): … because two sessions raced; the content is the Lerma change. |
The alarm-driven contract is the reason the trigger no longer needs a long
request budget: POST returns in ~1 s, and the operator polls the read-only
action with a budget they own (:migrate_wait_timeout_ms, 30 min).
Deployment (2026-11-08, one combined rollout)
- Bundle + both runtime images:
unstable-92a21080233dca606cc4e6f91a99eb22397968cf(fouria:unstable-…@sha256:7d984068b1269b4a2ed32c2cdd6dd7afdb5b48ec68fd9f4f3ff48b0ed2dd7801andfouria:v1-unstable-…onregistry.cloudflare.com, R2fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}). - Dispatch Worker (staging) redeployed — required by the contract and easy to
miss: without it the tenant-facing action exists but
/_control/:slug/migration-statusanswers400 Unknown control action(observed before the deploy), so a LermaGETthat reaches the control plane can only fail. Deployed from this branch (bun run --cwd workers/fouria-dispatch deploy:staging, version8da2700f-f886-4d68-a923-8022c947953d);workers/fouria-dispatchhas no develop-only commits, so the deploy is an additive superset. - Staging Lerma image:
ghcr.io/0xcab0/lerma:v1-sandbox-92a21080, built from the repo root and shipped bydocker save | ssh … docker load(no ghcr push); compose backup/opt/lerma/docker-compose.yml.bak-pre-92a21080;lerma-prehealthy,/health200. - One promotion:
Lerma accepted unstable-92a21080…: 2 running instance(s)→client-sandbox-v1 is running unstable-92a21080…(both instances rolled with the same image + the digest-pinnedSandboxV1sandbox image).
Live validation of the async contract (2026-11-08)
Provisioning application key + X-FourIA-Application: fouria; the key is never
printed. Read-only calls only — no POST on a legacy tenant.
| Drill | Result |
|---|---|
GET converted client (client-sandbox-v1) | 200 in 1.59 s: {ok: true, state: "committed", already: true, record_id: "19753f1a-2554-4f39-b021-73602719af06", reasons: ["already_converted"], v1_record_id: "19753f1a-…", duration_ms: 0} — the already_converted short-circuit, so no control call at all (Lerma logs show only the two pointer GETs) |
POST converted client (no dry_run) | 200 in 1.58 s: identical body (state: committed, already: true, record_id: 19753f1a-…, duration_ms: 0) — no enqueue and no control call |
GET reserved slug (client) | 409, code: MIGRATION_NOT_ELIGIBLE, reasons: ["reserved_client"], no control call |
GET client with no instances (tarjetero) | 409, code: MIGRATION_NOT_ELIGIBLE, reasons: ["no_active_instance"] — no control call |
Read-only action, converted tenant (deployed Lerma → dispatch → tenant Worker → v1ConversionState) | Containers.migration_status(%{cf_worker_name: "tenant-client-sandbox-v1-dev"}) → {:ok, %{state: :committed, record_id: "77cadb66-cc11-4561-a6b5-edc1f1a52248", duration_ms: 355}} (the conversion record, not the pointer) |
Read-only action, legacy tenant on the new bundle (tenant-client-sandbox-v1) | {:ok, %{state: :absent, record_id: nil, duration_ms: 282}} — the legacy branch answers absent without addressing a v1 DO, and nothing was enqueued |
Same action through the dispatch edge (POST /_control/client-sandbox-v1-dev/migration-status) | 200 {ok: true, status: 200, state: "committed", recordId: "77cadb66-…"}; …/client-sandbox-v1/migration-status → 200 {state: "absent"} |
| No conversion started by any of the above | sandbox-migration/client-sandbox-v1/stable.json identical before and after ({"recordId":"19753f1a-…","committedAt":"2026-11-08T12:20:51.098Z"}); conversion.json identical (77cadb66-…, createdAt 2026-11-05T12:04:26.260Z); no new records/<id>.json; Lerma logs show no migration queued/committed/failed line for these calls |
Park → resume → backup (client-sandbox-v1-dev, v1) | 200 / 200 / 200: checkpoint-park {backupId: 6b1f7d89-9e5b-459a-87b1-a1068deadc6f, parked: true, sessionsPatched: 0, sessionsDrained: false} (5.5 s), resume {resumed: true} (1.6 s), backup (4.5 s) |
| Recovery point after the backup | stable.json advanced to 54eac3a0-d489-441e-ac99-8f65e96af4d9 @ 2026-11-08T12:33:53.317Z; retention pruned to 3 records (19753f1a-…, 6b1f7d89-…, 54eac3a0-…) |
/api/health on the converted dev tenant | 200, status: running, startup.ok + readiness.ok true, container/durableObject/r2/aiGateway up, 1.35 s |
/api/health on the production-env legacy tenant | 503, status: stopped, components.container: sleeping — the idle-slept legacy container is not woken by a headless probe (by design); POST /api/warmup with the health-monitor token answers 302 (host Access app), so a wake needs an operator browser session. Same state the earlier entries recorded as sleeping |
Status gate (GET /api/provisioning/upgrade/client-sandbox-v1) | both instances status: running, deployed_version = serving_version = unstable-92a21080…, container_health.errors: 0, healthy 2 (production-env) / 3 (dev), rollout_active: false |
What could not be exercised live. The Worker’s 202 {state: "queued"}
enqueue path (and the migration-status reads queued/running) require an
eligible legacy tenant: on staging the only legacy client is the reserved
client slug (409 by contract C2) and tarjetero has no instances, so a real
enqueue would have meant converting a tenant — explicitly out of scope. That
path is covered by src/index.test.ts (202 queued/running, 200 committed, 502
failed, 503 unreachable), src/sandbox-v1.test.ts (queue/arm, alarm commit,
cadence-after-conversion, overlapping alarms, lost-alarm re-arm) and
workers/fouria-dispatch/test/unit/control.test.ts (status forwarding + echo).
Two follow-ups worth knowing: (1) a tenant still on a pre-change bundle
answers 400 Unknown control action for migration-status (observed on
tenant-client-dev) — the status action only exists on instances promoted with
this bundle, which is also the bundle that makes migrate durable; (2) the
instance image_tag column still shows the image recorded at provisioning
(unstable-e7c70d3e…) while deployed_version/serving_version and the
container’s actual image are unstable-92a21080… — a pre-existing field that
no path in this change writes.
Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4050 passed, 8 skipped, 199 files), cctr apps/fouria/test/business-rules/ (600/600), scripts/staging-validation.test.ts
(83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, the
full Lerma suite (1327 passed, 4 excluded) and workers/fouria-dispatch
--project unit (129 passed). The dispatch integration project fails 11/27
with asyncLocalStorage.disable() is not implemented in this environment
(Node 22.23.2 / workerd); it is environmental and predates the change.
Provisioning a tenant directly on Sandbox v1 (“born v1”), live-validated (2026-11-08)
A tenant no longer has to be converted: a fresh instance can be provisioned
straight onto the SandboxV1 runtime. The only durable evidence of that choice is
the runtime marker Lerma writes before the script deploy, which the Worker reads
(tri-state, fail-closed) instead of the conversion intent.
| Commit | Slice |
|---|---|
99ed1d0d | Worker contract. src/sandbox-migration/runtime-marker.ts (strict runtime.json shape + tri-state read, the marker’s tenant must match the scope it was read from), resolveRuntimeIntent reads the conversion intent and the marker in parallel (intent wins; present marker → v1; unreadable → unknown; marker absent and SANDBOX_RUNTIME=v1 → pending, so a half-provisioned tenant never boots legacy), /control migrate on a fresh tenant answers 409 MIGRATION_ALREADY_V1 without addressing the DO (503 SANDBOX_V1_RUNTIME_UNKNOWN for an unreadable marker) and migration-status answers absent without touching the DO. |
f73b3fab | First recovery point after the boot. v1EnsureGateway dispatches a detached, best-effort runInitialSnapshot() after a successful boot (owner run and the adopted-gateway path) so a born-v1 tenant is not loss-bounded by the first idle-cadence tick. Only for a tenant with no recovery point (verifyRecoveryPoint = pointer-absent and no conversion.json); single-flight + a shared takeOwnedSnapshot() with the cadence; a failure logs sandbox.v1.initial_snapshot_failed and retries on a later boot (10-minute floor) without failing readiness. |
35e37628 | Lerma side. instances.sandbox_runtime (legacy default | v1, dev-only and staging-only, immutable after creation), SANDBOX_RUNTIME on both the provisioning and re-deploy binding paths (SANDBOX_V1_CUTOVER=true, SandboxV1 DO binding) and the :sandbox_runtime_marker provisioning step that writes sandbox-migration/<canonical-slug>/runtime.json create-only, immediately before :dispatch_deploy (idempotent :existing, refuses an existing conversion intent, deletes the marker on rollback only when that run created it). The migration preflight reports already_v1: true / reason sandbox_runtime_v1 with no control call. |
82bc961e | Docs-only (ENV-VARIABLES table reformat); deployed as the bundle tag so the promoted artifact matches the pushed tree exactly. |
Deployment (2026-11-08)
- Bundle + both images:
unstable-82bc961e95022541bd5abc688293c5e5219b14a9(fouria:unstable-…@sha256:371bcb013fb577f9c07ce0f7ccbbf6edacabaaf109dd86c0a104b630a3e6e400andfouria:v1-unstable-…@sha256:37e48974e2d0c6d39d370f54986dbd02461e60eb651b0a7d38de238e8cadc303, R2fouria-bundles/unstable-…/{fouria.mjs,container-manifest.json}). - Staging Lerma:
ghcr.io/0xcab0/lerma:v1-sandbox-82bc961e(same layers as the35e37628build; the tag commit is docs-only and no file it changes is copied into the image), shipped bydocker save | ssh … docker load, compose backup/opt/lerma/docker-compose.yml.bak-pre-82bc961e;lerma-prehealthy,/health200, migration20261108120000applied at boot. - Dispatch (staging) not redeployed: it already forwards
migration-status(POST /_control/client-sandbox-v1-dev/migration-statusanswered200 {state: "committed", recordId: "77cadb66-…"}before the rollout), andworkers/fouria-dispatchhas no develop-only divergence. - One promotion:
Lerma accepted unstable-82bc961e…: 2 running instance(s)→client-sandbox-v1 is running unstable-82bc961e…(both instances, same image + the digest-pinnedSandboxV1sandbox image).
Live born-v1 drill (2026-11-08)
A throwaway staging client (select-runtime-smoke-82bc, id 63) was provisioned
through POST /api/clients/63/instances with
{"instance": {"environments": ["dev"], "sandbox_runtime": "v1", "version": "unstable-82bc961e…"}}
(provisioning application key + X-FourIA-Application: fouria; the key is never
printed). 202 {run_id, sandbox_runtime: "v1"}.
| Check | Result |
|---|---|
| Step order | provisioning_step_events: … fetch_bundle → telegram_ingress_queue → sandbox_runtime_marker (succeeded) → dispatch_deploy (succeeded) → … — the marker lands before the script deploy |
| Instance row | sandbox_runtime: "v1", environment: "dev", status: running, deployed_version: unstable-82bc961e… |
| Marker | fouria-backup-select-runtime-smoke-82bc-dev-staging/sandbox-migration/select-runtime-smoke-82bc/runtime.json = {"mode":"fresh","provisionedAt":"2026-11-08T19:37:44Z","runtime":"v1","tenant":"select-runtime-smoke-82bc","version":1}; the retried provisioning logged v1 marker existing (create-only, provisionedAt unchanged) |
| Deployed bindings | SANDBOX_RUNTIME=v1, SANDBOX_V1_CUTOVER=true, SandboxV1 (durable_object_namespace, class SandboxV1), WORKER_PLATFORM=staging (Cloudflare API, dispatch-namespace script settings) |
| Container | tenant-select-runtime-smoke-82bc-dev-sandbox-v1-staging (scheduling_policy: durable_object) started an instance in cdg14 with the digest-pinned v1 image fouria@sha256:37e48974… |
/api/health | 200, status: running, startup/readiness/liveness success, components: container/durableObject/r2/aiGateway all up, restoreStatus {hadBackup: false, restored: false} (fresh onboarding, no OPENCLAW_RESTORED) |
migration-status | POST https://dev.fouria.io/_control/select-runtime-smoke-82bc-dev/migration-status → 200 {"state":"absent"} (no conversion job; the read does not touch the DO) |
migrate | POST …/migrate → 409 {"code":"MIGRATION_ALREADY_V1","tenantError":"tenant is provisioned on SandboxV1; there is nothing to convert"} |
| Initial snapshot | sandbox-migration/select-runtime-smoke-82bc/stable.json → {"recordId":"c45449dc-ce29-4999-b7e2-471f3f4d8ce1","committedAt":"2026-11-08T20:09:01.264Z"} — 4 s after readiness (20:08:57), no operator action; records/c45449dc-….json (dir /home/openclaw, size 7 948 131, format tar+zstd/1, trigger "backup") and the encrypted payload sandbox-v1/openclaw/c45449dc-….tar.zst.enc (7 948 180 B) + .meta.json (254 B); no plaintext archive in the bucket |
| Operator snapshot | POST /_control/select-runtime-smoke-82bc-dev/backup → 200 in 4.0 s; pointer advanced to 0038d1e6-3c86-4c9a-8024-52579fbc84ff @ 20:10:10.196Z (two records retained) |
| Lerma preflight | GET /api/provisioning/migrate/select-runtime-smoke-82bc → 200 {already_v1: true, runtime: "v1", reasons: ["sandbox_runtime_v1","no_legacy_recovery_point","already_converted"], duration_ms: 0} — no control call |
| No conversion | conversion.json is absent from the bucket; the tenant was born on v1 and never converted |
One platform wedge, found and recovered, worth knowing. The first provision
(19:37) produced a SandboxV1 container application with zero instances even
though the DO reported running: true: containerFetch timed out on every health
probe and /control …/backup hung until the client gave up, so the gateway never
launched and no initial snapshot could happen. The container image itself was
fine (it runs locally and the same tag boots elsewhere). DELETE /api/instances/:id + a re-provision (same client, same marker) recreated the app
and the instance started in cdg14 within 30 s; the gateway reached readiness
~5 min later. The retry re-used the create-only marker (v1 marker existing),
which is also the idempotency proof. Treat a born-v1 instance whose container app
stays at zero instances as an operator action: deprovision and re-provision; the
marker survives, so the tenant stays v1.
Corroboration from an earlier run of the same code: arua was provisioned born
v1 at 19:05:59 (runtime.json mode: "fresh", migration-status absent, a
running tenant-arua-dev-sandbox-v1-staging instance) and has been committing v1
records on its snapshot cadence (stable.json advanced through 19:31 / 19:46 /
20:01, all trigger "backup"), so the runtime decision, serving and persistence
also hold outside this drill.
Cleanup (2026-11-08)
The throwaway client, its instance, the tenant Worker script and both container
applications were removed (DELETE /api/instances/:id → 204, which deprovisions
the script and both container applications; DELETE /api/clients/63 → 204,
which decommissions the client — the row stays in the archived state, the
platform’s normal soft delete, and is no longer listed as active). No
select-runtime-smoke script or container application remains. Its R2 bucket was emptied (11 objects deleted, including
the marker and the recovery point) but the bucket itself is still refused by R2
as “not empty” because the initial snapshot left an in-progress multipart upload;
R2 aborts those automatically, after which the empty bucket can be deleted. No
existing tenant was converted and no production instance was touched.
Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4077 passed, 8 skipped, 199 files), cctr apps/fouria/test/business-rules/ (606/606), scripts/staging-validation.test.ts
(83/83), bun run plan:sandbox-v1:check, mix check, mix license_audit, the
full Lerma suite (1373 passed, 4 excluded) and workers/fouria-dispatch --project unit (129 passed).
Wake-lifecycle merge into v1-sandbox and the combined deployment (2026-11-08)
Staging had been serving unstable-c8aae8cd…, a build of the pushed feature
branch fix/sandbox-v1-wake-lifecycle. The born-v1 promotion
(unstable-82bc961e…) superseded it, so the two lines were combined on
v1-sandbox instead of leaving the staging tenant on the older build: the wake
lifecycle (keep-alive renewal, the provisioned instance type on wake, the
no-empty-recovery-point skip, and the durable automation wake) is part of the
migration track now, not a sibling branch.
Merge
git merge --no-ff origin/fix/sandbox-v1-wake-lifecycle into v1-sandbox
(merge commit cd854394, first parent 6fac3e42). The branch had already
merged this track at 8ce6a547 (based on 99ed1d0d), so its content replayed
cleanly: apps/fouria/src/sandbox-v1.ts auto-merged and the two features
compose there — the cadence turn routes through takeOwnedSnapshot (born-v1)
while the gateway-owner promise schedules the initial snapshot and the restart
path re-arms the keep-alive (wake lifecycle).
Two conflicts, both in docs, resolved preserving both sides:
| File | Resolution |
|---|---|
apps/fouria/AGENTS.md | kept the wake-lifecycle recent-changes bullet and the newer born-v1 bullet (the one that also documents the first recovery point taken after the first boot) |
docs/ENV-VARIABLES.md | kept the reformatted table, merged the wake-corrected SANDBOX_RUNTIME text (keep-alive renewal, no snapshot wake, production 6h cap), the production 3h cadence cap and added the CONTAINER_INSTANCE_TYPE row |
No code conflict needed a judgement call, so no resolution was guessed.
Deployment
- Bundle + both runtime images:
unstable-cd8543948638f27e98081fb47b3312d5dad15acd(fouria:unstable-…@sha256:371bcb01…,fouria:v1-unstable-…@sha256:37e48974…, R2fouria-bundles/unstable-cd854394…/{fouria.mjs,container-manifest.json}). - The published bundle carries both halves (grep of the R2
fouria.mjs):debug/lifecycle,keepalive-pending,wokenFor,CONTAINER_INSTANCE_TYPE(wake lifecycle) andruntime-marker,MIGRATION_ALREADY_V1(born v1). - Staging Lerma:
ghcr.io/0xcab0/lerma:v1-sandbox-cd854394, built from the repo root and shipped bydocker save | ssh … docker load, compose backup/opt/lerma/docker-compose.yml.bak-pre-cd854394, retargeted fromv1-sandbox-82bc961e;lerma-prehealthy, in-container/health200. - One promotion:
Lerma accepted unstable-cd854394…: 2 running instance(s)→client-sandbox-v1 is running unstable-cd854394….
Live verification (2026-11-08)
| Check | Result |
|---|---|
| Status gate | both instances deployed_version = serving_version = unstable-cd854394…, status: running, container_health.errors: 0, healthy: 2 / 3, rollout_active: false |
checkpoint-park (dev) | 200 parked: true, backupId 05aa5c01-b8dd-42f1-bbf9-4946b7f813d0, sessionsPatched: 0 (no live sessions) |
resume (dev) | 200 resumed: true |
backup after resume | 200 (ok: true) — proves the park guard was released (a parked tenant answers 409 TENANT_PARKED) |
| Recovery point (R2) | sandbox-migration/client-sandbox-v1/stable.json → recordId 19fa77db-87f1-47de-9771-04e1cbcad4cb, committedAt 2026-11-08T20:36:17Z (12 s before the check) |
migration-status (Lerma) | GET /api/provisioning/migrate/client-sandbox-v1 → 200 {state: "committed", already: true, record_id: 19fa77db…} (read-only; the non-dev instance still reports sandbox_runtime: legacy) |
/api/health (both hosts) | 200 on staging-client-sandbox-v1.fouria.io and staging-client-sandbox-v1-dev.fouria.io; status: running, startup/readiness/liveness success, container/durableObject/r2/aiGateway all up |
| Drill client decommissioned | no select-runtime-smoke script in fouria-tenants-staging (5 scripts: tenant-client, tenant-client-dev, tenant-client-sandbox-v1, tenant-client-sandbox-v1-dev, tenant-arua-dev) and no matching container application (9 total) |
| Drill bucket | fouria-backup-select-runtime-smoke-82bc-dev-staging still exists and is empty, but r2 bucket delete is refused again (10008 not empty, the in-progress multipart upload from the initial snapshot). The abort needs the S3 API, which this machine cannot reach — *.r2.cloudflarestorage.com resolves to real Cloudflare IPs but the egress path presents core1.netops.test, so curl --aws-sigv4 fails TLS verification before any request. R2 auto-aborts the upload; the empty bucket is deletable afterwards. Only tenant-arua-dev (a separate born-v1 tenant) was left running. |
The deployed artifact is unstable-cd854394… (the merge commit). The docs
commits on top of it (6852699a and this note) change no file that is copied
into the bundle or either image, so unstable-cd854394… still describes exactly
what staging is serving.
No tenant was converted by this deployment: the merge and the promotion changed
the deployed image only. client-sandbox-v1 keeps sandbox_runtime: legacy
(never migrated, per the reserved-path rule) and client-sandbox-v1-dev keeps its
existing v1 recovery point; no production instance was touched.
Verification for this bundle: bun run format:check && bun run lint && bun run typecheck && bun run test (4101 passed, 8 skipped, 201 files, no type
errors), cctr apps/fouria/test/business-rules/ (614/614), scripts/staging-validation.test.ts
(83/83), bun run plan:sandbox-v1:check (166 features, 14 milestones), mix check, mix license_audit (62 deps), the full Lerma suite (1373 passed, 4
excluded) and workers/fouria-dispatch --project unit (129 passed). The live
drill suite self-skips without E2E_V1_LIVE=1 (3/3 skipped).
Rollback coverage + upgrade-pipeline repair (2026-11-09)
The OCM-4a incident’s rollback fix was one-sided. Instances.upgrade_instance/3
(the promotion lane) passed preserve_do_namespaces: true, but the
dashboard/API pipeline (Instances.update_instance_version/3 →
TenantProvisioner.upgrade_instance/3 → :dispatch_deploy) called
Containers.deploy_script_and_container/1 without it, so a failed container
binding could still delete a live tenant’s routing script exactly as in the
incident. Two commits closed it:
| Commit | Change |
|---|---|
5e3e5618 | TenantProvisioner’s :dispatch_deploy passes preserve_do_namespaces: upgrade?(meta); a fresh provision keeps the destructive rollback so a retry starts clean. |
d57bec6d | The shared Instances.deploy_script_and_container/6 takes the flag explicitly (fresh create_user_worker_instance/3 = false, upgrade / AI-gateway reassignment = true), and the upgrade test asserts the full binding set a rebuilt script needs (gateway token, derived CDP secret, AI Gateway/API token, backup key, Telegram queue, runtime). |
Deployment: unstable-d57bec6d… published; branch Lerma
v1-sandbox-d57bec6d deployed (compose backup
/opt/lerma/docker-compose.yml.bak-pre-d57bec6d); one promotion of
client-sandbox-v1 converged.
| Check | Result |
|---|---|
| Both tenant scripts | identical etag b0a58070…, modified 12:58Z / 13:02Z |
| Bindings | 33 (production-env) / 34 (dev); SANDBOX_RUNTIME=legacy, SANDBOX_V1_CUTOVER=false |
| Health | HealthMonitor.check_instance_now/1 → :ok on both |
| v1 park/resume/backup (dev) | 200 / 200 / 200; recovery point advanced to ed7f0db5-… at 13:05Z |
Born-v1 tenant-arua-dev | upgraded to unstable-d57bec6d…; deployed_version = serving_version, 35 bindings, SANDBOX_RUNTIME=v1, marker intact, recovery point 8c2729a2-…, 3/3 healthy |
| Staging Lerma | v1-sandbox-d57bec6d, healthy |
Operational follow-up: the staging provisioning key was rotated (old key
401, new key 200; SECRETS.yaml and the GitHub
STAGING_PROVISIONING_API_KEY secret updated). The born-v1 tenant-arua-dev
was moved off the unreachable f1590509 merge and onto this build.
Gateway proxy attribution fix, validated live (2026-11-09)
Every dev instance on the v1 runtime served
{"error":{"message":"Proxy client attribution is required…","type":"proxy_attribution_required"}}
on the root HTML/WS proxy. Root cause: OpenClaw attributes proxy-shaped traffic
(Forwarded / X-Forwarded-* / X-Real-IP) against gateway.trustedProxies
before Gateway auth and rejects a source it cannot trust. The legacy runtime
reaches the container from the sandbox bridge (10.1.0.x, healed into
trustedProxies by moltlazy), but the v1 transport forwards through
ctx.container.getTcpPort(port) from the SandboxV1 Durable Object, whose
ingress address is platform-owned and is not a range a tenant config can list.
Fix (eddef5e0): SandboxV1.fetch() probes attribution once per gateway
incarnation (GET / with the rebuilt forwarded headers) and, when the Gateway
refuses it, strips every proxy-shaped header so OpenClaw attributes the request
as direct-remote (always accepted, same token/device auth). The verdict is
cached per DO lifetime, re-probed after v1KillGateway / v1Destroy /
startGatewayOwner, and guarded by a generation counter.
Publishing for the live drill reused the running container image (unchanged by
this commit): the unstable-d57bec6d… legacy + v1-unstable-d57bec6d… manifests
were copied to unstable-eddef5e0… / v1-unstable-eddef5e0… in
registry.cloudflare.com/35afea16440634aa2350331d2a736eec/fouria (identical
digests), and the new Worker bundle + container-manifest.json were published to
fouria-bundles/unstable-eddef5e0…/. Promotion: POST /api/provisioning/upgrade/arua {"version":"unstable-eddef5e0…"}.
| Check | Result |
|---|---|
| Promotion convergence | status: running, deployed_version = serving_version = unstable-eddef5e0…, container_health 3 healthy / 0 failed / 0 errors, rollout_active: false |
| Container | not replaced (same image digest): instance 0899aee8… running since 2026-11-08 — only the script changed |
/api/health (service token) | 200 status: running, degraded: false, startup/readiness/liveness success, gateway pid 4 / 1.39 GiB / 23 threads |
| Live gateway attribution (new) | tenant log Gateway refused proxy attribution; forwarding without forwarded client headers — the DO probe reproduced the rejection against the real Gateway |
| Proxied request through the fix | GET /api/status/attribution-live-{1,2,3} (an allowed dispatch health path that falls through to the catch-all proxy) → [PROXY] Handling request + [HTTP] Response status: 404 (the Gateway’s answer for an unknown path), not 403 proxy_attribution_required |
| Post-cleanup | tenant observability settings restored (persist: false, 10 % logs), temporary Access service token + its two app policies deleted, public host back to 302 (Access) |
Validation tooling note: staging tenants are ROOT-only at the dispatch gate, so
the public proxy path cannot be exercised with a service token. The reachable
/api/status/* prefix is allowlisted as a health path and falls through to the
gateway proxy, which made the end-to-end request observable; tenant Worker logs
were read by briefly enabling persist (they normally go only to Lerma’s OTLP
ingest) and the setting was restored immediately afterwards.
Preflight bundle verification (2026-11-09)
The known-gap “Preflight asymmetry” is closed. cfd2dda8 adds
DispatchNamespace.get_script_content/2 and makes
Instances.at_requested_version?/2 verify that the requested bundle’s main
module is embedded in the deployed script (the content endpoint returns the whole
uploaded script; the parsed main module is a verbatim substring). Both the
promotion candidate filter and the error-retry preflight use it, so a failed
script upload (image at target, bundle older) is retried; an unreadable
bundle/content probe falls back to the image so a transient failure cannot loop.
685d686a makes the endpoint report the number of instances actually marked
upgrading.
Deployment: branch Lerma v1-sandbox-685d686a (compose backup
/opt/lerma/docker-compose.yml.bak-pre-685d686a), healthy. No Worker promotion
was needed; staging keeps unstable-d57bec6d….
| Check | Result |
|---|---|
| Deployed script content | 10,875,960 bytes; contains the parsed main module (2,855,594 bytes) and the d57bec6d commit marker |
| Re-promotion preflight | POST /api/provisioning/upgrade/client-sandbox-v1 (version unstable-d57bec6d…) → 202 {instance_count: 0}; both instances stayed running at d57bec6d; Lerma fetched the bundle once per instance and did not deploy |
| Negative path (bundle older) | Suite: at_requested_version?/2 image/bundle matrix + controller retry test; live failure injection remains the next step |
Born-v1 first-provision self-heal (2026-11-11)
51df6b0a closes the zero-instance first-provision failure recorded above
(2026-11-08: the SandboxV1 application existed with zero instances, the DO
reported running, containerFetch timed out and the gateway never booted; the
only recovery was a manual deprovision + re-provision).
Containers.recreate_v1_application/1deletes and recreates the v1 application against the same DO namespace (bounded create retry). The deployed script and its Durable Object state are untouched.TenantProvisioner.verify_health/3takes that action only for a fresh provision (reconcile_v1) with no committed recovery point whose v1 application reports no instances (or is missing), then retries the probe once. An upgrade never opts in and a tenant with a pointer is never reconciled.
Deployment: branch Lerma v1-sandbox-51df6b0a (compose backup
/opt/lerma/docker-compose.yml.bak-pre-51df6b0a), healthy; no migrations. All
five staging instances remain running (tenant-arua-dev on
unstable-eddef5e0…, both client-sandbox-v1 tenants on
unstable-d57bec6d…, tenant-client{,-dev} on unstable-73e820e6…).
| Check | Result |
|---|---|
| Module loaded | function_exported?(Containers, :recreate_v1_application, 1) → true; SandboxV1Status and TenantProvisioner loaded |
| Tests | Targeted containers_test + tenant_provisioner_pipeline_test: 148 passed; full Lerma suite: 1404 passed / 4 excluded; mix format --check-formatted + mix credo --strict clean |
| Live wedge drill | Not run — it needs a disposable born-v1 tenant on this image. tenant-arua-dev is ineligible: it has a recovery point, and the reconcile deliberately refuses those |
Born-v1 self-heal: live drill attempt and the active fix (2026-11-11)
Live drill attempt on a disposable born-v1 tenant (bornv1-heal-7fb503, client
64, dev, unstable-eddef5e0…; deprovisioned and deleted afterwards). Two
findings, one fixed and one still open.
1. Fixed (b0a22c02) — a durable_object application reports active, not
healthy. The live health payload for a v1 application is
{"instances": {"active": 1, "starting": 0}}; there is no healthy key.
Containers.fetch_application_health/1 read healthy, so a live v1 tenant
looked instance-less: the promotion preflight would re-deploy it on every
promotion, and the new born-v1 reconcile would have recreated a running
application. The v1 summary now maps active into healthy (falling back to
healthy). Live proof: GET /api/provisioning/upgrade/arua reports
container_health {runtime: v1, healthy: 1, rollout_active: false}, and
re-promoting arua at its current version returns 202 {instance_count: 0} —
before the fix the preflight retried it.
2. The zero-instance wedge could not be reproduced by deleting the app
record. The v1 application was created at 00:32:25 and deleted at
00:32:29 (API 200, absent from the account list). The first health probe
still started a container, the gateway booted, and the initial snapshot
committed at 00:35:02. The app record is therefore not the only thing that
gates a placement (or its deletion is not synchronous), so recreating only the
application may not be sufficient for the original transient — the documented
operator workaround also re-deployed the script. The async health task was also
lost to an unrelated Lerma BEAM restart at 00:36:25 (the container did not
restart; Startup health re-ran), so the reconcile path was never reached. The
drill is inconclusive, not a failure of the path.
Cleanup: DELETE /api/instances/2537773c-… → 204, DELETE /api/clients/64 →
204, the tenant script and both container applications are absent, and the R2
bucket was emptied (5 objects) and deleted.
Next step: fault-inject a placement failure (or make the provisioner re-deploy the script when the application stays empty), since deleting the app record alone does not reproduce the wedge.
Develop integration sync, production-surface removal and the local gate (2026-11-12)
Merged origin/develop (e62f5a37, Agent Studio #572) into v1-sandbox at
305d489e. Only the two business-rules corpora conflicted; both were resolved
additively (the activity/alarm/boot sections and the Agent Studio sections, each
with its own count trailer). Code and docs auto-merged (agentStudioRoutes +
activityRoutes, the Studio API client, and the Agent Studio sections in
WORKER-DEVELOPMENT.md / TESTING.md).
Decision. ADR-SV1-010 records that the track integrates into develop as one
reviewed merge commit and that integration is not a production cutover. The
production wrangler.jsonc environment no longer declares the
fouria-sandbox-v1 application or the SandboxV1 binding (019a22f1); the
class and the append-only v7 migration remain. validate-runtime-image-pins
now requires the v1 surface in local/staging and fails if production declares it
(803831fa).
Local gate. On the merged tree (Elixir files identical to 305d489e):
| Check | Result |
|---|---|
bun run format:check, lint, typecheck | pass |
bun run test | 4254 passed, 8 skipped (209 files), no type errors |
mix check, mix license_audit | pass (3630 mods/funs, 62 deps) |
mix workspace.run -t test | 1417 passed, 4 excluded |
bun run plan:sandbox-v1:check | pass (166 features / 14 milestones) |
mix precommit remains blocked by 20 pre-existing compiler warnings under
--warnings-as-errors across Lerma/Fouria; none are introduced by this merge,
and the mandatory gates above are green.
CI. On 803831fa the PR checks (unit, test, quality, build-and-push,
changes, validate-secrets, Validation lane tests) passed and PR #565 is
MERGEABLE/CLEAN. The manual Test workflow (workflow_dispatch,
run_e2e=true, full_tests=true, production) was started as run
43199464508; its observation is appended when it completes.
Staging validation after the cec359fa promotion (2026-11-12)
- Promotion:
client-sandbox-v1converged onunstable-cec359fa…at10:55:30Z(2 instances); both tenant scripts share etagb7893a73…(tenant-client-sandbox-v1modified 10:52:03,…-dev10:55:19). The legacy production application servesfouria:unstable-cec359fa…; the dev instance serves through the v1 application (v1active: 1). - v1 snapshot (PER-2):
POST /_control/client-sandbox-v1-dev/backup→200; the stable pointer advanced7beb1684…(10:55:08Z) →3b9030fa…(10:56:40Z). - SEC-3 park/resume:
checkpoint-park→{parked: true, backupId: 7e4b11c1…}; abackupwhile parked →409 TENANT_PARKED;resume→{resumed: true}; a post-resumebackup→200and the pointer advanced to7b6f95d5…(10:57:40Z). The cold boot served from the committed record. - Tenant isolation:
migration-statusforclient-sandbox-v1-dev→200 {state: "committed", recordId: "77cadb66…"}; for the production-environmentclient-sandbox-v1→200 {state: "absent"}(it is still legacy; its bucket contains onlybackups/*.sqsh). No Access token was used: the checks went through the dispatch control plane. - BIL-3 finding (cost attribution): the AE
fourai_cost_metricsrows from the v1 snapshot/park are written withblob2 = "default", notclient-sandbox-v1. The tenant script carriesWORKER_URL=https://staging-client-sandbox-v1-dev.fouria.iobut Lerma binds noCUSTOMER_SLUG/WORKER_NAME.tenantSlugFromEnvalready falls back toWORKER_URL;deriveCustomerSlug(used bygetCostContext) did not, so it returneddefault. Fixed in this PR: Lerma now binds the canonical base slug asCUSTOMER_SLUGon both fresh provision and upgrade; the cost resolver sanitizes the explicit binding with the shared slug allowlist, fails closed for a non-empty invalid value (never adoptingWORKER_NAME/WORKER_URL), and never uses a URL as the identity;isValidSummaryCustomeraccepts exactly the emitted domain so rows are never write-only. The next promotion re-deploys the scripts with the binding; the corrected AE rows are the live confirmation. - BIL-3 adversarial review (independent session): the fix was attacked
before merge, and it found a real cross-tenant hole in the pre-existing
guard:
otel-ingest.tsskipped the tenant check whenever the expected identity was thedefaultsentinel (expected !== 'default' && …), while the billable row’s customer came from the container-suppliedcustomer.slug. An unidentified Worker could therefore mint billable rows for another tenant and choose whichbilling-freeze/<slug>gate applied. The guard now falls back to the routing resolver (WORKER_NAME/WORKER_URL) and answers503 TENANT_UNRESOLVEDon a billable instance with no identity. The same review closed the consistency gaps: Lerma reservesdefaultand thetenant-dispatch prefix as client slugs,canonical_tenant_slug/1strips exactly one affix (sotenant-tenant-foono longer collapses tofoo), the shared sanitizer rejects padded values instead of trimming them, the cost summary and emitter share one slug domain, out-of-rangefrom/tovalues are clamped instead of throwing, and a loopbackWORKER_URLis a local (non-billable) instance. Covered bysandbox-otel-bridge.test.ts,metrics.test.ts,cost-summary.test.ts,dev-mode.test.ts,sandbox-tenant.test.ts, and the Lerma client/naming tests. - BIL-3 live confirmation (2026-11-16). Staging Lerma was redeployed from the
branch image
ghcr.io/0xcab0/lerma:v1-sandbox-69e2d9cc(compose backup/opt/lerma/docker-compose.yml.bak-pre-69e2d9cc) and bothclient-sandbox-v1scripts were re-uploaded at their existing bundle (unstable-cec359fa…) viaInstances.upgrade_instance/3. Both scripts now carryCUSTOMER_SLUGas aplain_textbinding with the valueclient-sandbox-v1(35 bindings on the dev script, 34 on the production-env script; previously absent). The dev instance isrunningand the legacy production-env instance issleeping;last_deployed_atadvanced to14:12:14Z/14:14:41Z. Analytics Engine (fourai_cost_metrics_staging) rows emitted after the flip are attributed toblob2 = "client-sandbox-v1"forhealth-check,gateway-memory-sampleandgateway-resources; the same row types wereblob2 = "default"before the redeploy. The remainingdefaultrows belong to the other staging tenants (client,arua), which have not been upgraded yet and pick up the binding on their next upgrade. - Cloud E2E harness repaired (2026-11-13). A fresh
wrangler deployfailed for hours with “Durable Object-managed Container application settings were not fully applied” after all five retries. Root cause: the top-level (local/E2E)wrangler.jsoncstill declares theSandboxV1application as the sharedfouria-sandbox-v1, which belongs to the production Worker, so every fresh E2E run tried to adopt it. The per-run rename is restored (<worker>-sandbox-v1), and teardown now validates the teardown token (a staleE2E_CLOUDFLARE_API_TOKENanswered9109and made every container-app delete silently no-op; three leaked apps were still billing and were deleted manually). With the per-group transcripts finally uploaded, the four legacy failures were triaged and fixed:h_cli_knowledge(the roster read raced the plugin reload → bounded retry inget-agent-id),y_production_profile(no scheduled snapshot becauseBACKUP_CHANGE_WATCHis off in E2E → on-demandPOST /debug/backup/sync),zz_gateway_hard_restart(/status/polling’s RPC fast path omitsprocessId→ read the authoritative/api/healthsnapshot, plus a post-restore boot wait; the snapshot now exposes the/procpid when the tracker misses a restore-relaunched gateway), andzzz_software_provisioning(missingCUSTOMER_SLUG→ fixture binding, now 9/9). Re-runs of every previously failing suite are green. - Not run (Access-free limitation): a v1
freezewould leave the dev tenant stopped (unfreezeis a no-op and there is no Access-free wake); the snapshot-before-destroy path is corpus-covered, and the safe SEC-3 park/resume path above proves the cold boot from the committed record. Tenant-HTTP drills (channels/RPC, browser) remain Cloudflare-Access-blocked.