Graceful shutdown and emergency checkpoints
On this page
Date: 2026-11-03
Status: Partially implemented (C1, C2, C3-policy)
Incident: tenant-tarjetero-dev, 2026-11-03 12:29 → 13:12 UTC (≈43 min outage, ≈7 h 45 m of work lost)
1. What happened (evidence)
| Time (UTC) | Fact | Source |
|---|---|---|
| 05:26 | Last successful archive 33c023e2 (450 MB) | R2 stable-backup.json |
| 12:29:40 | Last healthy gateway /health (16 ms) | Workers Observability |
| 12:33:43 | listProcesses returned 65 processes (1 running) — the only live process is a start-openclaw.sh wrapper; no gateway | Worker log |
| 12:34:35 | Every browser WS closed code=4000 inbound silence probe timeout | Worker log |
| ~12:39 → 12:42:32 | sandbox.restoreBackup timed out at RESTORE_BACKUP_TIMEOUT_MS (180 s) → restore-target.json outcome: unknown | R2 + persistence.phase |
| 12:45 → 13:11 | Every boot refused with RestoreInFlightError (restore-outcome-unknown); boot job fails; loading page loops | Worker logs |
| 13:03 | Operator containers ssh fails inside the Sandbox DO: ContainerControlConnection.fetchUpgradeAttempt → AbortError | Sandbox DO |
| 13:11:19 | Rollout to standard-3 (2 vCPU / 8 GiB / 16 GB) replaces the instance | Containers API |
| 13:14:43 | New instance answers sandbox.exec in 13 ms (was 30 s timeouts) | Container dataset |
Four independent defects combined:
- No memory signal.
attachGatewayResourcesonly samples inside/api/health*, and nothing polls health while the gateway is healthy (the Control UI talks to the gateway directly).gateway-resourcesAnalytics Engine rows stopped at 2026-11-02 19:20 — the watchdog was blind for ~17 h, including the whole incident. - No checkpoint on the way down. A 0.5 vCPU / 4 GiB instance running two heavy
agent sessions (xlsx parsing through an MCP, background
execjobs) exhausted the VM; there was no snapshot between 05:26 and the kill. - “Unknown” read as “empty”.
containerHasState()swallows every probe error and returnsfalse, so an unreachable container is indistinguishable from a fresh one and the recovery path chose a destructive restore on a starved VM. The restore then timed out, leaving anunknownlatch that blocked every later boot. - The scheduled snapshot could not run. The 12:35
backupjob was interrupted at itsgatestep, and afterwardsrestoreNotPending()refused every snapshot becauserestore-neededwas set — so the recovery path could not save the state it was about to destroy.
2. Data-loss budget (the number the policy must satisfy)
Today: 7 h 45 m lost, discovered only because the instance had to be replaced.
| Environment | Max acceptable loss | Mechanism |
|---|---|---|
| dev | 15 min | interval archive + emergency checkpoint |
| staging | 15 min | same |
| production | 5 min | same, plus Tier 1 container snapshots once the app moves to durable_object scheduling (persistent-workspace-phase1.md) |
Everything below is derived from that budget, not from a fixed cron.
3. Changes
C1 — containerHasState() must distinguish absent from unknown (highest priority)
apps/fouria/src/persistence.ts
export type StatePresence = 'present' | 'absent' | 'unknown';
async function probeContainerState(sandbox: Sandbox): Promise<StatePresence>;
present— any probe path reportsexists: true.absent— every path returned a definite{ exists: false }(no throw).unknown— any probe threw (control channel down, timeout,OperationInterruptedError).
restoreIfNeeded restores only on absent (or an explicit restore-needed
signal). On unknown it throws a retryable ContainerStateUnknownError, which the
durable boot job retries with backoff instead of running a destructive restore.
Do not write restore-needed on the unknown path.
This single change would have prevented the entire 12:39 → 13:12 loop.
C2 — Sample memory on the scheduler, not on health polls
- Move the
sampleGatewayResources()+maybeRestartGatewayForMemory()call into theActivityScheduleralarm (apps/fouria/src/activity/scheduler.ts), one sample perGATEWAY_RESOURCE_SAMPLE_INTERVAL_MS(60 s), independent of request traffic. - Keep the health-path sample as a fast path (it already throttles), but pass
executionCtxfromhealthWsHandlerso the detached watchdog promise is not cancelled when the frame is sent (routes/public.ts:719callsattachGatewayResources(snapshot, sandbox, c.env)with no context). - Record
rssBytes/threads/containerLimitByteson every sample, including when no gateway is running (resources: null+ astatelabel), so a gap in the series is visibly “not sampling” rather than “healthy”.
C3 — Two-stage memory policy: checkpoint at 60 %, replace at 75 %
New config in apps/fouria/src/config.ts:
| Key | Value | Meaning |
|---|---|---|
GATEWAY_MEMORY_CHECKPOINT_RATIO | 0.60 | ≥ this fraction of the cgroup limit → emergency checkpoint |
GATEWAY_RSS_RESTART_RATIO | 0.75 (unchanged) | ≥ this → replace the gateway |
EMERGENCY_SNAPSHOT_TIMEOUT_MS | 180_000 | bound for the checkpoint |
Order is mandatory, and it is what makes the shutdown graceful:
yellow(60 %): enqueue a durablesnapshotjob withtrigger: 'memory-pressure'; do not restart. CooldownEMERGENCY_SNAPSHOT_COOLDOWN_MS(10 min) so a tenant hovering at 60 % does not snapshot continuously.red(75 %): if the newest archive is older than the loss budget, run the checkpoint first (bounded byEMERGENCY_SNAPSHOT_TIMEOUT_MS), thenkillGateway+ensureGateway({ skipRestore: true })(today’shardRestartGateway). A failed or timed-out checkpoint must not block the restart — it is recorded aspersistence-snapshot { outcome: 'failed' }and the restart proceeds (the alternative is losing everything).
State-machine function stays pure and unit-testable:
export function decideMemoryAction(
sample,
{ limitBytes, newestArchiveAt, now },
): 'none' | 'checkpoint' | 'checkpoint-and-restart' | 'restart';C4 — The emergency checkpoint must not be gated by the restore latch
scheduleBackupIfNeeded starts with restoreNotPending(), which refused today’s
snapshot. Add an explicit intent rather than weakening the gate:
scheduleBackupIfNeeded(sandbox, bucket, env, ctx, {
trigger: 'memory-pressure',
ignoreRestoreLatch: true, // only when probeContainerState() === 'present'
});
Guarded by probeContainerState() === 'present', so a genuinely restoring tenant is
never snapshotted half-applied. Also bypasses the interval floor
(BACKUP_INTERVAL_MINUTES) — an emergency save is never “too soon”.
C5 — Checkpoint before any rollout or instance-type change
The standard-3 rollout today had no checkpoint: the park path
(checkpoint-park / resume) exists but only Lerma’s @upgrade_pipeline uses it.
Lerma.Cloudflare.Containers.change_instance_type/2must callPOST /control { X-FourIA-Control: checkpoint-park }first, exactly like the upgrade pipeline (fail-closed when the checkpoint is unavailable, exempt a sleeping instance), andresumeafter the rollout.- The same applies to
POST /accounts/{id}/containers/applications/{app}/rolloutsissued by the rescue tool (apps/fouria/src/diagnostics.ts): refuse to roll out an instance whose newest archive is older than the loss budget unless{"force": true}.
C6 — A crash artifact, because a VM kill leaves no trace
start-openclaw.shalready mirrors the gateway stdout into/tmp/gateway-stdout.log. On abnormal exit (non-zero, signal, or missing after a control-plane probe) write/tmp/gateway-crash.json({ lastSeenAt, pid, rssBytes, threads, exitCode, signal, tail: last 200 lines }) and upload it tocrashes/<ts>.jsoninBACKUP_BUCKETon the next boot.- Emit
gateway.process.gone { lastSeenAt, lastRssBytes, exitCode?, signal? }so the absence of a process is an event, not an inference. Today the SDK produced zero exit records for the dead gateway.
C7 — Stop the log/privacy leak while we are here
[WS] Container -> Client (raw) logs every frame with full model thinking, tool
arguments and tool results (≈48k events per 6 min during one agent session), and
Workers Logs persist them for 7 days — contradicting public/articles/PRIVACY.md.
Gate behind DEBUG_WS_FRAMES and log frame type/method/seq only.
4. Order of work
| # | Change | Risk | Value |
|---|---|---|---|
| 1 | C1 absent/unknown probe | low | removes the destructive-restore loop |
| 2 | C2 scheduler sampling | low | restores visibility + watchdog coverage |
| 3 | C3 + C4 emergency checkpoint | medium | bounds data loss to the budget |
| 4 | C6 crash artifact | low | makes the next incident diagnosable |
| 5 | C5 park before rollout (Lerma) | medium | protects manual resizes |
| 6 | C7 frame-log gating | low | privacy + log volume |
5. Tests and docs required by the repo covenants
src/persistence.test.ts— probepresent/absent/unknown;unknowndefers instead of restoring; norestore-neededonunknown.src/gateway/memory.test.ts—decideMemoryActiontable (none / checkpoint / restart / cooldown / not-dev / instance-too-small).src/activity/scheduler.test.ts— the alarm samples and enqueues the checkpoint; a failed checkpoint does not block the restart.src/routes/public.test.ts— WS path passesexecutionCtx.src/diagnostics.test.ts— rollout refused when the newest archive is stale.- Corpus:
test/business-rules/operational-modes.txt,cost-tracking.txt(persistence-snapshot { trigger: 'memory-pressure' }). - Docs:
apps/fouria/docs/WORKER-DEVELOPMENT.md,docs/plans/sandbox-memory-usage.md,docs/ENV-VARIABLES.md+.dev.vars.examplefor the new keys. - Changeset:
bun run changeset --patch fouria.
6. Out of scope
- Tier 1 container snapshots stay gated on the Phase 1 measurement
(
persistent-workspace-phase1.md); they need the tenant app onscheduling_policy: durable_object, which is a separate migration. - Restoring the work lost on 2026-11-03: not recoverable —
33c023e2(05:26) is the baseline. The fixes above exist so the next occurrence costs minutes, not hours.
7. Implementation status
Landed in this change:
- C1 —
probeContainerState()returnspresent | absent | unknown(apps/fouria/src/persistence.ts).restoreIfNeededrestores only onabsent, and throws the retryableContainerStateUnknownErroronunknown, without writing therestore-neededlatch. A runtime without anexists()probe keeps the historical “assume empty” reading so it can still restore. - C2 —
checkGatewayMemoryHealth()is the single sampler for both the request path and the durableActivityScheduleralarm (runMemoryWatch()), so a tenant is observed even when no browser polls/api/health. Every fresh sample now writes agateway-memory-samplerow carrying itsstate(running/absent) and the container limit, so a gap means “nobody sampled” instead of “nothing happened”. The health WebSocket handler now passes itsexecutionCtx, which previously let the detached watchdog promise be cancelled with the frame. - C3 (policy only) —
decideMemoryAction()implements the two-stage ladder (60 % checkpoint / 75 % restart,checkpoint-and-restartwhen the newest archive is older thanGATEWAY_MEMORY_LOSS_BUDGET_MS), and the alarm enqueues the checkpoint as atrigger: 'memory-pressure'snapshot job before the restart.maybeRestartGatewayForMemorydelegates to the same policy so the thresholds cannot drift; the request path still restarts only.
Still open: C4 (bypassing the restore latch for a proven-present container),
C5 (park before a manual rollout from Lerma), C6 (crash artifact +
gateway.process.gone) and C7 (frame-log gating).