F FourIA staging 2026.10.29 GitHub ↗

Sandbox memory growth: worker-thread heaps + oversized sessions

On this page

Status: in progress — diagnosis captured from a dev instance; investigation and remediation tracked in milestone 11 (v2026.10 - Perfomance enhacements). This plan bounds the OpenClaw gateway’s resident memory (RSS) growth inside the Cloudflare Sandbox container, from the Worker/container layer we own, so a dev instance does not drift toward the container/sandbox memory limit without an automatic recovery.

Landed so far: bounded transcript compaction (2.1) — hardened to 8 MB / 12k retained tokens with mid-turn precheck — the session-store disk/retention budget (2.1b), the soft-reload fixes (SIGUSR1 → SIGUSR2, and the signal now reaches the gateway server instead of the openclaw gateway run launcher), and the OPENCLAW_NO_RESPAWN keep decision (#463 — see the decision record), RSS/thread observability (Phase 1) and the dev-gated watchdog (2.4) — both in apps/fouria/src/gateway/memory.ts, surfaced on /api/health and the health stream. Remaining: re-measuring the drift on a dev instance with the watchdog enabled, and the worker-pool cap investigation (2.2).

Background

A dev instance (fouria / OpenClaw gateway in a Cloudflare Sandbox container) was observed at RSS ~1.85 GiB (peak 2.36 GiB), swap 0, with OpenClaw’s own memory diagnostic already firing:

subsystem=gateway/diagnostics/memory  reason=rss_threshold  rss=1.84 GiB
heap=339.9 MiB  threshold=1.5 GiB  thresholdRatio=122.7%

A V8 heap snapshot of the main isolate (/tmp/gw310-main.heapsnapshot, 290 MB) showed the main isolate heap is not the driver:

Metric (main isolate)Value
heapUsed~340 MB / limit 2.35 GiB
total self_size306 MiB
top retained: JSArrayBufferData35.6 MiB
empty strings ""25 MiB (186k)
(object properties)17.7 MiB
system/BytecodeArray9.4 MiB

No large JS retainer, no JS leak in the main isolate. The log file is tiny (316 KB → 1.5 MB); --verbose only raises log frequency, not memory.

Evidence

Measured from /proc/<pid> on the live gateway process:

ObservationValue
[anon]1,665 MB (Pss_Anon 1.82 GiB)
glibc [heap]99 MB
node binary (RSS)66 MB
anonymous mappings of ~256 KB4,885 ≈ 1.2 GiB
total threads31 (18 WorkerThread + 4 V8Worker + 4 libuv-worker)
OpenClaw worker startup state logworkerCount=8

The ~4,885 anonymous mappings of ~256 KB match V8’s heap page granularity (256 KB), i.e. per-isolate heap pages. The main isolate only accounts for ~412 MB of committed heap, so the remaining ~1.2 GiB lives in the heaps of the in-process worker_threads — each worker_thread is its own V8 isolate with its own heap, invisible to the main isolate’s getHeapStatistics().

Corroborating signals

  • workerCount=8 at startup vs 18 WorkerThread observed → thread/heap accumulation (workers not being reclaimed over the instance’s life).
  • SQLite contention: sessions.history.read / canonical-validation taking 9–10 s.
  • Session fouria-builder:main with 407,708 tokens in compaction preflight — a huge persisted transcript.

Side observation (soft-reload vs. inspector) — resolved

To take the snapshot, SIGUSR1 was sent to the gateway PID; it opened the Node inspector on 127.0.0.1:9229 (localhost-only; closes on gateway restart). This confirmed a real defect: OpenClaw’s documented restart signal is SIGUSR2 (SIGUSR1 only starts the inspector), while our softReloadGateway (apps/fouria/src/gateway/process.ts) sent SIGUSR1 expecting an in-process reload — so the soft reload was a silent no-op that left a debug socket open.

Fixed. buildSoftReloadScript() now sends kill -USR2, and test/e2e/n_gateway_soft_reload.txt asserts the inspector stays closed (inspector-closed) after a soft reload. SIGUSR2 is authorized by default: isRestartEnabled(config) is commands.restart !== false (default on) and server-start calls setGatewayRestartPolicy({ allowExternal: … }), so an external SIGUSR2 schedules the in-process restart rather than logging SIGUSR2 restart ignored (not authorized; commands.restart=false).

Second, independent defect (found while measuring the fix): the signal targeted the launcher, not the server. buildSoftReloadScript() signalled pgrep -f "[o]penclaw gateway run", which matches the launcher process (node /usr/local/bin/openclaw gateway run …) that supervises the real server. kill -USR2 <launcher> makes the launcher exit without restarting the server: the server is orphaned (PPID 1), readiness never dips, no new http server listening line appears — yet the endpoint still reports reloaded: true, so a broken reload looked successful. Measured on a live local worker (OpenClaw 2026.10.5): launcher-targeted reload ack_ms=30, readiness 123/123 at 200 true, launcher gone; a direct kill -USR2 <server> instead restarted the server (http server listening +1), dipped readiness at +120 ms (503) and recovered at ~3.3 s to listening / ~12–14 s to full readiness.

buildSoftReloadScript() now targets the server’s exact comm (pgrep -x 'openclaw-gatewa', Linux’s 15-char truncation, plus a defensive openclaw-gateway for kernels that keep longer names), deduped with sort -u, skipping zombies. Because /api/status.processId is the Worker-tracked supervisor (which survives an in-process reload), the E2E n_gateway_soft_reload.txt now asserts both the same processId and an increased http server listening count — processId stability alone passed in the broken case.

Root cause (working hypothesis)

Two coupled causes, both reachable from our layer:

  1. Oversized session transcripts. fouria-builder:main at 407k tokens keeps a large persisted transcript (JSONL + SQLite history), so the worker_threads that serve it hold large heaps; OpenClaw’s compaction may stay provider-side/server-side while the persisted transcript keeps growing. This also explains the 9–10 s SQLite sessions.history.read.
  2. In-process restarts + worker-thread accumulation. Inside a container the gateway never tears the process down on a restart, so the per-worker V8 heaps are never reclaimed — memory only returns to baseline on a hard restart (killGateway + reprovision). workerCount=8 → 18 WorkerThread is the accumulation signature.

Not yet proven: the report could not snapshot the worker isolates (linux-node-style per-worker inspector needed). The heap location is inferred from the 256 KB mapping analysis. Phase 1 of the plan confirms it before committing to the fixes.

What our layer controls

LeverLocationEffect on the problem
Session compaction configpackages/moltlazy/src/agents/defaults.ts (+ types.ts)Root cause #1 — bound the persisted transcript
OPENCLAW_NO_RESPAWN=1apps/fouria/DockerfileExplicit intent only — see the #463 decision
--verbosestart-openclaw.sh launch_gateway_process; process.ts buildGatewayRunScriptLog-noise only (coupled to boot-log legibility)
Gateway worker-thread pool(upstream OpenClaw config — to confirm)Root cause #2 — 18 vs 8 workers
Health probe (RSS/threads)apps/fouria/src/gateway/memory.ts, routes/public.ts, metrics.tsFixed — throttled /proc sample on the health snapshot + a gateway-resources AE row
Soft-reload SIGUSR2apps/fouria/src/gateway/process.ts buildSoftReloadScriptFixed — was SIGUSR1 (opened the inspector) and signalled the launcher instead of the server

Plan

Phase 0 — immediate containment (no code)

Hard-restart the dev instance (POST /debug/restart-openclaw → killGateway + reprovision) to drop RSS to baseline and tear down the accumulated worker threads. Containment only — confirms recoverability, buys time.

Phase 1 — confirm the diagnosis (low risk, high value)

Landed. apps/fouria/src/gateway/memory.ts reads VmRSS + Threads from /proc/<pid>/status for the gateway server in ONE shell exec, throttled to GATEWAY_RESOURCE_SAMPLE_INTERVAL_MS (60s) so the frequent health poll does not double its Sandbox RPC volume. The sample is attached to the health snapshot as gateway: { pid, rssBytes, threads } (only while the gateway is running, so a sleeping tenant is never woken) and written as a gateway-resources Analytics Engine row (double1 = RSS bytes, double2 = threads). This removes the need to SSH into a container to answer “was it memory?”.

The worker-heap growth was inferred, not proven. Before coding against it:

  1. Add a cheap read of the gateway process VmRSS (/proc/<pid>/status) and its thread count (/proc/<pid>/task or ps -p <pid> -o rss,nlwp) to the health snapshot / a new metric (src/metrics.ts, Analytics Engine), so the drift is visible over time.
  2. Distinguish a monotonic leak (threads/RSS unbounded) from a transient peak (one large session that later compacts).
  3. Verify SIGUSR1 semantics on the pinned OpenClaw version Done. SIGUSR1 opens the inspector and does not reload; the soft-reload signal is now SIGUSR2 (see “Side observation” above).

Phase 2 — root-cause fixes

2.1 Bound the session transcript (highest impact). Add compaction config to moltlazy so the persisted transcript cannot grow unbounded even when the model context is managed provider-side (OpenClaw key: agents.defaults.compaction.maxActiveTranscriptBytes, plus keepRecentTokens):

agents: { defaults: { compaction: {
  maxActiveTranscriptBytes: "8mb",
  keepRecentTokens: 12000,
  midTurnPrecheck: { enabled: true },
} } }

OpenClaw 2026.10.5 retired compaction.enabled (embedded auto-compaction is no longer toggleable) and compaction.truncateAfterCompaction; openclaw config validate rejects both keys. Byte-triggered compaction now opts in via maxActiveTranscriptBytes alone. 2026.10.7 adds midTurnPrecheck (default disabled), which bounds the in-memory prompt projection between tool calls.

  • Extend MoltLazyAgentDefaultsConfig (packages/moltlazy/types.ts) to Pick 'compaction'; add compaction to AGENT_DEFAULTS.
  • TDD in packages/moltlazy/tests/agents.test.ts; verify exact keys/values against docs.openclaw.ai before finalizing.
  • This also relieves the SQLite sessions.history.read cost (symptom).

2.1b Bound the session store (same PR). OpenClaw’s session.maintenance defaults are sized for a developer machine (maxDiskBytes: 10gb, maxEntries: 5000, pruneAfter: 30d) and cannot fit a 4–12 GB sandbox disk. patchSession now emits SESSION_MAINTENANCE_DEFAULTS:

session: { maintenance: {
  mode: "enforce",
  maxDiskBytes: "512mb",
  highWaterBytes: "384mb",
  maxEntries: 500,
  pruneAfter: "14d",
  archiveDashboardAfter: "7d",
  resetArchiveRetention: "14d",
  coldStorage: { enabled: true, afterDays: 7 },
} }

Cold storage moves inactive transcripts to compressed JSONL (authoritative history, restored on demand), so the working SQLite database — and the pages the worker threads read — stays small. Every key is fill-only.

2.2 Contain worker-thread accumulation. Re-evaluate OPENCLAW_NO_RESPAWN=1: decision (issue #463) — keep it. Removing the flag does not enable process-level restarts inside the Cloudflare Sandbox container. resolveGatewayRestartDecision() (dist/cli/gateway-lifecycle.runtime.js, checked on 2026.10.4 and 2026.10.5) returns mode: "disabled" both for OPENCLAW_NO_RESPAWN (reason: "no-respawn") and, with no supervisor marker, for any container (isContainerEnvironment() → reason: "unmanaged", detail container: use in-process restart to keep PID 1 alive). A restart therefore never re-spawns the process with or without the flag, so it cannot reclaim the worker-thread heaps. The flag stays as explicit intent; the Dockerfile comment and the operator docs were corrected to stop claiming it is what makes reloads in-process. The remaining lever is the Phase 2.4 watchdog (a hard killGateway + reprovision is the only thing that actually returns RSS to baseline). Also confirm whether OpenClaw exposes the worker pool size (workerCount) and cap it via moltlazy if configurable.

2.3 Reconsider --verbose (not “remove blindly”). It does not fix RAM; it only lowers log rate/IO. Removing it risks the boot-log legibility provided by start_boot_log_stream, which tails the gateway’s --verbose stdout. Options: drop to non-verbose after readiness, or gate it behind a flag.

2.4 Memory watchdog (materializes “memory isn’t reclaimed automatically”). With the Phase 1 metric: when gateway VmRSS exceeds a threshold, replace the gateway. Gate to dev first (non-billable, low risk); production only after validation.

Research — how hard can the limits be? (2026-10-26, OpenClaw 2026.10.7; rechecked on 2026.10.9)

Checked the schema (openclaw config validate) and config reference for every memory-related knob. 2026.10.7 introduced the keys below; 2026.10.8 and 2026.10.9 were confirmed to be config-schema-neutral, so the findings still hold. Findings:

LeverAvailable?Assessment
agents.defaults.compaction.maxActiveTranscriptBytesYesApplied — 20 MB → 8 MB. Bounds the persisted transcript + SQLite history.
agents.defaults.compaction.keepRecentTokensYesApplied — 20k → 12k. Smaller retained tail per turn.
agents.defaults.compaction.midTurnPrecheck.enabledYes (new in 2026.10.x)Applied — bounds the in-memory projection after tool results, before the next model call.
session.maintenance.maxDiskBytes / highWaterBytesYesApplied — 10 GB default → 512 MB / 384 MB per agent.
session.maintenance.maxEntries / pruneAfter / resetArchiveRetentionYesApplied — 500 / 14d / 14d (archives, not deletes).
session.maintenance.coldStorageYesApplied — inactive transcripts → compressed JSONL after 7 days.
agents.defaults.contextPruningYes (default off)Not applied. It trims old tool results per request (cache-ttl); a further prompt-size lever if the applied limits are insufficient, at the cost of reduced prompt-cache reuse.
Worker pool size (workerCount)NoInternal census only; no config key exists to cap or recycle the in-process worker threads.
RSS/heap thresholdNoOpenClaw’s 1.5 GiB rss_threshold diagnostic is code-owned; no config key.
Node heap cap (--max-old-space-size)Container-levelWould bound the main isolate (~340 MB used), not the 18 worker-thread isolates where the observed 1.2 GiB lives. Not applied.
Process-level restart to reclaim workersNoIn a container OpenClaw always takes the in-process restart path (#463), so worker heaps are never reclaimed by a restart.
Container instance memoryPlatformbasic = 1 GiB, standard-1 = 4 GiB, standard-2 = 6 GiB, standard-4 = 12 GiB.

Consequences:

  • The 1 GiB basic instance is below OpenClaw’s steady-state RSS (~1.1 GiB measured after an in-process restart). Any tenant running the gateway long enough will be OOM-killed there, so right-sizing basic is not a valid memory fix; standard-1 (4 GiB) is the practical floor for a tenant that runs agents, and standard-2 for Builder workloads.
  • Config limits can bound what causes growth (transcripts, SQLite history, tool-loop context), but they cannot reclaim the worker-thread heaps that grew while it happened. The only mechanism that returns RSS to baseline is the hard restart (killGateway + reprovision) — hence Phase 2.4 remains the safety net.
  • The applied limits are deliberately conservative (8 MB is ≈160k tokens, well above a normal working session) so they do not degrade ordinary conversations; if a dev instance still grows, the next step is contextPruning plus the watchdog, not a lower byte cap.

Instance sizing (headroom, not a fix)

TypeMemoryvCPUDiskWatchdog threshold (75%)Verdict
lite0.25 GiB0.06252 GB—Cannot unpack the image (disk too small).
basic1 GiB0.254 GB— (skipped)Below the gateway’s steady-state RSS; OOM-loops.
standard-14 GiB0.58 GB3 GiBPractical floor for an agent-running tenant.
standard-26 GiB112 GB4.5 GiB~1.5× memory for roughly half of standard-3.
standard-38 GiB216 GB6 GiBDeliberate headroom (2× memory, 4× core share); ~$28/mo.
standard-412 GiB420 GB9 GiBFor heavy Builder workloads.

Resizing is not free: it is a Cloudflare rollout (change_instance_type/2), which replaces the container — one cold start plus an R2 restore per tenant. Treat a bigger type as headroom against drift, not as the fix for it; the cheap wins (bounded transcripts, the instance-aware watchdog, a viable default) are what raise availability without a rollout.

Lerma.Instances.InstanceTypes.default/1 now returns standard-1 (it returned basic, which cannot host the gateway, so any instance provisioned without an explicit type OOM-looped on first use). Existing instances keep their type.

Phase 2.4 — memory watchdog (dev-gated)

Landed as maybeRestartGatewayForMemory (src/gateway/memory.ts), wired into the health route/stream. Design decisions that differ from the sketch above:

  • Threshold is the container limit, not OpenClaw’s 1.5 GiB warning. An observed healthy dev gateway sits at ~2.1 GiB (peak ~2.6 GiB), so a warning-level threshold would restart it in a loop — and it would make a permanently firing rss_threshold diagnostic look like an action item. The threshold is derived from the instance, not hardcoded: resolveMemoryRestartBytes() takes 75% of the container’s cgroup memory.max (3 GiB on a 4 GiB standard-1, 4.5 GiB on standard-2, 6 GiB on standard-3, 9 GiB on standard-4). A fixed 3 GiB would be only 37% of an 8 GiB standard-3 and restart a healthy gateway — worse availability, not better. Instances below 3 GiB (basic/lite) are skipped (instance-too-small): they cannot hold a healthy gateway, so a restart there can only loop. The gate sits under the 4 GiB standard-1 class so a cgroup that reports slightly less than nominal still gets a watchdog.
  • Resizing is a rollout, not an edit. An instance-type change goes through Lerma.Cloudflare.Containers.change_instance_type/2 → a full_auto rollout with target_configuration, which replaces the container. Plan one cold start plus an R2 restore per tenant; the next memory sample reads the new cgroup limit, so the watchdog threshold follows without a Worker deploy.
  • No soft-reload step. The measurement above shows an in-process SIGUSR2 restart does not reclaim RSS (1.14 → 1.38 → 1.18 GiB), so a reload first would only add a failed attempt before the real fix.
  • killGateway + reprovision, skipRestore. The filesystem is untouched by a kill, so re-entering the R2 restore barrier would deadlock against the backup lock for no benefit.
  • Cooldown (GATEWAY_MEMORY_WATCHDOG_COOLDOWN_MS, 30 min) is claimed before acting, so concurrent health polls cannot start two replacements.
  • The watchdog runs detached from the health response (a replacement takes minutes and must never delay or fail a health probe).

Phase 3 — SQLite (only if 2.1 is insufficient)

sessions.history.read / canonical-validation slowness is downstream of the oversized transcript. 2.1 should relieve it. WAL/checkpoint tuning is OpenClaw-internal and only pursued if 2.1 does not suffice.

Remaining follow-up

Tracked as a milestone 11 issue (see below). Items:

  1. Phase 1 observability: RSS + thread count in the health snapshot / Analytics Engine; confirm monotonic leak vs. transient peak. Done — gateway: { pid, rssBytes, threads } on /api/health + the health stream, and a gateway-resources AE row, sampled at most once a minute.
  2. Verify SIGUSR1 reload semantics Done — SIGUSR1 opens the inspector; the soft reload now uses SIGUSR2, targets the gateway server’s exact comm (openclaw-gatewa) rather than the launcher, and the E2E asserts a new http server listening line plus inspector-closed.
  3. Phase 2.1 compaction config in moltlazy (with TDD + docs verification). Done and hardened (2026.10.x schema, revalidated on 2026.10.9): maxActiveTranscriptBytes: '8mb' + keepRecentTokens: 12000 + midTurnPrecheck.enabled; enabled retired. Phase 2.1b session-store budget (maxDiskBytes 512 MB, 500 entries, 14-day retention, cold storage) is also in.
  4. Phase 2.2 OPENCLAW_NO_RESPAWN decision + worker-pool cap investigation. Decided: keep (see below); no worker-pool config key exists in 2026.10.9, so the cap is not reachable from moltlazy.
  5. Phase 2.3 --verbose scoping (preserve boot-log legibility).
  6. Phase 2.4 memory watchdog (dev-gated). Done — maybeRestartGatewayForMemory, dev-gated, instance-aware threshold (75% of the cgroup limit; 3 GiB on a 4 GiB standard-1), 30-minute cooldown, instance-too-small skip below 3 GiB.
  7. Re-measure RSS/thread drift after 2.1–2.4 on a dev instance. The drift was transcript-driven, so the 4 GiB box may already be adequate now that the compaction and session-store budgets have landed; compare gateway-resources AE rows before/after a week of normal use before buying a bigger instance.

Decision record — OPENCLAW_NO_RESPAWN (#463)

Decision: keep OPENCLAW_NO_RESPAWN=1.

The issue proposed removing the flag so the gateway could perform process-level restarts and reclaim memory. That premise does not hold on the pinned OpenClaw. resolveGatewayRestartDecision() returns mode: "disabled" when OPENCLAW_NO_RESPAWN is truthy (reason: "no-respawn") and — absent a supervisor marker — whenever isContainerEnvironment() is true (reason: "unmanaged", detail container: use in-process restart to keep PID 1 alive). Verified on both 2026.10.4 and 2026.10.5.

Consequences:

  • Removing the flag changes nothing inside the Cloudflare Sandbox container: restarts stay in-process, the PID is preserved, and the worker-thread V8 heaps are not reclaimed.
  • The flag is therefore explicit intent, not a memory lever. Its Dockerfile comment and the OPERATOR-GUIDE note were corrected (they claimed the flag was what made reloads in-process).
  • The only mechanism that returns RSS to baseline is the hard restart (POST /debug/restart-openclaw → killGateway + reprovision). Phase 2.4 automates it (dev-gated).
  • If process-level replacement is ever wanted, the supported route is OPENCLAW_SUPERVISOR_MODE=external plus a supervisor we own (a start-openclaw.sh loop or the watchdog). That is its own change, not a flag deletion: the supervisor must preserve the single-owner startup-lock invariant that ac_gateway_startup_concurrency.txt asserts.

Separately, this investigation found and fixed two soft-reload defects: the Worker sent SIGUSR1 (which starts Node’s inspector on 127.0.0.1:9229 and does not reload) instead of the documented SIGUSR2, and it signalled the openclaw gateway run launcher rather than the gateway server — killing the launcher and orphaning the server (PPID 1, no restart) while still reporting reloaded: true. Guards: buildSoftReloadScript unit tests (a “never sends SIGUSR1” regression and the exact comm target), and the n_gateway_soft_reload.txt E2E scenario, which requires a stable /api/status.processId and an increased http server listening count across the reload, plus inspector-closed via /debug/cli.

Measurement of a real in-process restart (local worker, OpenClaw 2026.10.5): ~3.3 s to listening, ~12–14 s to full readiness, server PID changes while the supervisor processId stays stable; RSS 1,142,368 KiB → 1,376,380 KiB (peak, +22 s) → 1,182,244 KiB (+49 s), threads 23 → 22. So the in-process restart does not reclaim memory — reinforcing the keep decision and leaving the hard-restart watchdog (Phase 2.4) as the only RSS lever.

Acceptance criteria

  • A dev instance’s gateway RSS is observable (RSS + thread count) from the Worker without an ad-hoc /proc inspection. Done — gateway: { pid, rssBytes, threads } on /api/health, /api/health-snapshot and /api/health-ws, plus a gateway-resources Analytics Engine row.
  • The fouria-builder session transcript is bounded by config; compaction fires locally before the persisted transcript grows unbounded. Done (2.1).
  • RSS returns toward baseline automatically (watchdog) instead of only on a manual hard restart. Done, dev-gated — maybeRestartGatewayForMemory replaces the gateway at 75% of the instance’s cgroup limit (3 GiB on a 4 GiB standard-1) with a 30-minute cooldown.
  • The soft-reload path is confirmed to reload (not open an inspector), or fixed. Done — soft reload uses SIGUSR2, targets the gateway server (not the launcher), and the E2E requires a new http server listening line as well as a stable processId.

Out of scope

  • SandboxV1 migration (docs/plans/sandbox-v1-scaffold.md).
  • R2 backup/restore paths.
  • Upstream OpenClaw internals beyond what config knobs we can set from moltlazy (no patching OpenClaw source).
  • Changing the container instance size / Cloudflare sandbox memory limits.