Sandbox memory growth: worker-thread heaps + oversized sessions
On this page
Status: in progress — diagnosis captured from a dev instance; investigation and remediation tracked in milestone 11 (v2026.10 - Perfomance enhacements). This plan bounds the OpenClaw gateway’s resident memory (RSS) growth inside the Cloudflare Sandbox container, from the Worker/container layer we own, so a dev instance does not drift toward the container/sandbox memory limit without an automatic recovery.
Landed so far: bounded transcript compaction (2.1) — hardened to 8 MB / 12k retained tokens with mid-turn precheck — the session-store disk/retention budget (2.1b), the soft-reload fixes (
SIGUSR1→SIGUSR2, and the signal now reaches the gateway server instead of theopenclaw gateway runlauncher), and theOPENCLAW_NO_RESPAWNkeep decision (#463 — see the decision record), RSS/thread observability (Phase 1) and the dev-gated watchdog (2.4) — both inapps/fouria/src/gateway/memory.ts, surfaced on/api/healthand the health stream. Remaining: re-measuring the drift on a dev instance with the watchdog enabled, and the worker-pool cap investigation (2.2).
Background
A dev instance (fouria / OpenClaw gateway in a Cloudflare Sandbox container)
was observed at RSS ~1.85 GiB (peak 2.36 GiB), swap 0, with OpenClaw’s own
memory diagnostic already firing:
subsystem=gateway/diagnostics/memory reason=rss_threshold rss=1.84 GiB
heap=339.9 MiB threshold=1.5 GiB thresholdRatio=122.7%
A V8 heap snapshot of the main isolate (/tmp/gw310-main.heapsnapshot,
290 MB) showed the main isolate heap is not the driver:
| Metric (main isolate) | Value |
|---|---|
heapUsed | ~340 MB / limit 2.35 GiB |
total self_size | 306 MiB |
top retained: JSArrayBufferData | 35.6 MiB |
empty strings "" | 25 MiB (186k) |
(object properties) | 17.7 MiB |
system/BytecodeArray | 9.4 MiB |
No large JS retainer, no JS leak in the main isolate. The log file is tiny
(316 KB → 1.5 MB); --verbose only raises log frequency, not memory.
Evidence
Measured from /proc/<pid> on the live gateway process:
| Observation | Value |
|---|---|
[anon] | 1,665 MB (Pss_Anon 1.82 GiB) |
glibc [heap] | 99 MB |
| node binary (RSS) | 66 MB |
| anonymous mappings of ~256 KB | 4,885 ≈ 1.2 GiB |
| total threads | 31 (18 WorkerThread + 4 V8Worker + 4 libuv-worker) |
OpenClaw worker startup state log | workerCount=8 |
The ~4,885 anonymous mappings of ~256 KB match V8’s heap page granularity
(256 KB), i.e. per-isolate heap pages. The main isolate only accounts for
~412 MB of committed heap, so the remaining ~1.2 GiB lives in the heaps of the
in-process worker_threads — each worker_thread is its own V8 isolate with
its own heap, invisible to the main isolate’s getHeapStatistics().
Corroborating signals
workerCount=8at startup vs 18WorkerThreadobserved → thread/heap accumulation (workers not being reclaimed over the instance’s life).- SQLite contention:
sessions.history.read/canonical-validationtaking 9–10 s. - Session
fouria-builder:mainwith 407,708 tokens in compaction preflight — a huge persisted transcript.
Side observation (soft-reload vs. inspector) — resolved
To take the snapshot, SIGUSR1 was sent to the gateway PID; it opened the Node
inspector on 127.0.0.1:9229 (localhost-only; closes on gateway restart). This
confirmed a real defect: OpenClaw’s documented restart signal is SIGUSR2
(SIGUSR1 only starts the inspector), while our softReloadGateway
(apps/fouria/src/gateway/process.ts) sent SIGUSR1 expecting an in-process
reload — so the soft reload was a silent no-op that left a debug socket open.
Fixed. buildSoftReloadScript() now sends kill -USR2, and
test/e2e/n_gateway_soft_reload.txt asserts the inspector stays closed
(inspector-closed) after a soft reload. SIGUSR2 is authorized by default:
isRestartEnabled(config) is commands.restart !== false (default on) and
server-start calls setGatewayRestartPolicy({ allowExternal: … }), so an
external SIGUSR2 schedules the in-process restart rather than logging
SIGUSR2 restart ignored (not authorized; commands.restart=false).
Second, independent defect (found while measuring the fix): the signal
targeted the launcher, not the server. buildSoftReloadScript() signalled
pgrep -f "[o]penclaw gateway run", which matches the launcher process
(node /usr/local/bin/openclaw gateway run …) that supervises the real server.
kill -USR2 <launcher> makes the launcher exit without restarting the
server: the server is orphaned (PPID 1), readiness never dips, no new
http server listening line appears — yet the endpoint still reports
reloaded: true, so a broken reload looked successful. Measured on a live local
worker (OpenClaw 2026.10.5): launcher-targeted reload ack_ms=30, readiness
123/123 at 200 true, launcher gone; a direct kill -USR2 <server> instead
restarted the server (http server listening +1), dipped readiness at +120 ms
(503) and recovered at ~3.3 s to listening / ~12–14 s to full readiness.
buildSoftReloadScript() now targets the server’s exact comm
(pgrep -x 'openclaw-gatewa', Linux’s 15-char truncation, plus a defensive
openclaw-gateway for kernels that keep longer names), deduped with sort -u,
skipping zombies. Because /api/status.processId is the Worker-tracked
supervisor (which survives an in-process reload), the E2E
n_gateway_soft_reload.txt now asserts both the same processId and an
increased http server listening count — processId stability alone passed in
the broken case.
Root cause (working hypothesis)
Two coupled causes, both reachable from our layer:
- Oversized session transcripts.
fouria-builder:mainat 407k tokens keeps a large persisted transcript (JSONL + SQLite history), so theworker_threads that serve it hold large heaps; OpenClaw’s compaction may stay provider-side/server-side while the persisted transcript keeps growing. This also explains the 9–10 s SQLitesessions.history.read. - In-process restarts + worker-thread accumulation. Inside a container the
gateway never tears the process down on a restart, so the per-worker V8 heaps
are never reclaimed — memory only returns to baseline on a hard restart
(
killGateway+ reprovision).workerCount=8→ 18WorkerThreadis the accumulation signature.
Not yet proven: the report could not snapshot the worker isolates (
linux-node-style per-worker inspector needed). The heap location is inferred from the 256 KB mapping analysis. Phase 1 of the plan confirms it before committing to the fixes.
What our layer controls
| Lever | Location | Effect on the problem |
|---|---|---|
| Session compaction config | packages/moltlazy/src/agents/defaults.ts (+ types.ts) | Root cause #1 — bound the persisted transcript |
OPENCLAW_NO_RESPAWN=1 | apps/fouria/Dockerfile | Explicit intent only — see the #463 decision |
--verbose | start-openclaw.sh launch_gateway_process; process.ts buildGatewayRunScript | Log-noise only (coupled to boot-log legibility) |
| Gateway worker-thread pool | (upstream OpenClaw config — to confirm) | Root cause #2 — 18 vs 8 workers |
| Health probe (RSS/threads) | apps/fouria/src/gateway/memory.ts, routes/public.ts, metrics.ts | Fixed — throttled /proc sample on the health snapshot + a gateway-resources AE row |
Soft-reload SIGUSR2 | apps/fouria/src/gateway/process.ts buildSoftReloadScript | Fixed — was SIGUSR1 (opened the inspector) and signalled the launcher instead of the server |
Plan
Phase 0 — immediate containment (no code)
Hard-restart the dev instance (POST /debug/restart-openclaw → killGateway +
reprovision) to drop RSS to baseline and tear down the accumulated worker
threads. Containment only — confirms recoverability, buys time.
Phase 1 — confirm the diagnosis (low risk, high value)
Landed.
apps/fouria/src/gateway/memory.tsreadsVmRSS+Threadsfrom/proc/<pid>/statusfor the gateway server in ONE shell exec, throttled toGATEWAY_RESOURCE_SAMPLE_INTERVAL_MS(60s) so the frequent health poll does not double its Sandbox RPC volume. The sample is attached to the health snapshot asgateway: { pid, rssBytes, threads }(only while the gateway isrunning, so a sleeping tenant is never woken) and written as agateway-resourcesAnalytics Engine row (double1= RSS bytes,double2= threads). This removes the need to SSH into a container to answer “was it memory?”.
The worker-heap growth was inferred, not proven. Before coding against it:
- Add a cheap read of the gateway process
VmRSS(/proc/<pid>/status) and its thread count (/proc/<pid>/taskorps -p <pid> -o rss,nlwp) to the health snapshot / a new metric (src/metrics.ts, Analytics Engine), so the drift is visible over time. - Distinguish a monotonic leak (threads/RSS unbounded) from a transient peak (one large session that later compacts).
VerifyDone.SIGUSR1semantics on the pinned OpenClaw versionSIGUSR1opens the inspector and does not reload; the soft-reload signal is nowSIGUSR2(see “Side observation” above).
Phase 2 — root-cause fixes
2.1 Bound the session transcript (highest impact). Add compaction config to
moltlazy so the persisted transcript cannot grow unbounded even when the model
context is managed provider-side (OpenClaw key:
agents.defaults.compaction.maxActiveTranscriptBytes, plus keepRecentTokens):
agents: { defaults: { compaction: {
maxActiveTranscriptBytes: "8mb",
keepRecentTokens: 12000,
midTurnPrecheck: { enabled: true },
} } }
OpenClaw 2026.10.5 retired
compaction.enabled(embedded auto-compaction is no longer toggleable) andcompaction.truncateAfterCompaction;openclaw config validaterejects both keys. Byte-triggered compaction now opts in viamaxActiveTranscriptBytesalone. 2026.10.7 addsmidTurnPrecheck(default disabled), which bounds the in-memory prompt projection between tool calls.
- Extend
MoltLazyAgentDefaultsConfig(packages/moltlazy/types.ts) toPick'compaction'; addcompactiontoAGENT_DEFAULTS. - TDD in
packages/moltlazy/tests/agents.test.ts; verify exact keys/values againstdocs.openclaw.aibefore finalizing. - This also relieves the SQLite
sessions.history.readcost (symptom).
2.1b Bound the session store (same PR). OpenClaw’s session.maintenance
defaults are sized for a developer machine (maxDiskBytes: 10gb,
maxEntries: 5000, pruneAfter: 30d) and cannot fit a 4–12 GB sandbox disk.
patchSession now emits SESSION_MAINTENANCE_DEFAULTS:
session: { maintenance: {
mode: "enforce",
maxDiskBytes: "512mb",
highWaterBytes: "384mb",
maxEntries: 500,
pruneAfter: "14d",
archiveDashboardAfter: "7d",
resetArchiveRetention: "14d",
coldStorage: { enabled: true, afterDays: 7 },
} }
Cold storage moves inactive transcripts to compressed JSONL (authoritative history, restored on demand), so the working SQLite database — and the pages the worker threads read — stays small. Every key is fill-only.
2.2 Contain worker-thread accumulation. Re-evaluate
OPENCLAW_NO_RESPAWN=1: decision (issue #463) — keep it. Removing the flag
does not enable process-level restarts inside the Cloudflare Sandbox container.
resolveGatewayRestartDecision() (dist/cli/gateway-lifecycle.runtime.js, checked
on 2026.10.4 and 2026.10.5) returns mode: "disabled" both for
OPENCLAW_NO_RESPAWN (reason: "no-respawn") and, with no supervisor marker,
for any container (isContainerEnvironment() → reason: "unmanaged", detail
container: use in-process restart to keep PID 1 alive). A restart therefore
never re-spawns the process with or without the flag, so it cannot reclaim the
worker-thread heaps. The flag stays as explicit intent; the Dockerfile comment and
the operator docs were corrected to stop claiming it is what makes reloads
in-process. The remaining lever is the Phase 2.4 watchdog (a hard
killGateway + reprovision is the only thing that actually returns RSS to
baseline). Also confirm whether OpenClaw exposes the worker pool size
(workerCount) and cap it via moltlazy if configurable.
2.3 Reconsider --verbose (not “remove blindly”). It does not fix RAM; it
only lowers log rate/IO. Removing it risks the boot-log legibility provided by
start_boot_log_stream, which tails the gateway’s --verbose stdout. Options:
drop to non-verbose after readiness, or gate it behind a flag.
2.4 Memory watchdog (materializes “memory isn’t reclaimed automatically”).
With the Phase 1 metric: when gateway VmRSS exceeds a threshold, replace the
gateway. Gate to dev first (non-billable, low risk); production only after
validation.
Research — how hard can the limits be? (2026-10-26, OpenClaw 2026.10.7; rechecked on 2026.10.9)
Checked the schema (openclaw config validate) and config reference for every
memory-related knob. 2026.10.7 introduced the keys below; 2026.10.8 and
2026.10.9 were confirmed to be config-schema-neutral, so the findings still
hold. Findings:
| Lever | Available? | Assessment |
|---|---|---|
agents.defaults.compaction.maxActiveTranscriptBytes | Yes | Applied — 20 MB → 8 MB. Bounds the persisted transcript + SQLite history. |
agents.defaults.compaction.keepRecentTokens | Yes | Applied — 20k → 12k. Smaller retained tail per turn. |
agents.defaults.compaction.midTurnPrecheck.enabled | Yes (new in 2026.10.x) | Applied — bounds the in-memory projection after tool results, before the next model call. |
session.maintenance.maxDiskBytes / highWaterBytes | Yes | Applied — 10 GB default → 512 MB / 384 MB per agent. |
session.maintenance.maxEntries / pruneAfter / resetArchiveRetention | Yes | Applied — 500 / 14d / 14d (archives, not deletes). |
session.maintenance.coldStorage | Yes | Applied — inactive transcripts → compressed JSONL after 7 days. |
agents.defaults.contextPruning | Yes (default off) | Not applied. It trims old tool results per request (cache-ttl); a further prompt-size lever if the applied limits are insufficient, at the cost of reduced prompt-cache reuse. |
Worker pool size (workerCount) | No | Internal census only; no config key exists to cap or recycle the in-process worker threads. |
| RSS/heap threshold | No | OpenClaw’s 1.5 GiB rss_threshold diagnostic is code-owned; no config key. |
Node heap cap (--max-old-space-size) | Container-level | Would bound the main isolate (~340 MB used), not the 18 worker-thread isolates where the observed 1.2 GiB lives. Not applied. |
| Process-level restart to reclaim workers | No | In a container OpenClaw always takes the in-process restart path (#463), so worker heaps are never reclaimed by a restart. |
| Container instance memory | Platform | basic = 1 GiB, standard-1 = 4 GiB, standard-2 = 6 GiB, standard-4 = 12 GiB. |
Consequences:
- The 1 GiB
basicinstance is below OpenClaw’s steady-state RSS (~1.1 GiB measured after an in-process restart). Any tenant running the gateway long enough will be OOM-killed there, so right-sizingbasicis not a valid memory fix;standard-1(4 GiB) is the practical floor for a tenant that runs agents, andstandard-2for Builder workloads. - Config limits can bound what causes growth (transcripts, SQLite history,
tool-loop context), but they cannot reclaim the worker-thread heaps that grew
while it happened. The only mechanism that returns RSS to baseline is the hard
restart (
killGateway+ reprovision) — hence Phase 2.4 remains the safety net. - The applied limits are deliberately conservative (8 MB is ≈160k tokens, well
above a normal working session) so they do not degrade ordinary conversations;
if a dev instance still grows, the next step is
contextPruningplus the watchdog, not a lower byte cap.
Instance sizing (headroom, not a fix)
| Type | Memory | vCPU | Disk | Watchdog threshold (75%) | Verdict |
|---|---|---|---|---|---|
lite | 0.25 GiB | 0.0625 | 2 GB | — | Cannot unpack the image (disk too small). |
basic | 1 GiB | 0.25 | 4 GB | — (skipped) | Below the gateway’s steady-state RSS; OOM-loops. |
standard-1 | 4 GiB | 0.5 | 8 GB | 3 GiB | Practical floor for an agent-running tenant. |
standard-2 | 6 GiB | 1 | 12 GB | 4.5 GiB | ~1.5× memory for roughly half of standard-3. |
standard-3 | 8 GiB | 2 | 16 GB | 6 GiB | Deliberate headroom (2× memory, 4× core share); ~$28/mo. |
standard-4 | 12 GiB | 4 | 20 GB | 9 GiB | For heavy Builder workloads. |
Resizing is not free: it is a Cloudflare rollout (change_instance_type/2),
which replaces the container — one cold start plus an R2 restore per tenant.
Treat a bigger type as headroom against drift, not as the fix for it; the cheap
wins (bounded transcripts, the instance-aware watchdog, a viable default) are
what raise availability without a rollout.
Lerma.Instances.InstanceTypes.default/1 now returns standard-1 (it returned
basic, which cannot host the gateway, so any instance provisioned without an
explicit type OOM-looped on first use). Existing instances keep their type.
Phase 2.4 — memory watchdog (dev-gated)
Landed as
maybeRestartGatewayForMemory(src/gateway/memory.ts), wired into the health route/stream. Design decisions that differ from the sketch above:
- Threshold is the container limit, not OpenClaw’s 1.5 GiB warning. An observed healthy dev gateway sits at ~2.1 GiB (peak ~2.6 GiB), so a warning-level threshold would restart it in a loop — and it would make a permanently firing
rss_thresholddiagnostic look like an action item. The threshold is derived from the instance, not hardcoded:resolveMemoryRestartBytes()takes 75% of the container’s cgroupmemory.max(3 GiB on a 4 GiBstandard-1, 4.5 GiB onstandard-2, 6 GiB onstandard-3, 9 GiB onstandard-4). A fixed 3 GiB would be only 37% of an 8 GiBstandard-3and restart a healthy gateway — worse availability, not better. Instances below 3 GiB (basic/lite) are skipped (instance-too-small): they cannot hold a healthy gateway, so a restart there can only loop. The gate sits under the 4 GiBstandard-1class so a cgroup that reports slightly less than nominal still gets a watchdog.- Resizing is a rollout, not an edit. An instance-type change goes through
Lerma.Cloudflare.Containers.change_instance_type/2→ afull_autorollout withtarget_configuration, which replaces the container. Plan one cold start plus an R2 restore per tenant; the next memory sample reads the new cgroup limit, so the watchdog threshold follows without a Worker deploy.- No soft-reload step. The measurement above shows an in-process
SIGUSR2restart does not reclaim RSS (1.14 → 1.38 → 1.18 GiB), so a reload first would only add a failed attempt before the real fix.killGateway+ reprovision,skipRestore. The filesystem is untouched by a kill, so re-entering the R2 restore barrier would deadlock against the backup lock for no benefit.- Cooldown (
GATEWAY_MEMORY_WATCHDOG_COOLDOWN_MS, 30 min) is claimed before acting, so concurrent health polls cannot start two replacements.- The watchdog runs detached from the health response (a replacement takes minutes and must never delay or fail a health probe).
Phase 3 — SQLite (only if 2.1 is insufficient)
sessions.history.read / canonical-validation slowness is downstream of the
oversized transcript. 2.1 should relieve it. WAL/checkpoint tuning is
OpenClaw-internal and only pursued if 2.1 does not suffice.
Remaining follow-up
Tracked as a milestone 11 issue (see below). Items:
- Phase 1 observability: RSS + thread count in the health snapshot / Analytics
Engine; confirm monotonic leak vs. transient peak. Done —
gateway: { pid, rssBytes, threads }on/api/health+ the health stream, and agateway-resourcesAE row, sampled at most once a minute. VerifyDone —SIGUSR1reload semanticsSIGUSR1opens the inspector; the soft reload now usesSIGUSR2, targets the gateway server’s exactcomm(openclaw-gatewa) rather than the launcher, and the E2E asserts a newhttp server listeningline plusinspector-closed.- Phase 2.1 compaction config in moltlazy (with TDD + docs verification).
Done and hardened (2026.10.x schema, revalidated on 2026.10.9):
maxActiveTranscriptBytes: '8mb'+keepRecentTokens: 12000+midTurnPrecheck.enabled;enabledretired. Phase 2.1b session-store budget (maxDiskBytes512 MB, 500 entries, 14-day retention, cold storage) is also in. - Phase 2.2
OPENCLAW_NO_RESPAWNdecision + worker-pool cap investigation. Decided: keep (see below); no worker-pool config key exists in 2026.10.9, so the cap is not reachable from moltlazy. - Phase 2.3
--verbosescoping (preserve boot-log legibility). - Phase 2.4 memory watchdog (dev-gated). Done —
maybeRestartGatewayForMemory, dev-gated, instance-aware threshold (75% of the cgroup limit; 3 GiB on a 4 GiBstandard-1), 30-minute cooldown,instance-too-smallskip below 3 GiB. - Re-measure RSS/thread drift after 2.1–2.4 on a dev instance. The drift was
transcript-driven, so the 4 GiB box may already be adequate now that the
compaction and session-store budgets have landed; compare
gateway-resourcesAE rows before/after a week of normal use before buying a bigger instance.
Decision record — OPENCLAW_NO_RESPAWN (#463)
Decision: keep OPENCLAW_NO_RESPAWN=1.
The issue proposed removing the flag so the gateway could perform process-level
restarts and reclaim memory. That premise does not hold on the pinned OpenClaw.
resolveGatewayRestartDecision() returns mode: "disabled" when
OPENCLAW_NO_RESPAWN is truthy (reason: "no-respawn") and — absent a
supervisor marker — whenever isContainerEnvironment() is true
(reason: "unmanaged", detail container: use in-process restart to keep PID 1 alive). Verified on both 2026.10.4 and 2026.10.5.
Consequences:
- Removing the flag changes nothing inside the Cloudflare Sandbox container: restarts stay in-process, the PID is preserved, and the worker-thread V8 heaps are not reclaimed.
- The flag is therefore explicit intent, not a memory lever. Its Dockerfile
comment and the
OPERATOR-GUIDEnote were corrected (they claimed the flag was what made reloads in-process). - The only mechanism that returns RSS to baseline is the hard restart
(
POST /debug/restart-openclaw→killGateway+ reprovision). Phase 2.4 automates it (dev-gated). - If process-level replacement is ever wanted, the supported route is
OPENCLAW_SUPERVISOR_MODE=externalplus a supervisor we own (astart-openclaw.shloop or the watchdog). That is its own change, not a flag deletion: the supervisor must preserve the single-owner startup-lock invariant thatac_gateway_startup_concurrency.txtasserts.
Separately, this investigation found and fixed two soft-reload defects: the Worker
sent SIGUSR1 (which starts Node’s inspector on 127.0.0.1:9229 and does not
reload) instead of the documented SIGUSR2, and it signalled the
openclaw gateway run launcher rather than the gateway server — killing
the launcher and orphaning the server (PPID 1, no restart) while still reporting
reloaded: true. Guards: buildSoftReloadScript unit tests (a “never sends
SIGUSR1” regression and the exact comm target), and the
n_gateway_soft_reload.txt E2E scenario, which requires a stable
/api/status.processId and an increased http server listening count across
the reload, plus inspector-closed via /debug/cli.
Measurement of a real in-process restart (local worker, OpenClaw 2026.10.5):
~3.3 s to listening, ~12–14 s to full readiness, server PID changes while the
supervisor processId stays stable; RSS 1,142,368 KiB → 1,376,380 KiB (peak,
+22 s) → 1,182,244 KiB (+49 s), threads 23 → 22. So the in-process restart
does not reclaim memory — reinforcing the keep decision and leaving the
hard-restart watchdog (Phase 2.4) as the only RSS lever.
Acceptance criteria
- A dev instance’s gateway RSS is observable (RSS + thread count) from the Worker
without an ad-hoc
/procinspection. Done —gateway: { pid, rssBytes, threads }on/api/health,/api/health-snapshotand/api/health-ws, plus agateway-resourcesAnalytics Engine row. - The
fouria-buildersession transcript is bounded by config; compaction fires locally before the persisted transcript grows unbounded. Done (2.1). - RSS returns toward baseline automatically (watchdog) instead of only on a
manual hard restart. Done, dev-gated —
maybeRestartGatewayForMemoryreplaces the gateway at 75% of the instance’s cgroup limit (3 GiB on a 4 GiBstandard-1) with a 30-minute cooldown. The soft-reload path is confirmed to reload (not open an inspector), or fixed.Done — soft reload usesSIGUSR2, targets the gateway server (not the launcher), and the E2E requires a newhttp server listeningline as well as a stableprocessId.
Out of scope
- SandboxV1 migration (
docs/plans/sandbox-v1-scaffold.md). - R2 backup/restore paths.
- Upstream OpenClaw internals beyond what config knobs we can set from moltlazy (no patching OpenClaw source).
- Changing the container instance size / Cloudflare sandbox memory limits.