# Sandbox v1 adversarial + fuzz suites

Two seeded suites attack the Sandbox v1 (Sandbox SDK 1.0) critical pipelines
instead of exercising curated examples. Every random case is reproducible from
its seed, and a failure message names the seed and the case index.

| Suite                                            | Scope                                                                                                                                |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ |
| `apps/fouria/src/sandbox-migration/fuzz.test.ts` | Pure state machines (no DO): tri-state reads, validators, C8 barrier, retention, lease, runtime marker, intent gate, catalog listing |
| `apps/fouria/src/sandbox-v1-adversarial.test.ts` | The `SandboxV1` Durable Object: encryption, secrets, snapshots, cadence, boot, process map, transport                                |

```bash
bun run --cwd apps/fouria test src/sandbox-migration/fuzz.test.ts
bun run --cwd apps/fouria test src/sandbox-v1-adversarial.test.ts
```

## Attack matrix

| Pipeline         | Attack                                                                                                       | Invariant asserted                                                                                      | Result           |
| ---------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- | ---------------- |
| Zero data loss   | Random pointer / catalog record / conversion-absent / malformed / transport-error combinations (1 000 cases) | `verifyRecoveryPoint` agrees with an independent oracle; `unknown` never falls through to a legacy read | holds            |
| Zero data loss   | Random retention sizes, duplicate timestamps, shuffled bucket order (1 000 cases)                            | The protected record is never pruned, pruning is order-independent and deduplicated                     | holds            |
| Zero data loss   | SDK backup / pointer write / post-encryption failures (60 fuzz errors)                                       | A failed snapshot never advances `stable.json`; the previous point stays complete                       | holds            |
| Zero data loss   | Simulated stopped container (`running = false`)                                                              | Snapshot returns `skipped`, never starts the container, never archives an empty home                    | holds            |
| Zero data loss   | Catalog listing reports truncation with a repeated/absent cursor                                             | Listing is bounded (page cap + non-advancing-cursor stop)                                               | **fixed**        |
| Zero data loss   | Post-commit catalog listing/retention failure                                                                | The snapshot is still reported `committed` (the point already advanced)                                 | **fixed**        |
| Zero data loss   | Recovery point dated in the future (clock skew / corruption)                                                 | The idle cadence keeps snapshotting instead of suppressing the loss bound forever                       | **fixed**        |
| Zero data loss   | Random 80-snapshot sequence with changing `BACKUP_MAX_VERSIONS` and injected failures                        | Every committed point is restorable; retention stays bounded; a failure never moves it                  | holds            |
| Secrets          | Encryption-required × absent/empty/malformed/short/valid key matrix                                          | Required + unusable key uploads nothing; a broken optional key is never ignored                         | holds            |
| Secrets          | 60 random rejected key materials                                                                             | The key never appears in a thrown error or any console log                                              | holds            |
| Secrets          | Boot attempt with sentinel token/key env                                                                     | Container env is never logged                                                                           | holds            |
| Secrets          | Path-ish tenant names (`../../etc/passwd`, `%2e%2e`, …)                                                      | The backup prefix stays one encoded key segment                                                         | holds            |
| Memory           | `MAX_TRACKED_PROCESSES + 144` execs                                                                          | The DO process map is bounded; exited handles are dropped oldest-first                                  | **fixed**        |
| Memory           | One still-running handle plus a flood of exited handles                                                      | A running handle is never dropped (gateway stays killable)                                              | holds            |
| Boot             | Conversion record present without its stable pointer                                                         | Boot refuses to onboard over converted state; never launches the setup body                             | holds            |
| Boot             | 40 corrupt `stable.json` bodies                                                                              | Boot refuses on an unreadable pointer instead of onboarding                                             | holds            |
| Boot             | Brand-new tenant (no record, no pointer)                                                                     | Launch runs without `OPENCLAW_RESTORED` (the setup body owns the state)                                 | holds            |
| Boot             | Malformed `X-FourIA-Sandbox-Port` (`1e3`, `0x50`, `3.5`, …)                                                  | Rejected with `400` before any container access                                                         | **fixed**        |
| Boot             | Stopped container request                                                                                    | `503 SANDBOX_V1_SLEEPING`, never wakes the container                                                    | holds            |
| Tenant isolation | Runtime marker copied under another tenant's scope; intent gate random states                                | Marker authority is scope-bound; an intent never falls back to legacy                                   | holds            |
| Secrets          | Snapshot with sentinel gateway token / AI key / backup key in env                                            | No R2 object written by the snapshot contains a secret value                                            | **fixed** (scan) |
| Secrets          | Conversion failure whose error embeds a credential (signed-URL shape)                                        | The persisted status and the log contain `[redacted]`, never the value                                  | **fixed**        |
| Memory           | A process whose `output()` returns 3× the stream cap                                                         | The decoded output is truncated at `MAX_PROCESS_OUTPUT_BYTES` and flagged `truncated`                   | **fixed**        |
| Zero data loss   | A snapshot taken while an operator restore is swapping the directory                                         | `skipped: 'restoring'`, no archive, the previous point stays                                            | **fixed**        |
| Zero data loss   | A restore that fails mid-swap, then a retry                                                                  | The point never moves on failure and the in-progress barrier is cleared                                 | holds            |
| Zero data loss   | A second restore (and a boot) while a restore is in flight                                                   | Refused; the boot never launches over the swap                                                          | holds            |
| Zero data loss   | 12 snapshots → 24 randomized restores of surviving records → boot the point (fuzz)                           | Every surviving record restores, the point matches, and the boot skips the setup body                   | holds            |
| Memory (disk)    | A conversion whose extractor / upload fails                                                                  | The streamed archive and partial extraction are removed best-effort; the error wins                     | **fixed**        |
| Boot             | `start-openclaw.sh start` exits 0 with no gateway process anywhere                                           | The boot fails after the zero-exit grace instead of waiting the whole cold-start budget                 | **fixed**        |
| Boot             | `start-openclaw.sh start` exits 0 while the lock winner is still booting                                     | The DO keeps waiting and adopts the peer's readiness                                                    | holds            |

## Defects found and fixed

1. **Unbounded catalog listing loop (OOM).** `listRecordEntries` followed
   `page.cursor` without bounding the iteration. A bucket (or a proxy) that
   reports `truncated: true` with a non-advancing cursor made the loop read the
   same page forever; the fuzz run reproduced it as a real
   `FATAL ERROR: Ineffective mark-compacts near heap limit` in the test worker.
   Fixed with `MAX_RECORD_LIST_PAGES`, repeated-cursor detection and
   per-`recordId` deduplication — an incomplete list can only retain **more**
   records, which is the safe direction for retention (C6).
2. **A committed snapshot reported as failed.** `v1CreateSnapshot` wrote the
   catalog entry and advanced `stable.json`, then let a listing/retention error
   escape, so the caller saw a failed snapshot whose recovery point had in fact
   moved (and retention never ran). Retention is now best-effort after the
   commit and logs `sandbox.v1.snapshot.retention_failed`.
3. **Future-dated recovery point suppressed the cadence forever.** The dedupe
   test was `Date.now() - committedAt < interval`, which is true for any future
   timestamp. It now only dedupes a non-negative age inside the interval.
4. **Unbounded process-handle map.** Every `v1Exec` retained its `ExecProcess`
   handle for the DO's lifetime. The map is now capped at
   `MAX_TRACKED_PROCESSES` (256), dropping exited handles oldest-first and never
   dropping a running one.
5. **Port-header coercion.** The DO transport accepted `Number()` coercions
   (`1e3`, `0x50`). The header must now be canonical decimal digits before the
   range check.

## Observations left as code, not changed here

- **Conversion disk amplification.** `materializeLegacyArchive` writes the
  archive to `/tmp/fouria-legacy.sqsh`, extracts it to
  `/tmp/fouria-legacy-extract`, then copies the staged roots into
  `/home/openclaw`. Peak disk is roughly twice the archive plus the restored
  home (~3× archive) on a 4–12 GB container. A preflight free-space check (or
  extracting through a pipe) is the follow-up; it is a deploy/runtime decision,
  not a pure invariant.
- **Process cap vs. running handles.** The bound only evicts exited handles. A
  pathological burst of still-running processes can exceed it rather than losing
  the ability to kill the gateway.
- **Backup key scope.** `directoryBackupPrefix` is derived from the DO id
  (`sandbox-v1/openclaw/` for the fixed production container id). Tenant
  isolation comes from the per-tenant `fouria-backup-<slug>` bucket, not from
  the key; keep that binding invariant when changing provisioning.

## Second run: critical-pipeline hardenings (2026-11-09)

The second attack pass targeted the four incident pipelines again, this time
around the restore/snapshot/boot interactions and persisted text:

1. **A snapshot could race an operator restore.** `v1CreateSnapshot` only
   checked the container's liveness; the worker's maintenance window prevented
   the overlap for route-driven calls, but the primitive itself would archive a
   half-swapped directory and then advance the recovery point to it.
   `v1CreateSnapshot` now returns `skipped: 'restoring'` while `#restoring`.
2. **A boot could launch over a swap.** `startGatewayOwner` checked `#restoring`
   only at entry (through `v1EnsureGateway`); a restore that began while the
   state probe ran still launched. The owner now re-checks before the launch and
   inside the readiness loop, and reports a retryable failure instead of
   readiness for a gateway the restore is about to kill.
3. **Unbounded process output.** `v1ProcessOutput` decoded whatever the process
   wrote, after the SDK had already buffered it in the DO isolate. Reads are now
   truncated at `MAX_PROCESS_OUTPUT_BYTES` (1 MiB per stream) with
   `truncated: true`. The SDK's own buffer is still bounded only by the command,
   so callers must keep shell commands bounded (the system does: `tail -n 200`,
   JSON CLI output).
4. **Zero-exit boots burned the full cold-start budget.** A wrapper that exits 0
   (the lost-startup-lock path) was only judged by the health poll, so a launch
   nobody owned waited `timeoutMs` (up to 5 min). After
   `SANDBOX_V1_ZERO_EXIT_GRACE_MS` (default 15s, `0` disables) the DO proves no
   gateway (or setup wrapper) process exists (`pgrep -f` on neither) and fails fast;
   a booting peer (lock winner) resets the grace and is adopted normally.
5. **A failed conversion leaked its scratch disk.** The streamed squashfs and
   the partial extraction were removed only on the success path, so every failed
   attempt (a truncated archive, a corrupt superblock) left up to ~2× the archive
   in `/tmp` on a 4–12 GB disk. Failures now clean up best-effort and rethrow the
   original error.
6. **A conversion error could persist a credential.** The durable
   `migration:convert-status` record and the failure log stored
   `error.message` verbatim; SDK/transport errors can embed a signed URL or a
   header value. `redactSecrets` (known env values, ≥ 8 chars, longest-first) now
   scrubs both before they are written.

## Third run: billing + client isolation before the `develop` merge (2026-11-16)

A third pass targeted the three workflows that must not remain a security or
consistency risk when the `v1-sandbox` track merges: **data consistency**,
**billing**, and **client isolation**. It lives in
`apps/fouria/src/sandbox-v1-critical-adversarial.test.ts` (17 deterministic
cases) and complements the existing persistence fuzz and DO lifecycle suites.

| Pipeline         | Attack                                                                                                 | Invariant asserted                                                                                  | Result    |
| ---------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- | --------- |
| Billing          | A legacy worker with only `WORKER_URL` sends both a trace and a metric                                 | `/v1/metrics` and `/v1/traces` attribute to the **same** tenant, never splitting it with `default`  | **fixed** |
| Billing          | A billable instance with `CUSTOMER_SLUG='bad slug'` posts a metric                                     | Fails closed with `503 TENANT_UNRESOLVED`, exactly like the traces path                             | **fixed** |
| Billing          | A metric claims a foreign customer through `customer.slug` / `customer_id` / `customer`                | Rejected with `403`; no row is written                                                              | **fixed** |
| Billing          | A trace reuses a padded/case/separator/homoglyph variant of the resolved tenant                        | Only the exact identity emits; padding that normalizes to the authority still emits **that** tenant | holds     |
| Billing          | One batch names a billing-frozen customer and one unfrozen customer                                    | Only the unfrozen customer emits; the frozen rows are dropped before any side effect                | holds     |
| Billing          | The same action id is submitted twice in one isolate                                                   | Emitted once, never double-charged                                                                  | holds     |
| Billing          | An adversarial customer (`' OR 1=1`, quote, newline, `` ` ``, path) reaches the cost summary           | Rejected by the shared sanitizer; no Analytics Engine SQL is issued                                 | holds     |
| Billing          | A legacy `WORKER_URL`-only worker reads its cost summary                                               | The read side resolves the same identity the write side emitted                                     | **fixed** |
| Client isolation | `CUSTOMER_SLUG` / `WORKER_NAME` / `WORKER_URL` carry `default` or a padded/path form                   | Resolves to `null`, never the sentinel                                                              | holds     |
| Client isolation | Platform/environment affixes (`staging-`, `-dev`) are stripped once; loopback/IP URLs are refused      | Exactly one affix removed; local instances have no tenant identity                                  | holds     |
| Client isolation | The global leads bucket is reached through the bare fixed sandbox id (`openclaw`) with no env identity | Refused (`400`); a global bucket must never use the container id as a tenant namespace              | **fixed** |
| Client isolation | Per-tenant prefixes for lookalike slugs (`a`, `acme`, `acme-corp`, `acme.corp`)                        | Pairwise disjoint; no prefix relationship                                                           | holds     |
| Data consistency | A crafted record id (`../victim`, `a/b`, `../../openclaw/stable`) is catalogued                        | Percent-encoded; the key stays under the tenant prefix                                              | holds     |
| Data consistency | Retention runs across 1–12 records × 1–5 versions                                                      | The just-committed recovery point is never pruned; the catalog stays bounded                        | holds     |
| Data consistency | Leads keys/prefixes carry traversal (`../`, `..`, `//`)                                                | Rejected; a relative key can never escape the tenant root                                           | holds     |

**Fixes landed with the suite**

1. **One OTLP identity authority.** `resolveCostIdentity(env)`
   (`src/sandbox-tenant.ts`) is now the single resolver for `/v1/metrics`,
   `/v1/traces`, and the cost-summary read. It prefers the explicit trusted
   `CUSTOMER_SLUG` (or the `WORKER_NAME` it mirrors), then falls back to the
   routing resolver for a legacy `WORKER_URL`-only worker. A billable instance
   with no resolvable identity answers `503 TENANT_UNRESOLVED`; a metric that
   names a foreign customer through any tenant key answers `403`.
2. **Global leads bucket refuses the fixed-id fallback.** The leads bucket is
   shared across tenants, so `openclaw` can never be a tenant namespace. When no
   env identity resolves, the bridge answers `400` instead of scoping reads and
   writes to `openclaw/`.

**Residual (documented, not changed).** `resolveTenantSlug` still accepts a
**bare** container id as a last-resort slug for the per-tenant docs/software
bridges (their buckets are per-tenant, so the fallback is harmless there). Only
the global leads bridge needed the refusal. The migration paths themselves are
already barred from the context fallback by contract C10.
