# Testing Guide

Testing infrastructure reference for all FourIA products. See [CONTRIBUTING.md](/llms/platform/reference/contributing/index.md) for contribution workflow and [DEVELOPMENT.md](/llms/platform/reference/development/index.md) for onboarding.

## Overview

| Product             | Type              | Framework                  | Unit Tests              | E2E Tests                      |
| ------------------- | ----------------- | -------------------------- | ----------------------- | ------------------------------ |
| `apps/fouria`       | Cloudflare Worker | Vitest + cctr + Playwright | Colocated `*.test.ts`   | Cloud + Local + Business Rules |
| `apps/lerma`        | Phoenix Web App   | ExUnit + cctr              | Mirror `lib/` structure | Sequential cctr scenarios      |
| `packages/moltlazy` | JS Config + SDK   | Vitest + cctr              | `tests/*.test.ts`       | cctr CLI scenarios             |

> The `cloudflare-unified-billing` plugin's unit and live-integration tests now live in [Paso4/openclaw-plugins](https://github.com/Paso4/openclaw-plugins).

---

## Quick Start

```bash
# All JS/TS tests (fouria + moltlazy)
bun run test

# fouria admin SPA UI tests (Playwright, local, no credentials)
bun run --cwd apps/fouria test:ui

# TypeScript type checking (Worker, moltlazy, cost-calculator, dispatch, tail)
bun run typecheck

# Lint all TS/JS sources
bun run lint

# Full CI suite (test + typecheck + local integration + moltlazy E2E)
bun run test:all
```

See each product section below for product-specific commands.

### Release tooling tests

`scripts/changeset-changelog.test.ts`, `scripts/release-lib.test.ts`,
`scripts/sync-versions.test.ts`, and `scripts/version-packages.test.ts` cover
the Changesets changelog format, the release-notes renderer, CalVer changelog
generation, the Lerma `package.json` → `mix.exs` version sync, and the
no-changesets no-op path. They run as part of `bun run test`.

### Sandbox v1 migration plan

The `v1-sandbox` track carries a machine-readable evaluation plan at
[`docs/plans/sandbox-v1-evaluation-plan.json`](/assets/docs/plans/sandbox-v1-evaluation-plan.json)
(human summary: [`sandbox-v1-evaluation-plan.md`](/llms/platform/reference/plans/sandbox-v1-evaluation-plan/index.md)).
It maps every implemented V1 capability to a domain, a milestone and a v1
disposition, and encodes the five-step protocol each milestone executes.

```bash
bun run plan:sandbox-v1          # regenerate from docs/releases/V1-RELEASE-NOTES.md
bun run plan:sandbox-v1:check    # stale-plan + structural + coverage gate
bun run plan:sandbox-v1:sync     # merge origin/develop into v1-sandbox
```

The plan's `e2e_suitability` block is the source of truth for which existing
suites apply to the v1 runtime and which new `apps/fouria/test/e2e-v1/**`
scenarios a milestone must add. Staging validation targets the Lerma client
`client-sandbox-v1` (id 20), never the reserved `client` slug or production.

### Sandbox v1 live drills

`apps/fouria/test/e2e-v1/` is a corpus of **live** drills against a deployed
tenant (default skip unless `E2E_V1_LIVE=1`). Each suite drives the shared
`fixture/v1-drill` (one JSON object per mode) and self-skips when the live
environment is absent, so collection is always safe:

| Suite                                  | Covers                                                                   |
| -------------------------------------- | ------------------------------------------------------------------------ |
| `00_runtime_selection.txt`             | Served lane + lifecycle both report `v1`                                 |
| `01_fresh_boot_and_migration.txt`      | Real readiness after a boot **and** a committed recovery point (C8)      |
| `02_concurrency_and_serving_owner.txt` | A probe burst folds into exactly one gateway owner                       |
| `03_passive_health_and_wake.txt`       | Non-waking passive probe, idle stop → automation wake, keep-alive policy |
| `04_channels_and_rpc_ws.txt`           | Agent/model/channel CLI over the v1 process shim                         |
| `05_freeze_and_control.txt`            | Freeze snapshots then destroys; a probe wakes; `unfreeze` is a no-op     |
| `06_backup_restore_retention.txt`      | A record taken before a destroy is the record restored                   |
| `07_canary_gates.txt`                  | The served bundle identity matches the promoted build; not degraded      |

`05`/`06` destroy the container and require `E2E_V1_DESTRUCTIVE=1`. Run it with
`bun run --cwd apps/fouria test:e2e:v1:live`; see
[`apps/fouria/test/e2e-v1/README.md`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e-v1/README.md)
for prerequisites, per-suite timing knobs and the temporary billable procedure.
Use a **dev** instance: Lerma's `HealthMonitor` polls `production` clients every
2 minutes and a passive probe is still a Durable Object request that prevents
the v1 inactivity stop.

The business-rules corpus also covers the v1 admin/billing/cost surface without a
live tenant: `apps/fouria/test/business-rules/sandbox-v1-{agents,billing,cost}.txt`
pin the runtime-independent route suites against the v1 transport tests
(process shim, DO entrypoint, single owner), the OTLP/billable emit path, the
freeze semantics and the cost-summary query. `src/sandbox-v1-outbound.test.ts`
pins the v1 outbound host set and the "a host never throws" contract (an
escaping exception is an opaque `ECONNRESET` in the container).

The gateway proxy attribution fallback is unit-tested without a live tenant:
`apps/fouria/src/gateway/proxy-attribution.test.ts` covers the proxy-shaped
header set and the `403` + `proxy_attribution_required` payload classifier, and
the `gateway proxy attribution` block in `src/sandbox-v1.test.ts` covers the
transport (`direct` fallback once the probe is refused, cached verdict, the
forwarded headers kept when attribution is accepted or the probe cannot reach a
booting gateway, no probe for a request without forwarded headers, and the
re-probe after `v1Destroy`).

### Sandbox v1 adversarial / fuzz suites

Two seeded suites attack the Sandbox v1 critical pipelines (zero data loss,
secrets, memory, boot). `apps/fouria/src/sandbox-migration/fuzz.test.ts` covers
the pure state machines (tri-state reads, validators, the C8 recovery-point
barrier, retention, the migration lease, runtime-marker authority and the
runtime intent gate); `apps/fouria/src/sandbox-v1-adversarial.test.ts` covers the
`SandboxV1` Durable Object (encryption fail-closed matrix, secret non-leakage,
failed-snapshot recovery-point safety, post-commit retention, cadence clock
skew, boot refusals, the process-map bound, the transport port guard and
randomized catalog integrity). A third, deterministic suite —
`apps/fouria/src/sandbox-v1-critical-adversarial.test.ts` — attacks the three
workflows that must not be a security or consistency risk at merge time:
**billing** (one OTLP identity authority for metrics/traces and the cost summary,
cross-tenant claims, freeze isolation, replay dedupe, Analytics Engine SQL
injection), **client isolation** (sentinel/affix resolution, the fixed
sandbox-id fallback on the global leads bucket, lookalike-slug prefix
disjointness) and **data consistency** (record-id percent-encoding, retention
never pruning the committed point, traversal rejection). All three run as part of
`bun run test`. The attack matrix, the defects the runs found, and the remaining
observations are in
[`docs/plans/sandbox-v1-adversarial-fuzz.md`](/llms/platform/reference/plans/sandbox-v1-adversarial-fuzz/index.md).

### Docs channel tests

`webs/fouria-docs/scripts/channel.test.ts` covers the versioned docs plumbing:
channel selection/validation, base-path normalization, channel-relative link
prefixing, and the `versions.json` manifest (`latest`/`staging` paths). It runs
as part of the root `bun run test` (`vitest.config.ts` includes
`webs/fouria-docs/scripts/**/*.test.ts`).

---

## apps/fouria — Cloudflare Worker

### Unit Tests

Colocated `.test.ts` files alongside source files. Run with vitest.

```bash
cd apps/fouria
bun run test                  # All unit tests
bun run test:watch            # Watch mode
bun run test:coverage         # With Istanbul coverage (alias of coverage)
bun run test:client           # SPA React component/page tests (jsdom)
```

> The Worker unit project (`vitest.config.ts`, Node pool) excludes `src/client/**`.
> The SPA component and page tests run under the dedicated jsdom project
> (`vitest.client.config.ts`) via `bun run test:client` — they need the React
> plugin and a DOM, and declare `/** @vitest-environment jsdom */` per file.

#### Authentication, backup, and billing regression coverage

- `src/auth/middleware.fallbacks.test.ts` drives the Hono authentication and role-resolution
  chain: malformed forwarded JWTs, E2E cookies, local-role fallback, and Dispatch plan metadata.
  `src/auth/routing.test.ts` covers invalid tiers, input coercion, and hostname edges.
- `src/activity/scheduler.test.ts` covers durable boot success/retry, snapshot interval and idle
  guards, deprovision checkpoint-before-stop, restore jobs, maintenance jobs, and automation wake
  lead windows. `src/backup-watch/durable-object.test.ts` covers enqueue failures without clearing
  dirty state. `src/routes/api/storage.errors.test.ts` covers backup/restore conflicts, error
  responses, background failures, and job/config read failures. The persistence and encryption
  suites remain the owners of archive, lease, retention, and recovery-point guarantees.
- `src/restore-trigger-adversarial.test.ts` and the restore-trigger section of
  `src/sandbox-v1-adversarial.test.ts` encode the destructive-operation invariant for both
  runtimes: a restore may run only when an administrator explicitly requests one, when the
  durable debug/hard-restart path armed `restore-needed`, or when the container positively
  reports **no tenant state**. A gateway that is merely down, crashed, or being rebooted keeps
  its container filesystem, so the launch must never apply an older archive over it. Inconclusive
  signals (control-channel error, malformed probe, unreadable pointer) fail closed, and a request
  owner never runs the restore inline (the durable boot job owns it). The same suites pin the two
  hardening rules this matrix relies on: a container-state probe that is unavailable (`no exists()`)
  or failed (`success: false`, or a v1 probe exit code other than `1`) is **never** read as empty,
  and the legacy stuck-start watchdog checkpoints and verifies the recovery point advanced before
  it is allowed to reprovision a container.
- `src/routes/api/billing.freeze.test.ts` exercises persisted freeze/unfreeze, acting-user
  attribution, and the OR of stored and Dispatch freeze flags. `src/billable-actions.emit.test.ts`
  covers producer chunking, single-send fallback, delivery failures, frozen-customer exclusion,
  duplicate suppression, and fail-closed outcome normalization. A producer with neither `sendBatch`
  nor `send` must reject, never report unsent actions as enqueued.

Run `bun run test:coverage` from the repository root for comparable cross-package coverage.
The root `typecheck` delegates to the five workspace configurations listed above rather than
recursively checking vendored Elixir dependencies and marketing-site sources under one Node-only
configuration. Moltlazy has a package-local strict `tsconfig.json`; it no longer inherits the root
configuration. These runs do not prove deployed Cloudflare Access policies or live Stripe delivery;
unit line coverage is not a claim of complete end-to-end or branch coverage.

> **Billing-surface coverage.** `src/client/pages/__tests__/HomePage.test.tsx`
> renders the **Usage & Billing** widget with cost data present (total plus the
> billable action/credit rollup) and asserts the empty (`$0.00`) and role-hidden
> cases. `src/client/pages/__tests__/AccountPage.test.tsx` covers the **Plan &
> Billing** card (tier, seat/credit usage, past-due alert, non-admin hidden). The
> `/api/admin/costs/v2/summary` aggregation (AE SQL query → summary) is unit-tested
> in `src/cost-summary.test.ts` and `src/routes/api/costs.test.ts`.

### UI Tests (Playwright, local)

The admin SPA has a Playwright suite that drives the **real** dashboard served by
the Vite dev server (`bun run dev`), with every `/api/*` call mocked per test.
No Worker, sandbox container, or Cloudflare credentials are required, so it runs
fully offline and is the fastest way to exercise the UI end-to-end.

```bash
cd apps/fouria
bun run test:ui           # headless Chromium
bun run test:ui:headed    # watch the browser
```

Specs live in [`apps/fouria/test/ui/`](../apps/fouria/test/ui/) and the config is
[`apps/fouria/playwright.config.ts`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/playwright.config.ts). The
shared mock helper (`test/ui/support/api.ts`) stubs `GET /api/admin/whoami`
(role/dev/demo flags), the page data endpoints, and the `/api/health-ws` +
`/api/admin/devices/ws` WebSocket streams. Coverage:

- **App shell** — sidebar, notification bell, ADMIN navigation, dev-only nav.
- **Role access** — ROOT → Diagnostics redirect, ADMIN blocked from Diagnostics,
  BASE_USER limited nav + route guard.
- **Pages** — every admin route renders for its role.
- **Agent Studio API regression** (`src/routes/api/agent-studio.test.ts`) — the
  dedicated automations endpoint normalizes cron and interval schedule objects
  to display strings (including the `kind`/`everyMs`/`anchorMs` crash case).
  `GatewayStatusLed.test.tsx` verifies non-ROOT users never query debug status.
- **Agent Studio** (`agent-studio.spec.ts`) — the roster + overview render, all
  seven tabs (Capabilities, MCP, Memory, Automations, Connections, Health) open
  over the mocked RPC payloads, `AGENTS.md` opens in the RPC editor, and a
  sleeping instance shows the wake affordance without any read waking it. The
  mock helper exposes `studio` / `studioDetail` options plus
  `SAMPLE_STUDIO_ROSTER` / `SAMPLE_STUDIO_DETAIL` fixtures. Unit coverage in
  `src/agents/studio.test.ts` and `src/routes/api/agent-studio.test.ts`.
- **Health plugin pagination** (`agent-studio.spec.ts`) — 10 plugins per page,
  navigation to the final partial page and back, disabled boundary controls, and
  no pagination controls for a single-page list.
- **Capabilities pagination** (`agent-studio.spec.ts`) — independent 10-item pages
  for allowed/denied tools, subagents, skills, available/effective tools across
  groups, and notices; filtering resets pages before displaying matching items.
- **Releases** (`releases.spec.ts`) — the promotion page stages a release on a
  dev instance (with a commit message), applies it on production, and warns when
  production lacks the `PROMOTION_SOURCE_BUCKET` binding. The mock helper exposes
  `releaseEnvironment` and `promotionSourceBinding` for
  `GET /api/admin/releases/status` and an empty `GET /api/admin/releases/history`.
  Unit coverage in `src/client/pages/__tests__/ReleasesPage.test.tsx`.
- **Notification bell** — pending-device badge + dropdown, empty state.
- **Persistence lock indicator** (`PersistenceLockIndicator`) — the lock widget
  beside the bell reflects the shared `PersistenceLockContext` state, which the
  `BackupHistoryPanel` raises/clears around a backup or restore (no polling):
  open/green when idle, closed/red while an operation is in flight, and hidden
  for BASE_USERs. Unit coverage in
  `src/client/components/__tests__/PersistenceLockIndicator.test.tsx` and
  `src/client/components/__tests__/BackupHistoryPanel.test.tsx`.
- **Gateway circuit breaker** (`gateway-circuit-breaker.spec.ts`) — with the
  health stream `down`, gateway queries are parked (no `/api/admin/agents`
  request) and the waiting indicator shows; flipping the mocked health stream to
  `up` flushes the queued queries and renders the data. The mock helper exposes
  `installApiMocks(...).sendHealth(snapshot)` and the `HEALTH_UP` / `HEALTH_DOWN`
  fixtures for this. Unit coverage lives in `src/client/gateway-gate.test.ts`,
  `src/client/GatewayHealthContext.test.tsx`, and `src/client/api.test.ts`.
- **Diagnostics dashboard (skip-not-park)** — unit coverage in
  `src/client/pages/__tests__/DiagnosticDashboard.test.tsx` and
  `src/client/components/__tests__/ConfigSection.test.tsx`: with the gateway
  down the page still renders but its container reads (`/api/admin/config*`,
  devices) are never dispatched, gateway-only debug probes are skipped with an
  inline explanation, and the Configuration Editor stays visible with a
  non-blocking waiting notice instead of a blocking spinner.
- **Billing widget** (`billing-widget.spec.ts`) — the Home "Usage & Billing"
  widget shows the billed total and billable action/credit rollup when the cost
  summary carries data, `$0.00` when it is empty, and is hidden for BASE_USERs.
  The mock helper exposes a `costSummary` option.
- **Cloudflare Components widget** (`cloudflare-components.spec.ts`) — the ROOT
  diagnostics widget renders each component's status/latency/href, hides the AI
  Gateway when the API omits it, reveals the binding-derived `facts` panel on
  hover **and** keyboard focus, opens each card's Cloudflare dashboard logs page
  in a new tab (offline dashboard stub), renders a plain non-clickable card when
  a component has no `logsUrl`, re-fetches on Refresh, and reflects
  `sleeping`/`down` states. The mock helper's `HEALTH_COMPONENTS` fixture carries
  the enriched `facts` + `logsUrl` shape returned by `GET /api/health/components`.
- **Diagnostics responsive layout** (`diagnostics-responsive.spec.ts`) — at a
  phone viewport the Cloudflare Components strip collapses into a compact status
  widget and expands on tap, the gateway + persistence status widgets sit
  parallel to it (the topbar omits them on the diagnostics page), the Fouria
  DevOps options render as a vertical icon + name menu (collapsed by default,
  tap to open), the Configuration Editor reflows to the viewport, and the page
  never scrolls sideways (checked at 320px). The mock helper exposes
  `debugRoutes` (to render the Configuration Editor) and a `config` fixture for
  the parsed sections + raw JSON.

> **NixOS / Nix dev shell.** `apps/fouria/flake.nix` provides the Playwright
> browsers via `PLAYWRIGHT_BROWSERS_PATH` (`pkgs.playwright-driver.browsers`)
> plus the `PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS` and
> `PLAYWRIGHT_HOST_PLATFORM_OVERRIDE` NixOS workarounds. Because the Nix browser
> bundle can lag the project's npm `playwright` version, `playwright.config.ts`
> resolves the newest `chromium-*` under `PLAYWRIGHT_BROWSERS_PATH` and passes it
> as an explicit `executablePath`; outside Nix the variable is unset and
> Playwright's default resolution applies.

**Key test files** (see [WORKER-DEVELOPMENT.md](/llms/platform/apps/fouria/worker-development/index.md) for full list):

| File                                  | Tests                                                                                                                                                         |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `auth/jwt.test.ts`                    | JWT decoding and validation                                                                                                                                   |
| `auth/jwks.test.ts`                   | JWKS fetching and caching                                                                                                                                     |
| `auth/middleware.test.ts`             | Auth middleware, DEV_MODE, E2E_TEST_MODE, dev-instance ROOT→ADMIN role preview                                                                                |
| `gateway/env.test.ts`                 | Container environment variable building                                                                                                                       |
| `gateway/process.test.ts`             | Process finding, lifecycle, stuck-start watchdog, and structured gateway-start failure correlation                                                            |
| `gateway/classify.test.ts`            | `classifyGatewayStart` running-only stuck-start verdict                                                                                                       |
| `gateway/sandbox-exec.test.ts`        | Sandbox command results and structured process failure logs                                                                                                   |
| `gateway/utils.test.ts`               | Process wait lifecycle and non-zero exit reporting                                                                                                            |
| `index.test.ts`                       | Worker middleware, proxy routes, and request failure trace context                                                                                            |
| `gateway/r2.test.ts`                  | R2 mounting logic                                                                                                                                             |
| `auth/rbac.test.ts`                   | RBAC permission matrix                                                                                                                                        |
| `metrics.test.ts`                     | Cost/usage Analytics Engine instrumentation                                                                                                                   |
| `persistence.test.ts`                 | R2 backup/restore, retryable cold-restore intent, scheduled-snapshot safety, cross-isolate lease, crash-consistent catalog + orphan reconciliation            |
| `restore-trigger-adversarial.test.ts` | Restore-trigger matrix: gateway down/reboot never restores live state; only an explicit latch or an empty container may restore; unknown probes fail closed   |
| `sandbox-v1-adversarial.test.ts`      | v1 zero-data-loss and boot attacks, including "a v1 boot restores only an empty container" (gateway down, stopped container, probe failure, operator restore) |

### E2E Tests (Cloud)

Deploys a real worker to `*.workers.dev` via Terraform + `wrangler deploy`. Requires Cloudflare credentials.

```bash
cd apps/fouria

# Full E2E with browser
bun run test:e2e

# CLI-only (no browser)
bun run test:e2e:cli
```

**Setup** (`test/e2e/_setup.txt`):

1. Provisions cloud infrastructure via Terraform (R2 bucket) — the content-hash worker cache is **temporarily disabled** (issue #297), so every run provisions fresh
2. Deploys worker via `wrangler deploy --var E2E_TEST_MODE:true`
3. Starts Playwright browser via `plwr`
4. Polls `/api/health` for cold start up to the startup budget — 30s on a warm cache hit, otherwise `STARTUP_TIMEOUT_SECS` (600s default)
5. Opens browser to worker URL, waits for "Device pairing required"

> **Fresh-deploy failures are usually the shared v1 app.** The top-level
> (local/E2E) `wrangler.jsonc` still declares the `SandboxV1` container
> application as `fouria-sandbox-v1`, which belongs to the production worker;
> `server/deploy` renames it per run (`<worker>-sandbox-v1`). If a fresh E2E
> deploy reports "Durable Object-managed Container application settings were not
> fully applied" after five attempts, check that rename first. Teardown probes
> the teardown token against the account endpoint and falls back to the deploy
> token, then reports the real container-app delete status — a stale
> `E2E_CLOUDFLARE_API_TOKEN` used to make every delete silently no-op and leak
> billing applications.
>
> **Phase-D suite assumptions.** `y_production_profile` creates its recovery
> point on demand (`POST /debug/backup/sync`) because `BACKUP_CHANGE_WATCH` is
> off on the E2E worker, so no scheduled snapshot exists. `zz_gateway_hard_restart`
> reads `processId` from `/api/health` (the `/status/polling` RPC fast path
> deliberately omits the process scan) and waits for the post-restore boot
> before capturing its baseline. `get-agent-id` retries the roster for a bounded
> window because `h_cli_knowledge` runs it right after a plugin reload.

> **Tenant identity is explicit.** Both fixtures bind `CUSTOMER_SLUG`
> (`acme-corp` for the cloud worker, matching the corpus' mock tenant and the
> admin cross-tenant guard's `E2E_EXPECTED_CUSTOMER`; `start-local` sets it for
> the billable profile). Without it the Worker derives its identity from
> `WORKER_NAME`, which a bare `*.workers.dev` E2E worker does not set, so
> `getCostContext().customer` falls back to the `default` sentinel. That is not a
> valid tenant slug, so the software-manifest store refuses to build and
> `PUT /api/admin/software/manifest` answers an instant 500 — the fixture bug that
> failed `zzz_software_provisioning`, not a route defect.

> **Gateway readiness — shared helper.** After provisioning, the cctr
> `_setup.txt` waits for gateway readiness through
> [`apps/fouria/test/lib/e2e-helpers.sh`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/lib/e2e-helpers.sh).
> Its `e2e_wait_for_health` probes `/api/health` **immediately** (the first probe
> also triggers the cold boot — there is no fixed 180s pre-poll grace) and
> requires **both** `startup.ok` and `readiness.ok`, polling every 5s. The wait
> stays bounded by the existing warm/cold startup budget: 30s on a warm cache
> hit, otherwise `STARTUP_TIMEOUT_SECS` (600s default). The same helper backs
> `test/e2e-agent/_setup.txt` and the parallel driver's top-level readiness wait.
> It does **not** back every readiness check: the parallel runner's generated
> per-group `_setup.txt` and `ab_gateway_ws_origin.txt` keep their own
> grace-free, bounded `/api/health` loops (the latter also uses the shared
> `e2e_classify_ws_upgrade` verdict helper). The helpers are unit-tested in
> `apps/fouria/src/e2e-helpers.test.ts`.

> **E2E cost safety (issue #297):** the container idle timeout is **derived** from the
> instance hostname (staging 15m / dev 2h — the Sandbox SDK 1.0 `setInactivityTimeout()`
> platform cap); there is no `SANDBOX_SLEEP_AFTER` knob. Billable production instances
> (production platform, production environment) do **not** auto-sleep. A container that
> outlives its run (failed teardown, workflow timeout) therefore stops billing at the
> derived idle instead of running 24/7, and the workflow teardown deletes the workers
> outright. Worker caching is disabled until the feature is reworked to reuse only
> within working hours and to delete on eviction.

> **Boot profiles:** Both profiles provision the same **production-profile
> image** (`apps/fouria/Dockerfile`, `BUILD_PROFILE=production`, the slim
> default) **and the same `standard-2` instance type** (1 vCPU / 6 GiB — the
> smaller `basic` 1 GiB / `standard-1` 0.5 vCPU instances stall the gateway under
> the suite's CLI load). The optional PDF/document tooling (`pandoc` +
> `weasyprint`) ships only in the `dev` build profile used by the dev/staging
> environment — see `apps/fouria/docs/WORKER-DEVELOPMENT.md`. PR runs use the
> light `basic` profile (`E2E_TEST_MODE=true`, short plugin timeouts) for fast
> feedback; develop runs use the `production` profile (real plugin timeouts,
> container OTLP export + Cloudflare Tunnel). The profiles differ by env only —
> not the image or instance type. Both profiles bind the worker OTLP receiver
> with a **random per-run `OTEL_SHARED_SECRET`** (no GitHub secret required), so
> the OTel billing-queue test runs in either profile.

> **Single-worker parallel E2E:** the `e2e` workflow provisions one isolated
> worker and runs dependency-grouped cctr suites against it. Independent phase A
> and B groups run concurrently; config/lifecycle/restore groups remain serial.
> Pull requests run boot-only by default. `develop` and a full manual dispatch
> run the corpus; selective retries link only the suites recorded in the prior
> run's `e2e-failed-*` cache. The runner primes the CLI/device path once per
> group so device-admin suites do not each pay the cold first-connect cost.

> **E2E run statistics:** `run-parallel.sh` writes `e2e-stats.md` with initial
> and per-group Gateway readiness times, cctr group outcomes and case durations,
> and per-group counts of Worker-launched OpenClaw CLI calls, Gateway WebSocket
> connections, and Admin HTTP RPC calls. The report and debug transcripts are
> uploaded as the `e2e-logs-*` run artifact, and mirrored to the
> `e2e-artifacts-*` branch on a best-effort basis (the branch push never fails
> the job). Transport counters require an explicit
> `X-FourIA-E2E-Stats: 1` request, `DEBUG_ROUTES=true`, and an explicitly bound
> `E2E_TEST_MODE` profile marker (`true` for `basic`, `false` for `production`).
> The long-lived Gateway process is excluded from CLI totals. CI pins cctr to
> `0.31.0` for stable report parsing.

> **Failed-only reruns:** when a run reaches the gateway (`gateway-ready`) but
> some suites fail, the job records those suite names and caches them
> (`e2e-failed-<ref>-<shard>`). The next run restores that list and passes it to
> `run-parallel.sh`, which links only those suites into the generated cctr groups
> (the PR already passed unit + `validate-secrets`, enforced by `needs`). A full pass clears the list
> and the following run is a full pass again. A committed seed
> (`.github/e2e-rerun-suites.txt`, only when present) bootstraps a subset before
> any cache exists; it is not required in steady state.

**Teardown** (`test/e2e/_teardown.txt`):

1. Dumps gateway logs + doctor output
2. Stops browser (saves video)
3. Destroys cloud resources (worker, container app, R2 bucket, Terraform)

**Test files** (alphabetical order — naming determines execution order):

| File                                 | What it tests                                                                                                                                                                                                   |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `b_cli_devices.txt`                  | `openclaw devices list` through `/debug/cli`                                                                                                                                                                    |
| `c_cli_models.txt`                   | `openclaw models list`, model catalogs                                                                                                                                                                          |
| `d_cli_channels.txt`                 | Channel probe, config inspection                                                                                                                                                                                |
| `e_cli_config.txt`                   | Config validate, set, unset via CLI                                                                                                                                                                             |
| `g_cli_secrets.txt`                  | Secrets audit, reload via CLI                                                                                                                                                                                   |
| `h_cli_knowledge.txt`                | Wiki init, status, get via CLI                                                                                                                                                                                  |
| `i_cli_cron.txt`                     | Cron list, add dream job                                                                                                                                                                                        |
| `j_cli_integrations.txt`             | Integration config verification                                                                                                                                                                                 |
| `k_secrets_store.txt`                | Secrets Store (gateway token, CDP, container config, device pairing)                                                                                                                                            |
| `l_cli_billing.txt`                  | Pricing constants, costs, instance types, billable events                                                                                                                                                       |
| `m_cli_agent_tokens.txt`             | Agent turns, token usage metrics                                                                                                                                                                                |
| `provision_and_dispatch.txt`         | **Cloud-only** — dispatch worker + Supabase tenant lifecycle. Its control-plane steps need `INTERNAL_DISPATCH_INVALIDATE_TOKEN` (staging value) and self-skip when it is unset; there is no hardcoded fallback. |
| `a_rbac_admin_isolation.txt`         | ADMIN role isolation (management API, cross-tenant)                                                                                                                                                             |
| `a_rbac_api_matrix.txt`              | ROOT/ADMIN/BASE_USER matrix + deny/allow decisions over dashboard + loading-page endpoints                                                                                                                      |
| `a_rbac_base_user_isolation.txt`     | BASE_USER isolation (whoami, blocked from management/secrets/writes)                                                                                                                                            |
| `a_rbac_device_pairing.txt`          | Device pairing permissions for all 3 roles (with retry on timeout)                                                                                                                                              |
| `a_rbac_root_isolation.txt`          | ROOT isolation (diagnostics access, blocked from management/agents/secrets)                                                                                                                                     |
| `a_rbac_ui_pages.txt`                | UI page access for all 3 roles (25 test cases; every nav page route served)                                                                                                                                     |
| `aa_health_gate.txt`                 | Gateway health check between RBAC and CLI test groups                                                                                                                                                           |
| `ac_gateway_startup_concurrency.txt` | Cross-isolate cold-start serialization after destroying the container                                                                                                                                           |
| `ad_observability_canary.txt`        | **Local-only** — proxy header redaction + live-gateway-log RPC-stream warning (Local Explorer SQL API)                                                                                                          |
| `o_opencode_credits_task.txt`        | OpenCode credits automation executed in the sandbox (direct consumer)                                                                                                                                           |
| `o_leads_proxy.txt`                  | Leads R2 proxy over the Worker WebSocket (self-skips without `cdp-secret.txt`)                                                                                                                                  |
| `r_browser_cdp.txt`                  | Cloudflare Browser Rendering `BROWSER` binding over the public `/cdp` shim (discovery, shared-secret auth, target + DOM + PNG); self-skips when the binding is unavailable                                      |
| `r_browser_remote_profile.txt`       | Platform `browser.profiles.cloudflare` remote-CDP profile → container proxy → private `cdp.internal` bridge (registration, open, cross-call tabs)                                                               |
| `p_tasks_queue.txt`                  | TASKS queue producer → consumer → sandbox round trip (self-skips w/o key)                                                                                                                                       |
| `p_otel_billing_queue.txt`           | OTLP traces receiver → billable reconcile → BILLING_QUEUE producer (self-skips on a non-billable instance)                                                                                                      |
| `test/e2e-billing/`                  | **Local-only** — billable worker emits to BILLING_QUEUE (observed via Local Explorer) + freeze drop                                                                                                             |
| `p_scheduled_cron.txt`               | **Local-only** — both crons fire via `/cdn-cgi/local/scheduled`                                                                                                                                                 |
| `z_cli_staging.txt`                  | API staging tests (costs, storage, users)                                                                                                                                                                       |
| `zzz_cron_wake.txt`                  | **Cloud-only** — container destroy, cron wake cycle                                                                                                                                                             |

> **Backup/restore suites are excluded from this corpus.** They live in
> [`apps/fouria/test/e2e-backup/`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e-backup/README.md) so a
> default `bun run test:e2e` / `test:e2e:local` run does not trigger their
> multi-minute cold boots and ~250 MB R2 snapshots. Run them with
> `bun run test:e2e:backup` (local) or through `phaseE_restore` in the parallel
> runner. Files: `zz_reprovision_guard`, `zzz_backup_cold_restore`,
> `zzz_backup_restore_skip_setup`, `zzz_backup_semantics`, and the legacy
> `zzz_persistence_lifecycle`.

> **Retry diagnostics in cctr tests.** cctr merges a test's stdout **and stderr**
> into the value matched by `{{ … }}`. A retry/progress line printed to `>&2`
> therefore breaks a strict matcher (`json object`, `json bool`, a bare number,
> …) once the retry path is taken — the flake only appears when the first probe
> is unhealthy. Write such diagnostics to a side log instead:
> `echo "…" >>"$CCTR_WORK_DIR/e2e-retry.log" 2>&1`.

> **`o_leads_proxy` requires `cdp-secret.txt`.** The test self-skips (and passes
> in ~0s) when the fixture file is missing, so a regression in the fixture wiring
> looks like a green test. `test/e2e/fixture/server/start` persists
> `$CDP_SECRET` to `cdp-secret.txt` on **both** the cache-hit and fresh-provision
> paths; if the test reports `skipped: true`, the fixture is the place to look.

> **`p_otel_billing_queue` self-skips on a non-emitting instance.** The OTLP
> traces receiver gates emission on `isBillableEventsEnabled` (`src/dev-mode.ts`):
> billable production **and the staging platform always emit**, while a
> dev-environment instance accepts + ignores approved spans (HTTP 200) and
> reports `billable: false`, never emitting to `BILLING_QUEUE`. The local E2E
> worker runs with `DEV_MODE=true` and no routable `WORKER_URL`, so it is a dev
> instance — the test skips (like it already does for a missing secret / `401`)
> instead of asserting an emit that cannot happen locally. The emit path is
> asserted by the cloud profile (a production `workers.dev` hostname / the
> staging platform) and by `src/routes/otel-traces.test.ts`. A dev instance can
> opt in explicitly with `BILLABLE_EVENTS=true`; the override changes emission
> only, so the sleep policy and the cost-row dimensions are untouched.
> `provision_and_dispatch` likewise self-skips in local mode (no dispatch worker
>
> - Supabase).

> **Worker billing-emit E2E (`test/e2e-billing/`, local-only).** The
> producer half of the billing workflow is covered locally by opting the worker
> into a **billable** identity (`E2E_BILLABLE=true` → `start-local` passes a
> production-looking `WORKER_URL`/`CUSTOMER_SLUG`). The suite POSTs a synthetic
> OTLP span, asserts `emitted >= 1`, observes the `BILLING_QUEUE` producer via
> the Local Explorer `queue*` span, then freezes the customer and asserts
> `emitted == 0` with no new queue span. Analytics Engine writes are not
> observable locally (no AE binding in Local Explorer; no `writeDataPoint`
> span), which is why the lerma consumer suite uses a fake AE. Run with
> `bun run --cwd apps/fouria test:e2e:billing:local`; see
> `apps/fouria/test/e2e-billing/README.md`.

> **`h_cli_knowledge` caches the owner agent id.** The corpus resolves the owner
> agent id **once per suite** through `test/e2e/fixture/get-agent-id`, which
> caches it in `$CCTR_FIXTURE_DIR/agent-id.txt` and reuses it across the
> sequential `wiki init` / `wiki status` / `wiki get` cases. The helper reads
> `/api/admin/agents` through Admin HTTP RPC, then Gateway WebSocket RPC, with
> the route's CLI fallback for compatibility; caching avoids repeating that
> roster lookup per case while preserving the plugin/wiki assertions. The cache
> is suite/worker-run scoped (cctr copies `fixture/` into
> `$CCTR_FIXTURE_DIR` once and reuses that directory across cases) and is
> discarded when cctr cleans up the per-suite temporary fixture/work directory,
> not by `_teardown.txt`. The helper fails closed: an empty or malformed roster
> exits nonzero instead of silently defaulting to `main`.

### E2E Tests (Local)

Runs tests against the local Worker dev server (Cloudflare Vite plugin, `vite dev`) with containers via Docker. Avoids 5-10 minute Terraform provisioning. Cloud-only tests (dispatch worker + Supabase, container destroy/recreate) self-skip in local mode.

```bash
cd apps/fouria

# Local E2E with Docker + cctr (auto-builds Docker image)
bun run test:e2e:local

# CLI-only (no browser)
bun run test:e2e:local:cli
```

**Setup** (`fixture/server/start-local`):

1. Sources `test/e2e/.dev.vars` and installs it as the Worker's `.dev.vars` for the run (the Cloudflare Vite plugin has no `--env-file`; `stop-local` restores the developer's file)
2. Finds unused port (8787-8797)
3. Starts `vite dev --port <PORT> --strictPort` (also generates `CDP_SECRET`/`OTEL_SHARED_SECRET`, injected as vars through the plugin's `config` hook, and uses `/cdn-cgi/local/scheduled` for the cron test). The outer `test:e2e:local*` scripts run `bun run build:docker` first (chained with `&&`, so it completes before the dev server starts); `vite dev` then builds/launches the container image through Docker.
4. Polls `GET /api/health?passive=1` until the Worker answers **any** HTTP status (`200`/`503`/…) — a cold boot legitimately answers **503** while the container/gateway is still starting, so the gate must not require a `200` and must not use `curl -f`. It probes the fast, **non-waking** passive endpoint so a poll does not wake the container or re-trigger the restore/`ensureGateway` boot on every attempt. The window defaults to 120s and is overridable with `E2E_STARTUP_TIMEOUT_SECS`. The real gateway readiness wait (`startup.ok` + `readiness.ok`) is the separate `wait for gateway API readiness` setup step.
5. Writes fixture files (dummy CF Access creds — ignored via `DEV_MODE=true`)

> **Resource headroom.** The local run drives the Docker daemon (container image +
> container), `workerd` (Vite dev server + Miniflare R2/DOs), and the `bun`/`cctr`
> drivers at once. On a memory-constrained host the whole process group can be
> killed by `SIGABRT` (OOM) mid-cold-boot — observed during
> `_setup: start moltworker server` after the image built and the dev server
> answered 503. Give Docker/the host at least ~8 GB and avoid running other
> containers alongside the suite. The readiness gate above fails fast when
> the dev server itself dies, but it cannot distinguish an OOM kill from a crash,
> so check `dmesg`/Docker logs when the process group dies during startup.

> **`Container failed to start` / no app container.** If the suite stalls at the
> readiness step while `docker ps` shows only a `…-proxy` sidecar that is
> recreated every ~20 s, the host cannot start local containers at all: on Linux
> hosts where `net.ipv4.conf.all.src_valid_mark=1` (Tailscale sets this), the
> sidecar handshake is dropped and workerd times out
> ([cloudflare/workerd#6860](https://github.com/cloudflare/workerd/issues/6860)).
> The dev server prints the fix before building images; the short version is
> `echo 'net.core.devconf_inherit_init_net = 2' | sudo tee
> /etc/sysctl.d/99-docker-netns-defaults.conf && sudo sysctl -w
> net.core.devconf_inherit_init_net=2`, then
> `docker rm -f $(docker ps -aq --filter name=workerd-)`. See
> [DEVELOPMENT.md](/llms/platform/reference/development/index.md#local-containers-never-start-container-failed-to-start).

**Teardown** (`fixture/server/stop-local`):

1. Kills the Vite dev server process
2. Restores the app's `.dev.vars`
3. Cleans up fixture files

**Prerequisites:**

```bash
# test/e2e/.dev.vars must contain:
DEV_MODE=true
E2E_TEST_MODE=true
DEBUG_ROUTES=true
MOLTBOT_GATEWAY_TOKEN=<value>
CF_ACCOUNT_ID=<id>
CLOUDFLARE_AI_GATEWAY_API_KEY=<key>   # plus CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID
# The REST transport uses CLOUDFLARE_API_TOKEN, derived from the same AI-scoped key.

# Docker must be running
# wrangler must be authenticated via OAuth (wrangler login)
```

**Cloud-only tests** (self-skip when `E2E_LOCAL=true`):

| File                         | Reason                              |
| ---------------------------- | ----------------------------------- |
| `provision_and_dispatch.txt` | Requires dispatch worker + Supabase |
| `zzz_cron_wake.txt`          | Requires container destroy/recreate |

> The Dispatch control-plane steps in `provision_and_dispatch.txt` (and the
> staging integration corpus) send `INTERNAL_DISPATCH_INVALIDATE_TOKEN` to the
> deployed Worker and self-skip when it is unset. Both the Worker and Lerma get
> that value from `STAGING_INTERNAL_DISPATCH_INVALIDATE_TOKEN` in CI; the Worker
> fails closed (403) without it, and there is no shared hardcoded key.

> **`ad_observability_canary` asserts on the local worker's own logs.** The
> Local Explorer exposes the same `logs` / `spans` tables as the Cloudflare
> Workers Observability API, scoped to the local dev worker, via
> `POST /cdn-cgi/local/explorer/api/local/observability/query` (wrapped by the
> `fixture/observability-query` helper). The canary uses it to (1) drive the
> catch-all proxy's request log with sentinel Cookie / Authorization /
> CF-Access-Jwt-Assertion / tenant-metadata values and assert **none** reach the
> log store, and (2) fetch the running gateway's logs and assert the fetch adds
> no `ReadableStream received over RPC disconnected prematurely` warning. Both
> cases are **local-only** (the API does not exist on a deployed worker), so
> they self-skip in cloud E2E; they **fail loudly** if the local API is
> unavailable rather than silently skipping, so the coverage cannot rot. These
> are the observability-validation canaries for the credential-logging and
> live-process-stream fixes.

### Admin UI Render Timing (local-only, `plwr`)

`test/e2e/ui_components_render.txt` is a **local-only** browser suite that
validates the fouria admin (root) UI components and reports how long each takes
to render. It drives the `plwr` session started by `_setup.txt` (the same one the
pairing step uses), sets the `X-Mock-Role` cookie, loads each admin route, waits
for the component's root selector, and measures, in milliseconds relative to the
page navigation start:

- `openMs` — navigation start → SPA loaded (`performance.now()` sampled right
  after `plwr open` returns)
- `componentMs` — navigation start → the component's selector is present
  (sampled right after `plwr wait` returns); this is the data-driven render
- `waitMs` — `componentMs − openMs` (client render + API round-trip after load)
- `nav.*` — Navigation Timing (`domInteractive`, `domContentLoaded`, `load`) and
  First Contentful Paint

Coverage mirrors the Playwright UI suite (`test/ui/pages.spec.ts`): the shell
(`.admin-layout`, `.sidebar`, gateway status LED, persistence lock, notification
bell) and every ADMIN page, plus the ROOT-only diagnostics page and its Fouria
DevOps + Cloudflare Components widgets.

The file also carries one **behavioral** case (driven directly through `plwr`)
that loads the diagnostics page against the local worker and asserts the
Cloudflare Components widget links the container card to its Cloudflare
dashboard page (`href` contains `dash.cloudflare.com`) and reveals its
binding-derived detail panel on hover (`visibility` flips `hidden` → `visible`,
panel text contains the `Runtime` fact). It self-skips when browser tests are
disabled or `plwr` is absent.

The file self-skips unless `E2E_LOCAL=true` (file-level `%skip`) and also
self-skips when `E2E_SKIP_BROWSER_TESTS` is set, so cloud runs stay green. The
`fixture/ui-render-measure` helper appends every record to
`ui-render-times.jsonl`; the final `ui-render-report` case writes a table to
`ui-render-report.txt` and prints a JSON rollup (stdout) with `maxComponentMs`
and the slowest component.

```bash
cd apps/fouria

# Just this suite — _setup.txt still starts the worker + browser:
bun run test:ui:render:local   # build:docker + E2E_LOCAL=true cctr -vv test/e2e/ -p 'ui render'

# Or run it as part of the whole local corpus:
bun run test:e2e:local
```

### E2E Agent Subset (OpenClaw migration smoke)

`apps/fouria/test/e2e-agent/` is a focused, browser-free subset of the CLI
surface used to validate an OpenClaw version migration. It reuses the shared
`test/e2e/fixture` (via symlink) and the same local-mode runner, but only runs
the commands the Worker shells out to:

- `b_cli_devices.txt` — `openclaw devices`
- `c_cli_models.txt` — `openclaw models`
- `d_cli_channels.txt` — `openclaw channels`
- `e_cli_config.txt` — `openclaw config validate`
- `g_cli_secrets.txt` — `openclaw secrets`
- `h_cli_knowledge.txt` — `openclaw wiki`
- `m_cli_agent_tokens.txt` — `openclaw agents` / token flows
- `n_gateway_soft_reload.txt` — gateway soft reload (`SIGUSR2`; asserts the Node inspector stays closed and proves the restart with a new `http server listening` line plus a stable `processId`)

> **Bounded agent turns.** `m_cli_agent_tokens.txt` bounds its Gateway turn with
> `openclaw agent --timeout 40`. The CLI's default turn deadline is 600s, which
> outlives the e2e HTTP budget (`debug-cli` waits up to 80s server-side and 90s
> via `curl-auth`; `/debug/cli` uses `timeout_ms`). A bounded turn returns its
> JSON envelope (result, timeout, or provider error) within the budget, and an
> empty response is reported as `{"status":"timeout"}` instead of failing the
> JSON-object assertion on an empty `jq` result.

```bash
cd apps/fouria
bun run test:e2e:agent   # build:docker + E2E_LOCAL=true + browser skipped
```

The [openclaw-migration](https://github.com/0xCAB0/fouria/blob/develop/skills/openclaw-migration/SKILL.md) skill runs this
subset after bumping `catalog.openclaw` to confirm the CLI commands still
behave.

### Gateway Boot Smoke Test (local)

Validates that the fouria sandbox image actually **boots a healthy OpenClaw
gateway** on the local machine — no Worker, no cloud e2e, no third-party
machines. It builds the exact production image (`apps/fouria/Dockerfile` +
repo-root context, same base `cloudflare/sandbox` image wrangler uses) and runs
the real boot path (`start-openclaw.sh` → onboard → plugin installs → moltlazy
patch/validate → gateway launch → health poll) inside Docker. The container's
exit code is the verdict: the in-container wrapper runs `start-openclaw.sh`
(whose own readiness loop already fails if the gateway dies or never binds),
then asserts `/health` returns `{ok: true}` while the gateway is still alive.

```bash
cd apps/fouria

# Default cold-boot smoke (no real credentials needed):
#   onboard runs --auth-choice skip, tunnel + OTEL steps self-skip.
bun run test:gateway:boot

# Standalone native phases in the smoke container: `start-openclaw.sh setup
# → start-openclaw.sh launch → post-start`, then assert /health. This is NOT
# the Worker's production cold start, which is separately `restore → atomic
# start (setup + Gateway under one lock) → post-start` (see the table below).
bun run test:gateway:boot --native

# Same, plus the real cloudflare-ai-gateway onboard path and an assertion
# that the provider plugin is installed + enabled. Sources credentials from
# apps/fouria/.dev.vars.e2e (never echoed) — generate it from SECRETS.yaml
# with `bun run secrets:generate`.
bun run test:gateway:boot:ai

# Leave the container running after the assertion, with port 18789
# published to 127.0.0.1 for manual inspection (Control UI / WS).
bun run test:gateway:boot --keep

# Skip build:docker + docker build and validate an existing image.
bash scripts/gateway-boot-smoke.sh --no-build --image <tag>
```

**Prerequisites:** Docker running, network access (the image build pulls the
ClawHub plugins; boot-time installs are a fallback only, bounded by
`start-openclaw.sh` timeouts and non-fatal for boot).

**What it validates vs. what it does not:**

| Validated locally                                                                                                                                                                                               | Still needs local e2e / cloud e2e                                 |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| Image builds from the production Dockerfile                                                                                                                                                                     | Sandbox SDK supervisor + Worker RPC layer                         |
| onboard (skip or real AI Gateway) completes                                                                                                                                                                     | RBAC / API routes / device pairing                                |
| Build-time plugin pre-install + enablement                                                                                                                                                                      | Dispatch worker routing                                           |
| moltlazy `patch` + `validate` (config contract)                                                                                                                                                                 | Cost observability / R2 backup flows                              |
| gateway binds `:18789`, `/health` returns ok                                                                                                                                                                    | Browser-driven Control UI flows                                   |
| `--native`: standalone `start-openclaw.sh setup` → `launch` → `post-start` phases in the smoke container (not the Worker's production `restore` → atomic start (setup + Gateway under one lock) → `post-start`) | Worker's `restore` + `sandbox.exec` supervision (handle/relaunch) |

### E2E Parallel (single-worker parallel runner)

`apps/fouria/test/e2e-parallel/` is the **parallel replacement for the serial
`test/e2e` run**. The `e2e` CI job no longer shards the corpus across N workers
(one per shard); it runs `run-parallel.sh`, which provisions **ONE** worker and
runs dependency-grouped cctr suites **concurrently** against it.

```bash
cd apps/fouria
bun run test:e2e:parallel        # == bash test/e2e-parallel/run-parallel.sh
```

**Suite scope (`E2E_BOOT_ONLY`).** By default the `e2e` CI job sets
`E2E_BOOT_ONLY=1` for **pull requests**: the runner provisions the worker, waits
for `/api/health` to report `startup.ok + readiness.ok`, and then tears down
**without running any cctr group** — a PR only proves the gateway boots. The full
corpus runs on `develop` (merge gate) and on a manual `workflow_dispatch` with
the **`full_tests`** input checked (the default for dispatch). Locally, set
`E2E_BOOT_ONLY=1` to reproduce the PR check.

Phases (see the suite README for the full grouping):

- `phaseA_devices` ∥ `phaseA_rbac` ∥ `phaseA_health` — parallel (the RBAC mock
  role is a per-request cookie, so the RBAC suites are independent; the two
  device-store suites share `phaseA_devices` to avoid racing the approve path)
- `phaseB_readonly` ∥ `phaseB_async` — parallel
- `phaseC_stateful` — config/secrets/cron (serial)
- `phaseD_lifecycle` — plugin + reload/restart + dispatch (serial)
- `phaseD_startup` — cross-isolate cold-start concurrency regression and the
  declarative software-provisioning suite (`zzz_software_provisioning.txt`:
  flag/report wiring, manifest validation + upload, cold reprovision installs the
  declared software, boot-terminal ticks, and an **on-demand** run that installs
  a newly declared package while skipping the already-installed one), serial,
  each with a separate startup-sized budget
- `phaseE_restore` — reprovision guard followed by the backup/restore lifecycle
  suites, resolved from `test/e2e-backup/` (serial, startup-sized budget).
  **Skipped in GitHub Actions by default** because it drives multi-minute cold
  boots and ~250 MB R2 snapshots; run it locally, or force it in CI with
  `E2E_RUN_RESTORE_PHASE=1`. For a standalone local run of just these suites use
  `bun run test:e2e:backup`.

> **Phase caps.** `phaseD_lifecycle` is bounded by `PHASE_D_TIMEOUT_SECS`
> (default **1200s / 20 min**; the recovery suites plus `provision_and_dispatch`
> now legitimately exceed the old 7-minute budget, and killing the group
> mid-dispatch left a restore in flight). The cold-start concurrency group has its own
> `PHASE_D_STARTUP_TIMEOUT_SECS` budget (default `STARTUP_TIMEOUT_SECS + 180s`);
> backup/restore suites use `PHASE_E_TIMEOUT_SECS` (default
> `STARTUP_TIMEOUT_SECS + 300s`). The full run's initial readiness budget is
> 1200s (`basic`) or 1500s (`production`). On a group timeout cctr fails
> (`exit 124`) and teardown still runs.

> **cctr 0.31.0 has no `#` comments.** cctr silently drops a corpus file's tests
> at the first top-level `#` line and reports the tests collected so far (often
> zero) with no error. Corpus files must begin with `===` (or a lone file-level
> `%skip`/`%shell`/`%platform` followed by the first `===`), and a `#` comment
> must never sit before the first `===`, between a file-level `%skip` and the
> first `===`, or between two tests. **Put rationale in test descriptions or docs,
> never in a `#` line** (comments _inside_ a test body/expected/`where` block are
> fine). `node scripts/check-corpus-comments.mjs` — run by the `unit` CI job —
> fails on any violating file, and `cctr -l <file>` prints the parsed test count.

> **Selective retry (`retry_failed`).** A slow container can make a single backup
> call exceed its curl budget, failing a suite that would pass on a warm run.
> Rather than inflate the curl timeouts (which hides real hangs), the workflow
> records the failed suites — the `Record failed e2e suites` step writes
> `✗ <group>/<suite>` markers to `/tmp/moltworker-e2e-failed.txt`, saved to the
> `e2e-failed-<ref>-<shard>-<run_id>` actions cache. Re-run **only those suites**
> with a manual `workflow_dispatch` and the **`retry_failed`** input checked:
> `run-parallel.sh` reads the restored file, links only the failed suites into
> their groups, skips every other group, and still provisions a single worker.
> The flag forces the suite to run (it overrides `full_tests`/boot-only). A
> `phaseX_group/*` entry (written when a group is killed by its phase cap)
> retries every suite in that group. The cache is saved only when something
> failed, so the restore always resolves to the most recent failing run.

> **Single-owner startup concurrency regression.**
> `ac_gateway_startup_concurrency.txt` is a cloud-only regression for the OpenClaw
> 2026.9.8 cross-isolate launch race: it checkpoints tenant state, destroys the
> container, fires several concurrent cold **non-passive** `/api/health` probes
> (the cold-boot fan-out across Worker isolates), then asserts the gateway reaches
> `startup.ok + readiness.ok`, that exactly **one** _running_ Gateway owner
> exists — the process may appear as `start-openclaw.sh start` before the atomic
> wrapper `exec`s, or as `openclaw gateway run` after — and that the observable
> gateway output never shows **either**
> `OpenClaw migration inputs changed during startup` **or**
> `state.sqlite is undergoing offline maintenance`. Because it destroys the
> shared worker's container it runs in `phaseD_startup`, after the ordinary
> lifecycle group and before restores, with its own startup-sized budget. It runs
> **serially** — never alongside the parallel phase A/B suites. Run it directly
> with `cctr apps/fouria/test/e2e/ac_gateway_startup_concurrency.txt`; it self-skips
> under `E2E_LOCAL=true`.

Each group is a generated cctr suite whose test files **symlink** the originals
in `test/e2e/` and whose `_setup.txt` **attaches** to the shared worker (copy
credentials → wait readiness → warm the CLI/device identity). It forces
`E2E_NO_CACHE=1` (full teardown, no container left billing) and self-skips under
`E2E_LOCAL=true`.

> **Readiness wait.** The driver's top-level wait uses the shared
> `e2e_wait_for_health` helper (immediate first probe, no fixed grace, bounded by
> `STARTUP_TIMEOUT_SECS`). The runner defaults that budget to **1200s**; the
> workflow overrides it per profile (basic **1200s**, production **1500s**). The
> generated per-group `_setup.txt` do **not** call the shared helper — each keeps
> its own grace-free, budget-bounded `/api/health` loop that attaches to the
> already-warm worker. Because the phases run concurrently against one worker,
> the per-group readiness savings do **not** sum to the same job-time reduction —
> wall time tracks the slowest phase, not the total of the individual waits.

`parallel-probe` + `a_parallel_spike.txt` remain as a raw-concurrency
**diagnostic** (how many concurrent CLI calls one worker sustains); run the
probe suite directly with `cctr -vv test/e2e-parallel/`.

**Logging policy.** Verbose transcripts — the provisioning output
(terraform/wrangler), each group's `cctr -vv` output, and the teardown/gateway
dumps — are written to `/tmp/moltworker-e2e-logs/*.log` and uploaded to the
`e2e-artifacts-<profile>` branch, so the CI job console only shows concise
`[parallel]` progress plus per-suite pass/fail markers and the summary. The
serial `test/e2e` `_teardown.txt` follows the same rule: gateway/doctor dumps
are written to disk only (never `tee`'d into the console).

See [`apps/fouria/test/e2e-parallel/README.md`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e-parallel/README.md)
for the per-file parallelizability classification, the ordering graph, and the
observed concurrency results.

### Backup Semantics E2E

`apps/fouria/test/e2e-backup/zzz_backup_semantics.txt` and
`.../zz_reprovision_guard.txt` prove the backup invariants against a real
container (local Docker or cloud) and real R2. They are registered in
`phaseE_restore` (serial; reprovision guard runs first) and are the E2E counterpart of the route/unit tests
(`src/routes/public.test.ts`, `src/routes/debug.test.ts`, `src/index.test.ts`) and
the dispatch fan-out unit test (`workers/fouria-dispatch/test/unit/cron-fanout.test.ts`).

Run the local-safe cases standalone (excluded from the default `test/e2e` corpus):

```bash
cd apps/fouria
bun run test:e2e:backup   # build:docker + E2E_LOCAL=true + browser skipped
```

See [`apps/fouria/test/e2e-backup/README.md`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e-backup/README.md).

### Declared-software restore E2E (`softwareProvisioning`)

`apps/fouria/test/e2e-backup/zzz_software_binaries_restore.txt` is the runnable
end-to-end example of a gateway whose agent skills need binaries that are **not**
in the image. The manifest
(`apps/fouria/test/e2e/fixture/software/gog-goplaces.json`) declares `gogcli`
(the `gog` binary for the `gog` skill) and `goplaces`, the two skills are
installed with `openclaw skills install skills-sh:openclaw/openclaw/{gog,goplaces}
--global`, and after a backup (`?inline=1`) → `/debug/destroy-container` →
restore it asserts that both binaries are back on `PATH` because the next
provisioning cycle re-installed them (the report `startedAt` advances), while the
skills come back from the snapshot rather than being reinstalled. It is
**local-only** (self-skips unless `E2E_LOCAL=true`) and needs container network
access (GitHub release downloads plus the ClawHub skills.sh resolver).

```bash
cd apps/fouria
E2E_LOCAL=true cctr -vv test/e2e-backup/zzz_software_binaries_restore.txt
```

The dependency-free equivalent — the same contract driven directly against the
sandbox image, without workerd/Miniflare — is
`bun run --cwd apps/fouria test:software:smoke`
(`scripts/software-restore-smoke.sh`): provision → install skills → snapshot →
fresh container → restore → re-provision, with five `PASS:` checks.

| Case                                                             | Invariant                       | Mode           |
| ---------------------------------------------------------------- | ------------------------------- | -------------- |
| health probes leave the catalog unchanged                        | I1 health is observability-only | local + cloud  |
| `POST /control` `X-FourIA-Control: backup` on a running instance | I2                              | local + cloud  |
| async sync commits a durable job and promotes a stable pointer   | I12 durable job + stable point  | local + cloud  |
| idle instance returns `skipped:'not-running'` without waking     | I2/I3                           | cloud          |
| control-backup → destroy → cold restore preserves a sentinel     | I4                              | cloud          |
| encryption at rest (`data.sqsh.enc`, no plaintext)               | I8                              | cloud (opt-in) |
| retention prunes catalog + objects to `BACKUP_MAX_VERSIONS`      | I9                              | cloud          |
| worker is bound only to its per-run bucket                       | I11                             | local + cloud  |
| reprovision 409 / `force` / running checkpoint                   | I5/I6/I7                        | cloud          |

### Promotion (dev → production) coverage

The former `test/e2e-push/` suite drove the **HTTP** Worker → Worker push
(`POST /api/admin/releases/promote` fetching `/api/admin/releases/import`). That
transport is gone: promotion is a **private R2 stage/pull** (the dev instance
writes `promotion/staged/<id>.tar.gz` + `promotion/pending.json` to its own
bucket; production reads it through `PROMOTION_SOURCE_BUCKET` on boot, or on
`POST /api/admin/releases/apply`). There is no import route to push to, and two
local Workers cannot share a miniflare R2 bucket, so the two-worker local harness
could not validate the new transport.

The transport is covered by:

- `src/services/promotion/r2-source.test.ts` — stage → pending → apply → applied
  pointer, content hash + commit message, staged/applied history entries, the
  merged dev+production history view, plus target/environment/tenant gating,
  already-applied, no tenant state, parked, busy, missing bundle, and idempotency.
- `src/routes/api/releases.test.ts` — `202` stage (message/author pass-through),
  production `409`, apply + checkpoint, status `available`, history, and RBAC.
- `src/index.test.ts` / `src/gateway/process.test.ts` — the bridge registration
  and the boot-time apply hook.

The dev/production axis remains the `-dev` **hostname**
(`isDevAgentsEnabled`), not `DEV_MODE`.

Cloud helpers: `test/e2e/fixture/r2-list` and `r2-clear` list/empty the per-run
bucket via the R2 REST API (needs `CLOUDFLARE_TEARDOWN_API_TOKEN` or
`CLOUDFLARE_API_TOKEN`). The E2E deploy sets deterministic
`BACKUP_INTERVAL_MINUTES=1` / `BACKUP_MAX_VERSIONS=2`; set
`E2E_BACKUP_ENCRYPTION=true` to run the encrypted-at-rest case. Note that the
configured interval is a **floor**: the Worker stretches the effective scheduled
cadence to `max(BACKUP_INTERVAL_MINUTES, 2 × the last snapshot duration)`, so a
slow snapshot cannot run back-to-back. Explicit snapshots (`POST
/api/admin/storage/sync`, checkpoints) bypass that gate and stay immediate.

The same invariants are asserted at the fast layer by
`workers/fouria-dispatch/test/unit/cron-fanout.test.ts` (the cron forwards
`backup` only to non-frozen tenants) and `src/persistence.test.ts` (the “stable
recovery point promotion” block: the recovery pointer advances only, fails
closed on a malformed pointer, and retention skips the prune when the pointer is
unreadable). Manual-restore hardening is unit-tested in `src/persistence.test.ts`
too: a `setRecoveryPoint: false` restore leaves the pointer untouched, an
explicit restore opens the maintenance window and stops the gateway while it
runs, and `restoreIfNeeded` defers while that window is active. The route’s
`?setCurrent=0` mapping is covered by `src/routes/api/storage.test.ts`, the
durable-flag round-trip by `src/activity/backup-jobs.test.ts` /
`src/activity/scheduler.test.ts`, and the Backups-page checkbox by
`src/client/components/__tests__/BackupHistoryPanel.test.tsx`, plus the
Cloudflare `@cloudflare/vitest-plugin` route tests planned in
[`apps/fouria/test/e2e/BACKUP-E2E-DESIGN.md`](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e/BACKUP-E2E-DESIGN.md).

Restore legibility (which backup/date is loading) is covered by
`src/persistence.test.ts` (`readRecoveryState`; the restore target is written
during `sandbox.restoreBackup` and cleared on success), `src/assets/loading.test.ts`
(the loading page consumes `data.recovery` / `recovery.restoring` /
`recovery.committed`), and `src/client/components/__tests__/BackupHistoryPanel.test.tsx`
(the recovery-point badge + summary). The local observability canary
`test/e2e/ad_observability_canary.txt` shows how to query the local dev server's
Local Explorer SQL API (`test/e2e/fixture/observability-query`) to confirm the
restore-loop guards (`Restore deferred — tenant maintenance in progress`,
`GatewayMaintenanceError`) fire from the worker's own logs.

> **Vitest 5 caveat.** The Cloudflare route-integration layer is deferred: both
> `@cloudflare/vitest-pool-workers` and the renamed `@cloudflare/vitest-plugin`
> peer `vitest ^4.1.0`, and under Vitest 5 the pool fails with
> `SyntaxError: Unexpected identifier 'file'` (known open issue
> [cloudflare/workers-sdk#15618](https://github.com/cloudflare/workers-sdk/issues/15618);
> workaround: pin `vitest ~4.1.11`).

> **Restore sequencing.** Cold‑restore survival (I4) is **opt‑in** via
> `E2E_BACKUP_RESTORE=1` (`zzz_backup_cold_restore.txt`) because restoring the
> ~250 MB local‑binding archive is far too slow for CI (observed >13 min without
> completing). The default `phaseE_restore` cases (catalog, health‑observability,
> running‑backup, idle‑skip, bucket isolation, encryption, retention) do not
> restore and use `fixture/wait-ready` (one warm request, then `?passive=1` polls)
> so overlapping restores cannot abort each other with
> `OperationInterruptedError: … runtime changed` (root cause:
> [`BACKUP-E2E-DESIGN.md` §9](https://github.com/0xCAB0/fouria/blob/develop/apps/fouria/test/e2e/BACKUP-E2E-DESIGN.md)).

> **cctr corpus gotcha.** cctr 0.31 silently registers **zero tests** for a
> `*.txt` corpus file whose first line is a `#` comment, or when comments sit
> between a file-level `%skip` and the first `===`. Corpus files must begin with
> `===` (or a lone directive followed by `===`); keep prose in test bodies or at
> the end of the file. `cctr -l <file>` prints the parsed test count.

### Business Rules Corpus

Contract tests at `apps/fouria/test/business-rules/` that validate cross-cutting concerns. These form the "T0 validation gate" per issue #137.

```bash
cd apps/fouria
cctr test/business-rules/
```

| Corpus File                  | Validates                                                                                                                                                                                                                                                                                             |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pricing.txt`                | Per-action billing, success-only, BYOL discounts, VISION.md alignment                                                                                                                                                                                                                                 |
| `rbac.txt`                   | ROOT/ADMIN/BASE_USER permission matrix, route guards, and dashboard + loading-page endpoint coverage                                                                                                                                                                                                  |
| `tenant-isolation.txt`       | Cross-tenant access controls, `canAccessCustomer`                                                                                                                                                                                                                                                     |
| `operational-modes.txt`      | DEV_MODE, E2E_TEST_MODE, DEBUG_ROUTES, DEMO_MODE, running-only backups, health observability-only, late-ready gateway adoption, health is not a routing gate (ROOT-revivable frozen), reprovision fail-closed guard, automatic `doctor --fix` repair for a launch blocked by a corrupt state database |
| `cost-tracking.txt`          | 6 Analytics Engine event types, field schemas                                                                                                                                                                                                                                                         |
| `dashboard-api-contract.txt` | Admin API endpoint existence                                                                                                                                                                                                                                                                          |
| `agent-visibility.txt`       | BASE_USER agent visibility rules                                                                                                                                                                                                                                                                      |
| `credits-and-plans.txt`      | Credit model, plan limits, BYOL exclusion, p95 pricing                                                                                                                                                                                                                                                |

---

## apps/lerma — Phoenix Web App

### Unit Tests

Mirror `lib/` structure under `test/lerma/` and `test/lerma_web/`. Run with ExUnit via mix.

```bash
cd apps/lerma

# All unit tests (creates database, runs migrations)
mix test

# Single file
mix test test/lerma/clients_test.exs

# From the repository root: compile + deps.unlock + format + Credo + tests
cd ../..
mix precommit
```

**Key test files:**

| File                                                          | Tests                                                                                                         |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `lerma/clients/client_test.exs`                               | Client CRUD, validation, Ecto schema                                                                          |
| `lerma/clients/user_test.exs`                                 | User model, invitation lifecycle fields                                                                       |
| `lerma/clients_test.exs`                                      | Invite/accept/reject, instance-scoped Access sync, reserved staging-client protections                        |
| `lerma/applications_test.exs`                                 | Application CRUD, one-time provisioning key generation/rotation/revocation and digest-only storage            |
| `lerma/staging_tenant_test.exs`                               | Staging-only `client` bootstrap, production isolation, and database delete/rename/truncate guards             |
| `lerma/emails_test.exs`                                       | Confirmation email content and links                                                                          |
| `lerma/invitations_test.exs`                                  | Background invitation email delivery and PubSub alert                                                         |
| `lerma/instances/instance_test.exs`                           | Instance lifecycle                                                                                            |
| `lerma/instances_upgrade_test.exs`                            | Upgrade version guards, per-application filtering, and repeat-promotion idempotency                           |
| `lerma/provisioning/tenant_provisioner_test.exs`              | Tenant provisioning logic                                                                                     |
| `lerma/billing_test.exs`                                      | Billing calculations                                                                                          |
| `lerma/billing_plans_test.exs`                                | Global plan templates, assign/clone, autobuy toggle                                                           |
| `lerma/billing_rules_corpus_test.exs`                         | Billing business rules (a–m) incl. production gating                                                          |
| `lerma/billing_local_workflow_test.exs`                       | Local workflow: AE rows → reconcile → consume; configured/unconfigured/duplicate events, watermark ownership  |
| `lerma/billing_becaria_trace_test.exs`                        | BecarIA flow: AE rows → reconcile → exact per-action credits + sums                                           |
| `lerma/credits/credit_pool_test.exs`                          | Credit pool management                                                                                        |
| `lerma_web/live/billing_plans_live_test.exs`                  | Global plan library + designer                                                                                |
| `lerma_web/components/billing_plan_component_test.exs`        | Client board plan assignment + summary                                                                        |
| `lerma/analytics/cf_analytics_test.exs`                       | Cloudflare Analytics Engine queries                                                                           |
| `lerma/observability/otlp_test.exs`                           | OTLP/HTTP JSON parsing, allowlist privacy guard, child-span drop, caps, stable dedupe keys                    |
| `lerma/observability/telemetry_test.exs`                      | Idempotent destination ingestion, uptime/failure ratios (unknown ≠ downtime), window helpers, retention prune |
| `lerma/cloudflare/observability_test.exs`                     | Worker upload metadata: destination names, per-signal sampling, `persist: false`, disabled switch             |
| `lerma/cloudflare/observability_destinations_test.exs`        | Destination reconcile: create/PATCH staging + production pairs, 403/500 handling, env-specific names          |
| `lerma_web/controllers/api/otlp_ingest_controller_test.exs`   | Destination ingest auth (fail closed), `415` for protobuf, gzip delivery, privacy drop, replay idempotency    |
| `lerma_web/plugs/otlp_body_reader_test.exs`                   | Gzip/JSON body decoding, size caps, malformed-body 400, path/method passthrough                               |
| `lerma_web/controllers/api/observability_controller_test.exs` | Uptime endpoint ratios/status/windows, unknown when no observations, sampling rate surfaced                   |
| `lerma_web/components/worker_telemetry_widget_test.exs`       | Worker telemetry widget unknown/healthy/degraded render                                                       |
| `lerma_web/components/health_events_widget_test.exs`          | Health events timeline empty state, recorded transitions, dev-instance rendering                              |
| `lerma_web/controllers/api/instance_controller_test.exs`      | Instance API endpoints                                                                                        |
| `lerma_web/controllers/api/provisioning_controller_test.exs`  | Global/application-key auth, application-scoped upgrade and status                                            |
| `lerma_web/controllers/api/ai_gateway_controller_test.exs`    | Global provisioning-key auth; fails closed when unset                                                         |
| `lerma_web/controllers/api/credit_usage_controller_test.exs`  | Billing reconcile + credit-usage API (bearer auth)                                                            |
| `lerma_web/live/instance_live_test.exs`                       | Instance LiveView                                                                                             |
| `lerma/client_docs_test.exs`                                  | Client docs sources (proposal + per-env stores), R2 listing/reading, default document formats, path traversal |
| `lerma/client_docs/markdown_test.exs`                         | Safe Markdown preview: frontmatter, headings, tables, lists, code, relative images, unsafe links, HTML escape |
| `lerma_web/controllers/client_docs_controller_test.exs`       | Raw docs responses: content type, inline/download disposition, sandbox CSP, not-found/invalid-path errors     |
| `lerma_web/live/client_live_test.exs`                         | Invitation UI, public acceptance flow, and the lazy documentation widget tree/preview/live                    |
| `lerma_web/live/application_live_test.exs`                    | ROOT-only application key generation and one-time reveal                                                      |
| `lerma_web/plugs/cf_access_auth_test.exs`                     | CF Access auth plug                                                                                           |

**Per-test tags:**

- `@tag cf_auth: %{email: "...", name: "..."}` — injects CF Access authentication
- `@tag :integration` — excluded by default, must run explicitly via `mix test --only integration`

### Staging Fouria build promotion

`.github/workflows/promote-staging.yml` listens for completed `Test` runs on
`develop`, then queries the source run's Actions jobs and proceeds only when
`build-and-push` concluded `success`. The script promotes the immutable
`unstable-<sha>` image/bundle pair, no-ops when the staging `client` tenant is
already running that version, and polls Lerma's status endpoint until the
running application instances converge. The GitHub secret
`STAGING_PROVISIONING_API_KEY` must match the one-time generated key for the
Fouria application in staging Lerma. This cloud workflow is not run by local
`mix test`; the application-scope/auth/idempotency contracts are covered by the
ExUnit tests above.

### E2E Tests (cctr)

Sequential cctr tests against real Cloudflare dispatch namespace infrastructure.

```bash
cd apps/lerma
mix test.e2e
```

**Setup** (`test/e2e/_setup.txt`):

1. Validates `jq`, `uuidgen` tools
2. Sources `.dev.vars` for Cloudflare credentials
3. Validates `CLOUDFLARE_API_TOKEN`, `CF_ACCOUNT_ID`, `WORKERS_SUBDOMAIN`, `CF_ENVIRONMENT`
4. Computes dispatch namespace (`fouria-tenants` vs `fouria-tenants-staging`)
5. Starts Phoenix on port 4002 via `mix phx.server`

**Teardown** (`test/e2e/_teardown.txt`):

1. Reads client/instance IDs from fixture directory
2. Deletes instance (tears down CF worker + container application) via API
3. Deletes client via API — fully decommissions the tenant (remaining
   instances and container applications, Access app/group, dedicated AI
   Gateways, R2 backup buckets) and archives the client record
4. Kills Phoenix process

**Test files:**

| File                      | Tests                                                                                          |
| ------------------------- | ---------------------------------------------------------------------------------------------- |
| `client_provisioning.txt` | Create client → create instance → wait for running → verify gateway → verify CF metadata       |
| `env_isolation.txt`       | Correct namespace, wrong namespace rejection, environment tags, Analytics Engine dataset names |

### Billing trace: deterministic + local

The former **staging** billing-trace suite (`test/e2e_billing_trace/`) drove a
deployed worker's public `POST /api/otel/v1/traces`. That endpoint no longer
exists: OTLP ingest is the **sandbox bridge** (`http://otel.internal/v1/traces`),
resolved by the Sandbox SDK inside the Worker, so it is only reachable from
inside the tenant container. There is no public ingest route and no Access path
app to configure.

The five BecarIA actions (`telegram_query`, `queue_reevaluation`,
`signal_analysis`, `email_draft`, `review_analysis`, 1 + 1 + 3 + 5 + 10 = 20) are
still covered end to end by:

- `test/lerma/billing_becaria_trace_test.exs` (deterministic, stubbed Analytics
  Engine stream, runs in CI), and
- `test/e2e_billing_local/` (`mix test.e2e.billing:local`, fake Analytics Engine,
  no Cloudflare account).

For a manual replay inside a container, run
`apps/fouria/scripts/emit-billing-actions.ts` with
`--base-url http://otel.internal` (the bridge keeps `OTEL_SHARED_SECRET`).
`ANALYTICS_ENGINE_API_TOKEN` is Worker-only and irrelevant to emission — the
Worker writes the billable row with its `ANALYTICS` binding; the token only
powers the dashboard read-back and is never given to a container.

### Local Billing-Workflow E2E (fake Analytics Engine)

`test/e2e_billing_local/` (opt-in, run separately from `mix test.e2e`) exercises
the billing consume path entirely locally — **no Cloudflare account, no staging
DB, no deployed worker**:

```
fake Analytics Engine (bun)
  └─ POST /accounts/:id/analytics_engine/sql   ← BillingReconcile reads here
       └─ lerma (mix phx.server, MIX_ENV=test)
            └─ POST /api/credits/v1/reconcile → BillingConsumption → GET /usage
```

It uses a **mock billing definition** (`fixture/seed_plan.exs`, not BecarIA) and
feeds synthetic `billable-action` AE rows directly (OpenClaw's OTel emission is
owned/tested by the OpenClaw team). Cases: configured + `match_rule` events,
unconfigured/unmatched/blank events, duplicate `action_id` redelivery, AE outage
(502, no billing), and tenant isolation.

```bash
cd apps/lerma
mix test.e2e.billing:local
```

**Why a fake AE and not Cloudflare Local Explorer?** Local Explorer
([docs](https://developers.cloudflare.com/workers/local-development/local-explorer/))
does not support Analytics Engine — it is absent from the supported-bindings
table and from the Local Explorer OpenAPI, and local traces emit no
`writeDataPoint` span. A fake of the AE SQL API is therefore the only local
observability surface for the billable-event stream. Local Explorer _is_ used on
the Worker side for logs/spans (`apps/fouria/test/e2e/ad_observability_canary.txt`).
See `apps/lerma/test/e2e_billing_local/README.md`.

### Test Infrastructure Configuration

**`apps/lerma/mix.exs`:**

```elixir
test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"]
"test.e2e": ["cmd cctr test/e2e/ --sequential -v"]
```

**Root `mix.exs`:** The `credo` alias forwards all arguments to each Elixir
workspace project, so `mix credo --strict` works from the repository root without
running Credo against the root workspace shell project.

```elixir
credo: &run_credo/1,
check: [
  "workspace.run -t format -- --check-formatted",
  "credo --strict"
],
precommit: [
  "workspace.run -t compile -- --warnings-as-errors",
  "workspace.run -t deps.unlock -- --unused",
  "workspace.run -t format",
  "credo --strict",
  "license_audit",
  "workspace.run -t test"
]

# Private alias helper:
defp run_credo(args) do
  args = if "--strict" in args, do: args, else: ["--strict" | args]
  Mix.Task.run("workspace.run", ["-t", "credo", "--" | args])
end
```

**`test_helper.exs`:** Excludes `:integration` tests, starts SQL sandbox.

**Mocking:** Uses `Bypass` for HTTP mocking and `Meck` for Elixir mocking. Not available outside `:test` env.

---

## packages/moltlazy — Config Module + SDK

### Unit Tests (Vitest)

```bash
cd packages/moltlazy
bun run test          # vitest run
bun run test:watch    # watch mode
```

**Config module tests** (`tests/config/`):

| File               | Tests                                                               |
| ------------------ | ------------------------------------------------------------------- |
| `gateway.test.ts`  | Gateway config section (port, mode, trustedProxies, allowedOrigins) |
| `session.test.ts`  | Session config (dmScope)                                            |
| `tools.test.ts`    | Tool search configuration                                           |
| `logging.test.ts`  | Logging section (level, format, transports)                         |
| `validate.test.ts` | CLI validation-error formatting (object-shaped `openclaw` errors)   |
| `include.test.ts`  | `$include` directive injection                                      |

**SDK module tests** (`tests/`):

| File                            | Tests                                 |
| ------------------------------- | ------------------------------------- |
| `sdk.test.ts`                   | SDK client initialization             |
| `sdk-cli.test.ts`               | SDK CLI fallback mechanisms           |
| `sdk-gateway-status-ws.test.ts` | Gateway WebSocket status checks       |
| `sdk-ws-client.test.ts`         | WebSocket client connection lifecycle |

**Agent tests** (`tests/`):

| File                    | Tests                                      |
| ----------------------- | ------------------------------------------ |
| `agents.test.ts`        | Agent defaults, list generation            |
| `index.test.ts`         | Main module (patch flow, section ordering) |
| `schema.test.ts`        | Zod schema validation for moltlazy config  |
| `obsidian-sync.test.ts` | Obsidian sync skill configuration          |
| `coverage-gaps.test.ts` | Coverage gap tracking across modules       |

### E2E Tests (cctr)

```bash
cd packages/moltlazy
bun run test:e2e        # Build + cctr
```

| Test Suite                | Scenarios                                                           |
| ------------------------- | ------------------------------------------------------------------- |
| `cli/basic.txt`           | `moltlazy patch` from empty, `--help` displays man-style docs, `-h` |
| `cli/flags.txt`           | `--file` override, `--help`, unknown commands                       |
| `cli/errors.txt`          | Validate pass/fail, nonexistent file, filesystem errors             |
| `full-flow/roundtrip.txt` | Full patch → verify output (gateway, `$include`, agents, session)   |
| `skills/discovery.txt`    | Skills directory discovery, empty skills dir, re-patching           |
| `workspace/seeding.txt`   | SOUL.md creation, fouria-builder workspace, safe mode, immutability |

**Setup:** Validates that `dist/cli.js` exists (pre-build check).

> **OpenClaw CLI version:** the `moltlazy-integration` CI job validates the
> generated config with the OpenClaw CLI **pinned to the version baked into the
> container** (root `package.json` catalog, currently `2026.7.1-2`) instead of
> `openclaw@latest`. This keeps the config-schema contract aligned with what is
> actually deployed and avoids flaky failures from upstream schema drift. Bump
> the pin in `.github/workflows/test.yml`
> and this doc together whenever the container's OpenClaw version changes.

### Smoke Tests (Vitest)

```bash
cd packages/moltlazy
bun run test:all       # Unit + E2E
```

---

## OpenClaw plugins (external)

The `cloudflare-unified-billing` plugin — its unit tests, live AI Gateway
integration tests (`infer-smoke`, `provider-smoke`, `cf-cost-api`), and CI —
moved to [Paso4/openclaw-plugins](https://github.com/Paso4/openclaw-plugins).
Run them from a checkout of that repo:

```bash
cd plugins/provider/cloudflare-unified-billing
bun run test              # unit
bun run test:integration  # live (requires AI Gateway creds + PLUGIN_LIVE_TESTS=true)
```

See [OPENCLAW-PLUGINS.md](/llms/platform/libraries/openclaw-plugins/index.md).

---

## CI Pipeline

The `.github/workflows/test.yml` workflow defines the CI pipeline (see reference table in [AGENTS.md](https://github.com/0xCAB0/fouria/blob/develop/AGENTS.md) for the full job graph):

```
unit ──┬─ build-and-push → e2e   (PR: single shard · develop: sharded matrix)
       └─ moltlazy-integration   (only when packages/moltlazy changed)

changes ── iac-test              (only when apps/*/iac changed)
```

The `changes` job git-diffs the event SHAs and gates `iac-test` and
`moltlazy-integration`, so unrelated changes no longer spin them up. The
`local-integration` job was removed.

**The Changesets version pull request runs no CI.** The "Version Packages" PR is
opened from the `changeset-release/develop` branch by
`.github/workflows/version-packages.yml`; it only bumps versions and writes
changelogs, so every PR-triggered workflow exits immediately for that head
branch — `test.yml` (`changes`, `unit`, `validate-secrets`, `build-and-push`),
`lerma.yml` (`quality`, `test`), `staging-validation-ci.yml` and
`changeset-status.yml`. The guard is the head-ref prefix
`!startsWith(github.head_ref, 'changeset-release/')`, which is empty on
`push`/`workflow_dispatch`, so those paths are unaffected. Skipped jobs report as
**skipped** (not pending) to branch protection, and no CD workflow can be
affected: both `promote-staging.yml` and `deploy-dashboard.yml` only ever act on
a successful run whose `head_branch` is `develop`.

`build-and-push` builds the worker, validates the wrangler multipart bundle via
`scripts/validate-bundle.mjs`, then builds the container image and pushes it to
the Cloudflare registry as `fouria:unstable-<sha>` and uploads the worker
bundle to the `fouria-bundles` R2 bucket as `unstable-<sha>/fouria.mjs`. On
develop, `e2e` consumes that image (via `IMAGE_REF`) instead of rebuilding it.
The `deploy-dispatch.yml` workflow also deploys `workers/fouria-dispatch`
(staging on push to develop, production on a stable CalVer release or a manual
`workflow_dispatch`). `deploy-fouria.yml` is the fouria CD: on a stable CalVer
release (`vYYYY.M.D` / same-day `vYYYY.M.D-N`, classified from the tag name, not
the GitHub `prerelease` flag) it **promotes the CI-built `unstable-<sha>`
artifacts** — retagging that image to `latest` + the release tag and copying the
bundle to R2 under the release tag + `latest`. Nothing is rebuilt. Release
commits are pushed with `[skip ci]`, so they never
re-trigger this CI pipeline.

Production deploys are stable-tag-only: lerma filters the deployable versions to
stable release tags and `Instances.update_instance_version/3` rejects unstable
ones (staging keeps the unstable channel). The `cleanup-stale-images.yml` cron
never deletes an image (or matching R2 bundle) still referenced by a container
application — including in-progress rollout versions — and fails closed if that
in-use set cannot be resolved. Covered by
`apps/lerma/test/lerma/cloudflare/image_registry_test.exs`,
`apps/lerma/test/lerma/cloudflare/registry_bundle_test.exs`,
`apps/lerma/test/lerma/instances_test.exs`, and
`apps/lerma/test/lerma_web/live/instance_live_test.exs`.

Container image upgrades roll out with `strategy: "rolling"` (a single
`step_percentage: 100` step) — see `Lerma.Cloudflare.Containers`. The tenant
container is Durable-Object-managed, and a `new_instances` rollout does **not**
replace a DO-bound instance: the platform stages the target as a rollout version
at 0% and the application keeps its previous image, so `new_instances` silently
left promoted tenants running the old image. Promotion convergence is also
sourced from the **live** container image tag (`Containers.fetch_deployed_config/1`)
in `Instances.at_requested_version?/2`, not the recorded `deployed_version`, so a
promotion cannot report success while the container still runs the previous
image (and a genuine miss can be re-promoted instead of being skipped). Covered
by `apps/lerma/test/lerma/cloudflare/containers_test.exs` and
`apps/lerma/test/lerma/instances_upgrade_test.exs`.

### Plugin Integration (unified billing e2e) — moved

The `cloudflare-unified-billing` e2e tests and their dedicated workflow moved to
[Paso4/openclaw-plugins](https://github.com/Paso4/openclaw-plugins). That repo
runs live AI Gateway integration tests on `develop` only (they cost real
credits), decoupled from this monorepo's `test.yml`.

**Key env vars for CI:**

- `SKIP_CLI_E2E=true` — used in `unit` job to skip CLI-dependent tests
- `E2E_SKIP_BROWSER_TESTS=1` — skip Playwright browser tests
- `RUN_REASONING_TESTS`, `RUN_VISION_TESTS` — gate expensive integration tests

---

## Writing Tests

### TypeScript (Vitest)

- **Colocate:** Place `*.test.ts` next to source files in `apps/fouria/src/` and `packages/moltlazy/tests/`
- **Strict mode:** Tests use `strict: true` TypeScript
- **Mocking:** Use vitest's `vi.mock()` for module-level mocking
- **Lifecycle:** `beforeAll`/`afterAll` for setup/teardown; `beforeEach`/`afterEach` for per-test state
- **Pattern:** Arrange → Act → Assert (AAA)
- **Coverage:** ≥80% for new code (≥95% for API routes in `apps/fouria`)

### Elixir (ExUnit)

- **Mirror structure:** Test files mirror `lib/` paths under `test/`
- **Database:** Use `Ecto.Adapters.SQL.Sandbox` for concurrent DB tests
- **HTTP mocking:** Use `Bypass` for external HTTP requests
- **Auth helpers:** Use `@tag cf_auth` in `conn_case.ex` to inject CF Access headers
- **Integration tests:** Tag with `@tag :integration` (excluded by default)
- **Fixtures:** Define in `test/support/fixtures/`
- **Convention:** `describe "function_name/arity"` blocks; test names as strings

### cctr (Corpus Contract Tests)

- **Format:** Plain text test files with `===` (description) → script → `---` → assertion sections
- **%require:** First test case in a file with `%require` cascades failures to skip remaining tests
- **Variables:** Capture dynamic output with `{{ name }}` — used for tokens, URLs, etc.
- **Assertions:** Use `where` clauses (`* s contains "ready"`, `* status == "200"`)
- **Fixtures:** The `fixture/` directory is mirrored to `$CCTR_FIXTURE_DIR`
- **Setup/teardown:** `_setup.txt` runs before the suite, `_teardown.txt` after
- **Shared helpers:** `apps/fouria/test/lib/e2e-helpers.sh` provides
  `e2e_wait_for_health` (immediate, bounded readiness probe) and
  `e2e_classify_ws_upgrade` (WS-upgrade verdict), unit-tested in
  `apps/fouria/src/e2e-helpers.test.ts`. `e2e_wait_for_health` is called by the
  serial (`test/e2e`) and agent (`test/e2e-agent`) `_setup.txt` corpora and by
  the parallel driver's top-level wait (`test/e2e-parallel/run-parallel.sh`).
  `ab_gateway_ws_origin.txt` sources the library only for
  `e2e_classify_ws_upgrade` and keeps its own health poll; the parallel driver's
  generated group `_setup.txt` also keeps its own grace-free bounded health poll.
- **Local mode guard:** Tests that require cloud infrastructure use:
  ```bash
  if [ "${E2E_LOCAL:-}" = "true" ]; then
      echo "SKIP: ..." >&2
      exit 1
  fi
  ```

### Business Rules

- **Location:** `apps/fouria/test/business-rules/`
- **Format:** cctr `.txt` files
- **Purpose:** Cross-cutting behavioral contracts (pricing, RBAC, tenant isolation, etc.)
- **When to update:** Any change affecting pricing, RBAC, tenant isolation, operational modes, cost tracking, dashboard API, agent visibility, or credits/plans
- **Run:** `cctr apps/fouria/test/business-rules/`

---

## Validation Checklist

Before submitting changes that modify test infrastructure or add new tests:

- [ ] `bun run typecheck` passes
- [ ] `bun run lint` passes
- [ ] `bun run test` passes (all JS/TS packages)
- [ ] For `apps/fouria`: `bun run test:coverage` passes
- [ ] For `apps/lerma`: `mix test` passes
- [ ] New code has ≥80% unit test coverage (≥95% for API routes)
- [ ] If adding routes: add tests
- [ ] If changing env vars: update `.dev.vars.example`, [ENV-VARIABLES.md](/llms/platform/reference/env-variables/index.md), and `apps/fouria/src/gateway/env.ts`
- [ ] If changing business rules: update corresponding corpus file and run `cctr apps/fouria/test/business-rules/`
- [ ] For `apps/fouria` local E2E: `bun run test:e2e:local:cli` passes (cloud-only tests self-skip)
