F FourIA staging 2026.10.29 GitHub ↗

Testing Guide

On this page

Testing infrastructure reference for all FourIA products. See CONTRIBUTING.md for contribution workflow and DEVELOPMENT.md for onboarding.

Overview

ProductTypeFrameworkUnit TestsE2E Tests
apps/fouriaCloudflare WorkerVitest + cctr + PlaywrightColocated *.test.tsCloud + Local + Business Rules
apps/lermaPhoenix Web AppExUnit + cctrMirror lib/ structureSequential cctr scenarios
packages/moltlazyJS Config + SDKVitest + cctrtests/*.test.tscctr CLI scenarios

The cloudflare-unified-billing plugin’s unit and live-integration tests now live in Paso4/openclaw-plugins.


Quick Start

# All JS/TS tests (fouria + moltlazy)
bun run test

# fouria admin SPA UI tests (Playwright, local, no credentials)
bun run --cwd apps/fouria test:ui

# TypeScript type checking (Worker, moltlazy, cost-calculator, dispatch, tail)
bun run typecheck

# Lint all TS/JS sources
bun run lint

# Full CI suite (test + typecheck + local integration + moltlazy E2E)
bun run test:all

See each product section below for product-specific commands.

Release tooling tests

scripts/changeset-changelog.test.ts, scripts/release-lib.test.ts, scripts/sync-versions.test.ts, and scripts/version-packages.test.ts cover the Changesets changelog format, the release-notes renderer, CalVer changelog generation, the Lerma package.json → mix.exs version sync, and the no-changesets no-op path. They run as part of bun run test.

Sandbox v1 migration plan

The v1-sandbox track carries a machine-readable evaluation plan at docs/plans/sandbox-v1-evaluation-plan.json (human summary: sandbox-v1-evaluation-plan.md). It maps every implemented V1 capability to a domain, a milestone and a v1 disposition, and encodes the five-step protocol each milestone executes.

bun run plan:sandbox-v1          # regenerate from docs/releases/V1-RELEASE-NOTES.md
bun run plan:sandbox-v1:check    # stale-plan + structural + coverage gate
bun run plan:sandbox-v1:sync     # merge origin/develop into v1-sandbox

The plan’s e2e_suitability block is the source of truth for which existing suites apply to the v1 runtime and which new apps/fouria/test/e2e-v1/** scenarios a milestone must add. Staging validation targets the Lerma client client-sandbox-v1 (id 20), never the reserved client slug or production.

Sandbox v1 live drills

apps/fouria/test/e2e-v1/ is a corpus of live drills against a deployed tenant (default skip unless E2E_V1_LIVE=1). Each suite drives the shared fixture/v1-drill (one JSON object per mode) and self-skips when the live environment is absent, so collection is always safe:

SuiteCovers
00_runtime_selection.txtServed lane + lifecycle both report v1
01_fresh_boot_and_migration.txtReal readiness after a boot and a committed recovery point (C8)
02_concurrency_and_serving_owner.txtA probe burst folds into exactly one gateway owner
03_passive_health_and_wake.txtNon-waking passive probe, idle stop → automation wake, keep-alive policy
04_channels_and_rpc_ws.txtAgent/model/channel CLI over the v1 process shim
05_freeze_and_control.txtFreeze snapshots then destroys; a probe wakes; unfreeze is a no-op
06_backup_restore_retention.txtA record taken before a destroy is the record restored
07_canary_gates.txtThe served bundle identity matches the promoted build; not degraded

05/06 destroy the container and require E2E_V1_DESTRUCTIVE=1. Run it with bun run --cwd apps/fouria test:e2e:v1:live; see apps/fouria/test/e2e-v1/README.md for prerequisites, per-suite timing knobs and the temporary billable procedure. Use a dev instance: Lerma’s HealthMonitor polls production clients every 2 minutes and a passive probe is still a Durable Object request that prevents the v1 inactivity stop.

The business-rules corpus also covers the v1 admin/billing/cost surface without a live tenant: apps/fouria/test/business-rules/sandbox-v1-{agents,billing,cost}.txt pin the runtime-independent route suites against the v1 transport tests (process shim, DO entrypoint, single owner), the OTLP/billable emit path, the freeze semantics and the cost-summary query. src/sandbox-v1-outbound.test.ts pins the v1 outbound host set and the “a host never throws” contract (an escaping exception is an opaque ECONNRESET in the container).

The gateway proxy attribution fallback is unit-tested without a live tenant: apps/fouria/src/gateway/proxy-attribution.test.ts covers the proxy-shaped header set and the 403 + proxy_attribution_required payload classifier, and the gateway proxy attribution block in src/sandbox-v1.test.ts covers the transport (direct fallback once the probe is refused, cached verdict, the forwarded headers kept when attribution is accepted or the probe cannot reach a booting gateway, no probe for a request without forwarded headers, and the re-probe after v1Destroy).

Sandbox v1 adversarial / fuzz suites

Two seeded suites attack the Sandbox v1 critical pipelines (zero data loss, secrets, memory, boot). apps/fouria/src/sandbox-migration/fuzz.test.ts covers the pure state machines (tri-state reads, validators, the C8 recovery-point barrier, retention, the migration lease, runtime-marker authority and the runtime intent gate); apps/fouria/src/sandbox-v1-adversarial.test.ts covers the SandboxV1 Durable Object (encryption fail-closed matrix, secret non-leakage, failed-snapshot recovery-point safety, post-commit retention, cadence clock skew, boot refusals, the process-map bound, the transport port guard and randomized catalog integrity). A third, deterministic suite — apps/fouria/src/sandbox-v1-critical-adversarial.test.ts — attacks the three workflows that must not be a security or consistency risk at merge time: billing (one OTLP identity authority for metrics/traces and the cost summary, cross-tenant claims, freeze isolation, replay dedupe, Analytics Engine SQL injection), client isolation (sentinel/affix resolution, the fixed sandbox-id fallback on the global leads bucket, lookalike-slug prefix disjointness) and data consistency (record-id percent-encoding, retention never pruning the committed point, traversal rejection). All three run as part of bun run test. The attack matrix, the defects the runs found, and the remaining observations are in docs/plans/sandbox-v1-adversarial-fuzz.md.

Docs channel tests

webs/fouria-docs/scripts/channel.test.ts covers the versioned docs plumbing: channel selection/validation, base-path normalization, channel-relative link prefixing, and the versions.json manifest (latest/staging paths). It runs as part of the root bun run test (vitest.config.ts includes webs/fouria-docs/scripts/**/*.test.ts).


apps/fouria — Cloudflare Worker

Unit Tests

Colocated .test.ts files alongside source files. Run with vitest.

cd apps/fouria
bun run test                  # All unit tests
bun run test:watch            # Watch mode
bun run test:coverage         # With Istanbul coverage (alias of coverage)
bun run test:client           # SPA React component/page tests (jsdom)

The Worker unit project (vitest.config.ts, Node pool) excludes src/client/**. The SPA component and page tests run under the dedicated jsdom project (vitest.client.config.ts) via bun run test:client — they need the React plugin and a DOM, and declare /** @vitest-environment jsdom */ per file.

Authentication, backup, and billing regression coverage

  • src/auth/middleware.fallbacks.test.ts drives the Hono authentication and role-resolution chain: malformed forwarded JWTs, E2E cookies, local-role fallback, and Dispatch plan metadata. src/auth/routing.test.ts covers invalid tiers, input coercion, and hostname edges.
  • src/activity/scheduler.test.ts covers durable boot success/retry, snapshot interval and idle guards, deprovision checkpoint-before-stop, restore jobs, maintenance jobs, and automation wake lead windows. src/backup-watch/durable-object.test.ts covers enqueue failures without clearing dirty state. src/routes/api/storage.errors.test.ts covers backup/restore conflicts, error responses, background failures, and job/config read failures. The persistence and encryption suites remain the owners of archive, lease, retention, and recovery-point guarantees.
  • src/restore-trigger-adversarial.test.ts and the restore-trigger section of src/sandbox-v1-adversarial.test.ts encode the destructive-operation invariant for both runtimes: a restore may run only when an administrator explicitly requests one, when the durable debug/hard-restart path armed restore-needed, or when the container positively reports no tenant state. A gateway that is merely down, crashed, or being rebooted keeps its container filesystem, so the launch must never apply an older archive over it. Inconclusive signals (control-channel error, malformed probe, unreadable pointer) fail closed, and a request owner never runs the restore inline (the durable boot job owns it). The same suites pin the two hardening rules this matrix relies on: a container-state probe that is unavailable (no exists()) or failed (success: false, or a v1 probe exit code other than 1) is never read as empty, and the legacy stuck-start watchdog checkpoints and verifies the recovery point advanced before it is allowed to reprovision a container.
  • src/routes/api/billing.freeze.test.ts exercises persisted freeze/unfreeze, acting-user attribution, and the OR of stored and Dispatch freeze flags. src/billable-actions.emit.test.ts covers producer chunking, single-send fallback, delivery failures, frozen-customer exclusion, duplicate suppression, and fail-closed outcome normalization. A producer with neither sendBatch nor send must reject, never report unsent actions as enqueued.

Run bun run test:coverage from the repository root for comparable cross-package coverage. The root typecheck delegates to the five workspace configurations listed above rather than recursively checking vendored Elixir dependencies and marketing-site sources under one Node-only configuration. Moltlazy has a package-local strict tsconfig.json; it no longer inherits the root configuration. These runs do not prove deployed Cloudflare Access policies or live Stripe delivery; unit line coverage is not a claim of complete end-to-end or branch coverage.

Billing-surface coverage. src/client/pages/__tests__/HomePage.test.tsx renders the Usage & Billing widget with cost data present (total plus the billable action/credit rollup) and asserts the empty ($0.00) and role-hidden cases. src/client/pages/__tests__/AccountPage.test.tsx covers the Plan & Billing card (tier, seat/credit usage, past-due alert, non-admin hidden). The /api/admin/costs/v2/summary aggregation (AE SQL query → summary) is unit-tested in src/cost-summary.test.ts and src/routes/api/costs.test.ts.

UI Tests (Playwright, local)

The admin SPA has a Playwright suite that drives the real dashboard served by the Vite dev server (bun run dev), with every /api/* call mocked per test. No Worker, sandbox container, or Cloudflare credentials are required, so it runs fully offline and is the fastest way to exercise the UI end-to-end.

cd apps/fouria
bun run test:ui           # headless Chromium
bun run test:ui:headed    # watch the browser

Specs live in apps/fouria/test/ui/ and the config is apps/fouria/playwright.config.ts. The shared mock helper (test/ui/support/api.ts) stubs GET /api/admin/whoami (role/dev/demo flags), the page data endpoints, and the /api/health-ws + /api/admin/devices/ws WebSocket streams. Coverage:

  • App shell — sidebar, notification bell, ADMIN navigation, dev-only nav.
  • Role access — ROOT → Diagnostics redirect, ADMIN blocked from Diagnostics, BASE_USER limited nav + route guard.
  • Pages — every admin route renders for its role.
  • Agent Studio API regression (src/routes/api/agent-studio.test.ts) — the dedicated automations endpoint normalizes cron and interval schedule objects to display strings (including the kind/everyMs/anchorMs crash case). GatewayStatusLed.test.tsx verifies non-ROOT users never query debug status.
  • Agent Studio (agent-studio.spec.ts) — the roster + overview render, all seven tabs (Capabilities, MCP, Memory, Automations, Connections, Health) open over the mocked RPC payloads, AGENTS.md opens in the RPC editor, and a sleeping instance shows the wake affordance without any read waking it. The mock helper exposes studio / studioDetail options plus SAMPLE_STUDIO_ROSTER / SAMPLE_STUDIO_DETAIL fixtures. Unit coverage in src/agents/studio.test.ts and src/routes/api/agent-studio.test.ts.
  • Health plugin pagination (agent-studio.spec.ts) — 10 plugins per page, navigation to the final partial page and back, disabled boundary controls, and no pagination controls for a single-page list.
  • Capabilities pagination (agent-studio.spec.ts) — independent 10-item pages for allowed/denied tools, subagents, skills, available/effective tools across groups, and notices; filtering resets pages before displaying matching items.
  • Releases (releases.spec.ts) — the promotion page stages a release on a dev instance (with a commit message), applies it on production, and warns when production lacks the PROMOTION_SOURCE_BUCKET binding. The mock helper exposes releaseEnvironment and promotionSourceBinding for GET /api/admin/releases/status and an empty GET /api/admin/releases/history. Unit coverage in src/client/pages/__tests__/ReleasesPage.test.tsx.
  • Notification bell — pending-device badge + dropdown, empty state.
  • Persistence lock indicator (PersistenceLockIndicator) — the lock widget beside the bell reflects the shared PersistenceLockContext state, which the BackupHistoryPanel raises/clears around a backup or restore (no polling): open/green when idle, closed/red while an operation is in flight, and hidden for BASE_USERs. Unit coverage in src/client/components/__tests__/PersistenceLockIndicator.test.tsx and src/client/components/__tests__/BackupHistoryPanel.test.tsx.
  • Gateway circuit breaker (gateway-circuit-breaker.spec.ts) — with the health stream down, gateway queries are parked (no /api/admin/agents request) and the waiting indicator shows; flipping the mocked health stream to up flushes the queued queries and renders the data. The mock helper exposes installApiMocks(...).sendHealth(snapshot) and the HEALTH_UP / HEALTH_DOWN fixtures for this. Unit coverage lives in src/client/gateway-gate.test.ts, src/client/GatewayHealthContext.test.tsx, and src/client/api.test.ts.
  • Diagnostics dashboard (skip-not-park) — unit coverage in src/client/pages/__tests__/DiagnosticDashboard.test.tsx and src/client/components/__tests__/ConfigSection.test.tsx: with the gateway down the page still renders but its container reads (/api/admin/config*, devices) are never dispatched, gateway-only debug probes are skipped with an inline explanation, and the Configuration Editor stays visible with a non-blocking waiting notice instead of a blocking spinner.
  • Billing widget (billing-widget.spec.ts) — the Home “Usage & Billing” widget shows the billed total and billable action/credit rollup when the cost summary carries data, $0.00 when it is empty, and is hidden for BASE_USERs. The mock helper exposes a costSummary option.
  • Cloudflare Components widget (cloudflare-components.spec.ts) — the ROOT diagnostics widget renders each component’s status/latency/href, hides the AI Gateway when the API omits it, reveals the binding-derived facts panel on hover and keyboard focus, opens each card’s Cloudflare dashboard logs page in a new tab (offline dashboard stub), renders a plain non-clickable card when a component has no logsUrl, re-fetches on Refresh, and reflects sleeping/down states. The mock helper’s HEALTH_COMPONENTS fixture carries the enriched facts + logsUrl shape returned by GET /api/health/components.
  • Diagnostics responsive layout (diagnostics-responsive.spec.ts) — at a phone viewport the Cloudflare Components strip collapses into a compact status widget and expands on tap, the gateway + persistence status widgets sit parallel to it (the topbar omits them on the diagnostics page), the Fouria DevOps options render as a vertical icon + name menu (collapsed by default, tap to open), the Configuration Editor reflows to the viewport, and the page never scrolls sideways (checked at 320px). The mock helper exposes debugRoutes (to render the Configuration Editor) and a config fixture for the parsed sections + raw JSON.

NixOS / Nix dev shell. apps/fouria/flake.nix provides the Playwright browsers via PLAYWRIGHT_BROWSERS_PATH (pkgs.playwright-driver.browsers) plus the PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS and PLAYWRIGHT_HOST_PLATFORM_OVERRIDE NixOS workarounds. Because the Nix browser bundle can lag the project’s npm playwright version, playwright.config.ts resolves the newest chromium-* under PLAYWRIGHT_BROWSERS_PATH and passes it as an explicit executablePath; outside Nix the variable is unset and Playwright’s default resolution applies.

Key test files (see WORKER-DEVELOPMENT.md for full list):

FileTests
auth/jwt.test.tsJWT decoding and validation
auth/jwks.test.tsJWKS fetching and caching
auth/middleware.test.tsAuth middleware, DEV_MODE, E2E_TEST_MODE, dev-instance ROOT→ADMIN role preview
gateway/env.test.tsContainer environment variable building
gateway/process.test.tsProcess finding, lifecycle, stuck-start watchdog, and structured gateway-start failure correlation
gateway/classify.test.tsclassifyGatewayStart running-only stuck-start verdict
gateway/sandbox-exec.test.tsSandbox command results and structured process failure logs
gateway/utils.test.tsProcess wait lifecycle and non-zero exit reporting
index.test.tsWorker middleware, proxy routes, and request failure trace context
gateway/r2.test.tsR2 mounting logic
auth/rbac.test.tsRBAC permission matrix
metrics.test.tsCost/usage Analytics Engine instrumentation
persistence.test.tsR2 backup/restore, retryable cold-restore intent, scheduled-snapshot safety, cross-isolate lease, crash-consistent catalog + orphan reconciliation
restore-trigger-adversarial.test.tsRestore-trigger matrix: gateway down/reboot never restores live state; only an explicit latch or an empty container may restore; unknown probes fail closed
sandbox-v1-adversarial.test.tsv1 zero-data-loss and boot attacks, including “a v1 boot restores only an empty container” (gateway down, stopped container, probe failure, operator restore)

E2E Tests (Cloud)

Deploys a real worker to *.workers.dev via Terraform + wrangler deploy. Requires Cloudflare credentials.

cd apps/fouria

# Full E2E with browser
bun run test:e2e

# CLI-only (no browser)
bun run test:e2e:cli

Setup (test/e2e/_setup.txt):

  1. Provisions cloud infrastructure via Terraform (R2 bucket) — the content-hash worker cache is temporarily disabled (issue #297), so every run provisions fresh
  2. Deploys worker via wrangler deploy --var E2E_TEST_MODE:true
  3. Starts Playwright browser via plwr
  4. Polls /api/health for cold start up to the startup budget — 30s on a warm cache hit, otherwise STARTUP_TIMEOUT_SECS (600s default)
  5. Opens browser to worker URL, waits for “Device pairing required”

Fresh-deploy failures are usually the shared v1 app. The top-level (local/E2E) wrangler.jsonc still declares the SandboxV1 container application as fouria-sandbox-v1, which belongs to the production worker; server/deploy renames it per run (<worker>-sandbox-v1). If a fresh E2E deploy reports “Durable Object-managed Container application settings were not fully applied” after five attempts, check that rename first. Teardown probes the teardown token against the account endpoint and falls back to the deploy token, then reports the real container-app delete status — a stale E2E_CLOUDFLARE_API_TOKEN used to make every delete silently no-op and leak billing applications.

Phase-D suite assumptions. y_production_profile creates its recovery point on demand (POST /debug/backup/sync) because BACKUP_CHANGE_WATCH is off on the E2E worker, so no scheduled snapshot exists. zz_gateway_hard_restart reads processId from /api/health (the /status/polling RPC fast path deliberately omits the process scan) and waits for the post-restore boot before capturing its baseline. get-agent-id retries the roster for a bounded window because h_cli_knowledge runs it right after a plugin reload.

Tenant identity is explicit. Both fixtures bind CUSTOMER_SLUG (acme-corp for the cloud worker, matching the corpus’ mock tenant and the admin cross-tenant guard’s E2E_EXPECTED_CUSTOMER; start-local sets it for the billable profile). Without it the Worker derives its identity from WORKER_NAME, which a bare *.workers.dev E2E worker does not set, so getCostContext().customer falls back to the default sentinel. That is not a valid tenant slug, so the software-manifest store refuses to build and PUT /api/admin/software/manifest answers an instant 500 — the fixture bug that failed zzz_software_provisioning, not a route defect.

Gateway readiness — shared helper. After provisioning, the cctr _setup.txt waits for gateway readiness through apps/fouria/test/lib/e2e-helpers.sh. Its e2e_wait_for_health probes /api/health immediately (the first probe also triggers the cold boot — there is no fixed 180s pre-poll grace) and requires both startup.ok and readiness.ok, polling every 5s. The wait stays bounded by the existing warm/cold startup budget: 30s on a warm cache hit, otherwise STARTUP_TIMEOUT_SECS (600s default). The same helper backs test/e2e-agent/_setup.txt and the parallel driver’s top-level readiness wait. It does not back every readiness check: the parallel runner’s generated per-group _setup.txt and ab_gateway_ws_origin.txt keep their own grace-free, bounded /api/health loops (the latter also uses the shared e2e_classify_ws_upgrade verdict helper). The helpers are unit-tested in apps/fouria/src/e2e-helpers.test.ts.

E2E cost safety (issue #297): the container idle timeout is derived from the instance hostname (staging 15m / dev 2h — the Sandbox SDK 1.0 setInactivityTimeout() platform cap); there is no SANDBOX_SLEEP_AFTER knob. Billable production instances (production platform, production environment) do not auto-sleep. A container that outlives its run (failed teardown, workflow timeout) therefore stops billing at the derived idle instead of running 24/7, and the workflow teardown deletes the workers outright. Worker caching is disabled until the feature is reworked to reuse only within working hours and to delete on eviction.

Boot profiles: Both profiles provision the same production-profile image (apps/fouria/Dockerfile, BUILD_PROFILE=production, the slim default) and the same standard-2 instance type (1 vCPU / 6 GiB — the smaller basic 1 GiB / standard-1 0.5 vCPU instances stall the gateway under the suite’s CLI load). The optional PDF/document tooling (pandoc + weasyprint) ships only in the dev build profile used by the dev/staging environment — see apps/fouria/docs/WORKER-DEVELOPMENT.md. PR runs use the light basic profile (E2E_TEST_MODE=true, short plugin timeouts) for fast feedback; develop runs use the production profile (real plugin timeouts, container OTLP export + Cloudflare Tunnel). The profiles differ by env only — not the image or instance type. Both profiles bind the worker OTLP receiver with a random per-run OTEL_SHARED_SECRET (no GitHub secret required), so the OTel billing-queue test runs in either profile.

Single-worker parallel E2E: the e2e workflow provisions one isolated worker and runs dependency-grouped cctr suites against it. Independent phase A and B groups run concurrently; config/lifecycle/restore groups remain serial. Pull requests run boot-only by default. develop and a full manual dispatch run the corpus; selective retries link only the suites recorded in the prior run’s e2e-failed-* cache. The runner primes the CLI/device path once per group so device-admin suites do not each pay the cold first-connect cost.

E2E run statistics: run-parallel.sh writes e2e-stats.md with initial and per-group Gateway readiness times, cctr group outcomes and case durations, and per-group counts of Worker-launched OpenClaw CLI calls, Gateway WebSocket connections, and Admin HTTP RPC calls. The report and debug transcripts are uploaded as the e2e-logs-* run artifact, and mirrored to the e2e-artifacts-* branch on a best-effort basis (the branch push never fails the job). Transport counters require an explicit X-FourIA-E2E-Stats: 1 request, DEBUG_ROUTES=true, and an explicitly bound E2E_TEST_MODE profile marker (true for basic, false for production). The long-lived Gateway process is excluded from CLI totals. CI pins cctr to 0.31.0 for stable report parsing.

Failed-only reruns: when a run reaches the gateway (gateway-ready) but some suites fail, the job records those suite names and caches them (e2e-failed-<ref>-<shard>). The next run restores that list and passes it to run-parallel.sh, which links only those suites into the generated cctr groups (the PR already passed unit + validate-secrets, enforced by needs). A full pass clears the list and the following run is a full pass again. A committed seed (.github/e2e-rerun-suites.txt, only when present) bootstraps a subset before any cache exists; it is not required in steady state.

Teardown (test/e2e/_teardown.txt):

  1. Dumps gateway logs + doctor output
  2. Stops browser (saves video)
  3. Destroys cloud resources (worker, container app, R2 bucket, Terraform)

Test files (alphabetical order — naming determines execution order):

FileWhat it tests
b_cli_devices.txtopenclaw devices list through /debug/cli
c_cli_models.txtopenclaw models list, model catalogs
d_cli_channels.txtChannel probe, config inspection
e_cli_config.txtConfig validate, set, unset via CLI
g_cli_secrets.txtSecrets audit, reload via CLI
h_cli_knowledge.txtWiki init, status, get via CLI
i_cli_cron.txtCron list, add dream job
j_cli_integrations.txtIntegration config verification
k_secrets_store.txtSecrets Store (gateway token, CDP, container config, device pairing)
l_cli_billing.txtPricing constants, costs, instance types, billable events
m_cli_agent_tokens.txtAgent turns, token usage metrics
provision_and_dispatch.txtCloud-only — dispatch worker + Supabase tenant lifecycle. Its control-plane steps need INTERNAL_DISPATCH_INVALIDATE_TOKEN (staging value) and self-skip when it is unset; there is no hardcoded fallback.
a_rbac_admin_isolation.txtADMIN role isolation (management API, cross-tenant)
a_rbac_api_matrix.txtROOT/ADMIN/BASE_USER matrix + deny/allow decisions over dashboard + loading-page endpoints
a_rbac_base_user_isolation.txtBASE_USER isolation (whoami, blocked from management/secrets/writes)
a_rbac_device_pairing.txtDevice pairing permissions for all 3 roles (with retry on timeout)
a_rbac_root_isolation.txtROOT isolation (diagnostics access, blocked from management/agents/secrets)
a_rbac_ui_pages.txtUI page access for all 3 roles (25 test cases; every nav page route served)
aa_health_gate.txtGateway health check between RBAC and CLI test groups
ac_gateway_startup_concurrency.txtCross-isolate cold-start serialization after destroying the container
ad_observability_canary.txtLocal-only — proxy header redaction + live-gateway-log RPC-stream warning (Local Explorer SQL API)
o_opencode_credits_task.txtOpenCode credits automation executed in the sandbox (direct consumer)
o_leads_proxy.txtLeads R2 proxy over the Worker WebSocket (self-skips without cdp-secret.txt)
r_browser_cdp.txtCloudflare Browser Rendering BROWSER binding over the public /cdp shim (discovery, shared-secret auth, target + DOM + PNG); self-skips when the binding is unavailable
r_browser_remote_profile.txtPlatform browser.profiles.cloudflare remote-CDP profile → container proxy → private cdp.internal bridge (registration, open, cross-call tabs)
p_tasks_queue.txtTASKS queue producer → consumer → sandbox round trip (self-skips w/o key)
p_otel_billing_queue.txtOTLP traces receiver → billable reconcile → BILLING_QUEUE producer (self-skips on a non-billable instance)
test/e2e-billing/Local-only — billable worker emits to BILLING_QUEUE (observed via Local Explorer) + freeze drop
p_scheduled_cron.txtLocal-only — both crons fire via /cdn-cgi/local/scheduled
z_cli_staging.txtAPI staging tests (costs, storage, users)
zzz_cron_wake.txtCloud-only — container destroy, cron wake cycle

Backup/restore suites are excluded from this corpus. They live in apps/fouria/test/e2e-backup/ so a default bun run test:e2e / test:e2e:local run does not trigger their multi-minute cold boots and ~250 MB R2 snapshots. Run them with bun run test:e2e:backup (local) or through phaseE_restore in the parallel runner. Files: zz_reprovision_guard, zzz_backup_cold_restore, zzz_backup_restore_skip_setup, zzz_backup_semantics, and the legacy zzz_persistence_lifecycle.

Retry diagnostics in cctr tests. cctr merges a test’s stdout and stderr into the value matched by {{ … }}. A retry/progress line printed to >&2 therefore breaks a strict matcher (json object, json bool, a bare number, …) once the retry path is taken — the flake only appears when the first probe is unhealthy. Write such diagnostics to a side log instead: echo "…" >>"$CCTR_WORK_DIR/e2e-retry.log" 2>&1.

o_leads_proxy requires cdp-secret.txt. The test self-skips (and passes in ~0s) when the fixture file is missing, so a regression in the fixture wiring looks like a green test. test/e2e/fixture/server/start persists $CDP_SECRET to cdp-secret.txt on both the cache-hit and fresh-provision paths; if the test reports skipped: true, the fixture is the place to look.

p_otel_billing_queue self-skips on a non-emitting instance. The OTLP traces receiver gates emission on isBillableEventsEnabled (src/dev-mode.ts): billable production and the staging platform always emit, while a dev-environment instance accepts + ignores approved spans (HTTP 200) and reports billable: false, never emitting to BILLING_QUEUE. The local E2E worker runs with DEV_MODE=true and no routable WORKER_URL, so it is a dev instance — the test skips (like it already does for a missing secret / 401) instead of asserting an emit that cannot happen locally. The emit path is asserted by the cloud profile (a production workers.dev hostname / the staging platform) and by src/routes/otel-traces.test.ts. A dev instance can opt in explicitly with BILLABLE_EVENTS=true; the override changes emission only, so the sleep policy and the cost-row dimensions are untouched. provision_and_dispatch likewise self-skips in local mode (no dispatch worker

  • Supabase).

Worker billing-emit E2E (test/e2e-billing/, local-only). The producer half of the billing workflow is covered locally by opting the worker into a billable identity (E2E_BILLABLE=true → start-local passes a production-looking WORKER_URL/CUSTOMER_SLUG). The suite POSTs a synthetic OTLP span, asserts emitted >= 1, observes the BILLING_QUEUE producer via the Local Explorer queue* span, then freezes the customer and asserts emitted == 0 with no new queue span. Analytics Engine writes are not observable locally (no AE binding in Local Explorer; no writeDataPoint span), which is why the lerma consumer suite uses a fake AE. Run with bun run --cwd apps/fouria test:e2e:billing:local; see apps/fouria/test/e2e-billing/README.md.

h_cli_knowledge caches the owner agent id. The corpus resolves the owner agent id once per suite through test/e2e/fixture/get-agent-id, which caches it in $CCTR_FIXTURE_DIR/agent-id.txt and reuses it across the sequential wiki init / wiki status / wiki get cases. The helper reads /api/admin/agents through Admin HTTP RPC, then Gateway WebSocket RPC, with the route’s CLI fallback for compatibility; caching avoids repeating that roster lookup per case while preserving the plugin/wiki assertions. The cache is suite/worker-run scoped (cctr copies fixture/ into $CCTR_FIXTURE_DIR once and reuses that directory across cases) and is discarded when cctr cleans up the per-suite temporary fixture/work directory, not by _teardown.txt. The helper fails closed: an empty or malformed roster exits nonzero instead of silently defaulting to main.

E2E Tests (Local)

Runs tests against the local Worker dev server (Cloudflare Vite plugin, vite dev) with containers via Docker. Avoids 5-10 minute Terraform provisioning. Cloud-only tests (dispatch worker + Supabase, container destroy/recreate) self-skip in local mode.

cd apps/fouria

# Local E2E with Docker + cctr (auto-builds Docker image)
bun run test:e2e:local

# CLI-only (no browser)
bun run test:e2e:local:cli

Setup (fixture/server/start-local):

  1. Sources test/e2e/.dev.vars and installs it as the Worker’s .dev.vars for the run (the Cloudflare Vite plugin has no --env-file; stop-local restores the developer’s file)
  2. Finds unused port (8787-8797)
  3. Starts vite dev --port <PORT> --strictPort (also generates CDP_SECRET/OTEL_SHARED_SECRET, injected as vars through the plugin’s config hook, and uses /cdn-cgi/local/scheduled for the cron test). The outer test:e2e:local* scripts run bun run build:docker first (chained with &&, so it completes before the dev server starts); vite dev then builds/launches the container image through Docker.
  4. Polls GET /api/health?passive=1 until the Worker answers any HTTP status (200/503/…) — a cold boot legitimately answers 503 while the container/gateway is still starting, so the gate must not require a 200 and must not use curl -f. It probes the fast, non-waking passive endpoint so a poll does not wake the container or re-trigger the restore/ensureGateway boot on every attempt. The window defaults to 120s and is overridable with E2E_STARTUP_TIMEOUT_SECS. The real gateway readiness wait (startup.ok + readiness.ok) is the separate wait for gateway API readiness setup step.
  5. Writes fixture files (dummy CF Access creds — ignored via DEV_MODE=true)

Resource headroom. The local run drives the Docker daemon (container image + container), workerd (Vite dev server + Miniflare R2/DOs), and the bun/cctr drivers at once. On a memory-constrained host the whole process group can be killed by SIGABRT (OOM) mid-cold-boot — observed during _setup: start moltworker server after the image built and the dev server answered 503. Give Docker/the host at least ~8 GB and avoid running other containers alongside the suite. The readiness gate above fails fast when the dev server itself dies, but it cannot distinguish an OOM kill from a crash, so check dmesg/Docker logs when the process group dies during startup.

Container failed to start / no app container. If the suite stalls at the readiness step while docker ps shows only a …-proxy sidecar that is recreated every ~20 s, the host cannot start local containers at all: on Linux hosts where net.ipv4.conf.all.src_valid_mark=1 (Tailscale sets this), the sidecar handshake is dropped and workerd times out (cloudflare/workerd#6860). The dev server prints the fix before building images; the short version is echo 'net.core.devconf_inherit_init_net = 2' | sudo tee /etc/sysctl.d/99-docker-netns-defaults.conf && sudo sysctl -w net.core.devconf_inherit_init_net=2, then docker rm -f $(docker ps -aq --filter name=workerd-). See DEVELOPMENT.md.

Teardown (fixture/server/stop-local):

  1. Kills the Vite dev server process
  2. Restores the app’s .dev.vars
  3. Cleans up fixture files

Prerequisites:

# test/e2e/.dev.vars must contain:
DEV_MODE=true
E2E_TEST_MODE=true
DEBUG_ROUTES=true
MOLTBOT_GATEWAY_TOKEN=<value>
CF_ACCOUNT_ID=<id>
CLOUDFLARE_AI_GATEWAY_API_KEY=<key>   # plus CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID
# The REST transport uses CLOUDFLARE_API_TOKEN, derived from the same AI-scoped key.

# Docker must be running
# wrangler must be authenticated via OAuth (wrangler login)

Cloud-only tests (self-skip when E2E_LOCAL=true):

FileReason
provision_and_dispatch.txtRequires dispatch worker + Supabase
zzz_cron_wake.txtRequires container destroy/recreate

The Dispatch control-plane steps in provision_and_dispatch.txt (and the staging integration corpus) send INTERNAL_DISPATCH_INVALIDATE_TOKEN to the deployed Worker and self-skip when it is unset. Both the Worker and Lerma get that value from STAGING_INTERNAL_DISPATCH_INVALIDATE_TOKEN in CI; the Worker fails closed (403) without it, and there is no shared hardcoded key.

ad_observability_canary asserts on the local worker’s own logs. The Local Explorer exposes the same logs / spans tables as the Cloudflare Workers Observability API, scoped to the local dev worker, via POST /cdn-cgi/local/explorer/api/local/observability/query (wrapped by the fixture/observability-query helper). The canary uses it to (1) drive the catch-all proxy’s request log with sentinel Cookie / Authorization / CF-Access-Jwt-Assertion / tenant-metadata values and assert none reach the log store, and (2) fetch the running gateway’s logs and assert the fetch adds no ReadableStream received over RPC disconnected prematurely warning. Both cases are local-only (the API does not exist on a deployed worker), so they self-skip in cloud E2E; they fail loudly if the local API is unavailable rather than silently skipping, so the coverage cannot rot. These are the observability-validation canaries for the credential-logging and live-process-stream fixes.

Admin UI Render Timing (local-only, plwr)

test/e2e/ui_components_render.txt is a local-only browser suite that validates the fouria admin (root) UI components and reports how long each takes to render. It drives the plwr session started by _setup.txt (the same one the pairing step uses), sets the X-Mock-Role cookie, loads each admin route, waits for the component’s root selector, and measures, in milliseconds relative to the page navigation start:

  • openMs — navigation start → SPA loaded (performance.now() sampled right after plwr open returns)
  • componentMs — navigation start → the component’s selector is present (sampled right after plwr wait returns); this is the data-driven render
  • waitMs — componentMs − openMs (client render + API round-trip after load)
  • nav.* — Navigation Timing (domInteractive, domContentLoaded, load) and First Contentful Paint

Coverage mirrors the Playwright UI suite (test/ui/pages.spec.ts): the shell (.admin-layout, .sidebar, gateway status LED, persistence lock, notification bell) and every ADMIN page, plus the ROOT-only diagnostics page and its Fouria DevOps + Cloudflare Components widgets.

The file also carries one behavioral case (driven directly through plwr) that loads the diagnostics page against the local worker and asserts the Cloudflare Components widget links the container card to its Cloudflare dashboard page (href contains dash.cloudflare.com) and reveals its binding-derived detail panel on hover (visibility flips hidden → visible, panel text contains the Runtime fact). It self-skips when browser tests are disabled or plwr is absent.

The file self-skips unless E2E_LOCAL=true (file-level %skip) and also self-skips when E2E_SKIP_BROWSER_TESTS is set, so cloud runs stay green. The fixture/ui-render-measure helper appends every record to ui-render-times.jsonl; the final ui-render-report case writes a table to ui-render-report.txt and prints a JSON rollup (stdout) with maxComponentMs and the slowest component.

cd apps/fouria

# Just this suite — _setup.txt still starts the worker + browser:
bun run test:ui:render:local   # build:docker + E2E_LOCAL=true cctr -vv test/e2e/ -p 'ui render'

# Or run it as part of the whole local corpus:
bun run test:e2e:local

E2E Agent Subset (OpenClaw migration smoke)

apps/fouria/test/e2e-agent/ is a focused, browser-free subset of the CLI surface used to validate an OpenClaw version migration. It reuses the shared test/e2e/fixture (via symlink) and the same local-mode runner, but only runs the commands the Worker shells out to:

  • b_cli_devices.txt — openclaw devices
  • c_cli_models.txt — openclaw models
  • d_cli_channels.txt — openclaw channels
  • e_cli_config.txt — openclaw config validate
  • g_cli_secrets.txt — openclaw secrets
  • h_cli_knowledge.txt — openclaw wiki
  • m_cli_agent_tokens.txt — openclaw agents / token flows
  • n_gateway_soft_reload.txt — gateway soft reload (SIGUSR2; asserts the Node inspector stays closed and proves the restart with a new http server listening line plus a stable processId)

Bounded agent turns. m_cli_agent_tokens.txt bounds its Gateway turn with openclaw agent --timeout 40. The CLI’s default turn deadline is 600s, which outlives the e2e HTTP budget (debug-cli waits up to 80s server-side and 90s via curl-auth; /debug/cli uses timeout_ms). A bounded turn returns its JSON envelope (result, timeout, or provider error) within the budget, and an empty response is reported as {"status":"timeout"} instead of failing the JSON-object assertion on an empty jq result.

cd apps/fouria
bun run test:e2e:agent   # build:docker + E2E_LOCAL=true + browser skipped

The openclaw-migration skill runs this subset after bumping catalog.openclaw to confirm the CLI commands still behave.

Gateway Boot Smoke Test (local)

Validates that the fouria sandbox image actually boots a healthy OpenClaw gateway on the local machine — no Worker, no cloud e2e, no third-party machines. It builds the exact production image (apps/fouria/Dockerfile + repo-root context, same base cloudflare/sandbox image wrangler uses) and runs the real boot path (start-openclaw.sh → onboard → plugin installs → moltlazy patch/validate → gateway launch → health poll) inside Docker. The container’s exit code is the verdict: the in-container wrapper runs start-openclaw.sh (whose own readiness loop already fails if the gateway dies or never binds), then asserts /health returns {ok: true} while the gateway is still alive.

cd apps/fouria

# Default cold-boot smoke (no real credentials needed):
#   onboard runs --auth-choice skip, tunnel + OTEL steps self-skip.
bun run test:gateway:boot

# Standalone native phases in the smoke container: `start-openclaw.sh setup
# → start-openclaw.sh launch → post-start`, then assert /health. This is NOT
# the Worker's production cold start, which is separately `restore → atomic
# start (setup + Gateway under one lock) → post-start` (see the table below).
bun run test:gateway:boot --native

# Same, plus the real cloudflare-ai-gateway onboard path and an assertion
# that the provider plugin is installed + enabled. Sources credentials from
# apps/fouria/.dev.vars.e2e (never echoed) — generate it from SECRETS.yaml
# with `bun run secrets:generate`.
bun run test:gateway:boot:ai

# Leave the container running after the assertion, with port 18789
# published to 127.0.0.1 for manual inspection (Control UI / WS).
bun run test:gateway:boot --keep

# Skip build:docker + docker build and validate an existing image.
bash scripts/gateway-boot-smoke.sh --no-build --image <tag>

Prerequisites: Docker running, network access (the image build pulls the ClawHub plugins; boot-time installs are a fallback only, bounded by start-openclaw.sh timeouts and non-fatal for boot).

What it validates vs. what it does not:

Validated locallyStill needs local e2e / cloud e2e
Image builds from the production DockerfileSandbox SDK supervisor + Worker RPC layer
onboard (skip or real AI Gateway) completesRBAC / API routes / device pairing
Build-time plugin pre-install + enablementDispatch worker routing
moltlazy patch + validate (config contract)Cost observability / R2 backup flows
gateway binds :18789, /health returns okBrowser-driven Control UI flows
--native: standalone start-openclaw.sh setup → launch → post-start phases in the smoke container (not the Worker’s production restore → atomic start (setup + Gateway under one lock) → post-start)Worker’s restore + sandbox.exec supervision (handle/relaunch)

E2E Parallel (single-worker parallel runner)

apps/fouria/test/e2e-parallel/ is the parallel replacement for the serial test/e2e run. The e2e CI job no longer shards the corpus across N workers (one per shard); it runs run-parallel.sh, which provisions ONE worker and runs dependency-grouped cctr suites concurrently against it.

cd apps/fouria
bun run test:e2e:parallel        # == bash test/e2e-parallel/run-parallel.sh

Suite scope (E2E_BOOT_ONLY). By default the e2e CI job sets E2E_BOOT_ONLY=1 for pull requests: the runner provisions the worker, waits for /api/health to report startup.ok + readiness.ok, and then tears down without running any cctr group — a PR only proves the gateway boots. The full corpus runs on develop (merge gate) and on a manual workflow_dispatch with the full_tests input checked (the default for dispatch). Locally, set E2E_BOOT_ONLY=1 to reproduce the PR check.

Phases (see the suite README for the full grouping):

  • phaseA_devices ∥ phaseA_rbac ∥ phaseA_health — parallel (the RBAC mock role is a per-request cookie, so the RBAC suites are independent; the two device-store suites share phaseA_devices to avoid racing the approve path)
  • phaseB_readonly ∥ phaseB_async — parallel
  • phaseC_stateful — config/secrets/cron (serial)
  • phaseD_lifecycle — plugin + reload/restart + dispatch (serial)
  • phaseD_startup — cross-isolate cold-start concurrency regression and the declarative software-provisioning suite (zzz_software_provisioning.txt: flag/report wiring, manifest validation + upload, cold reprovision installs the declared software, boot-terminal ticks, and an on-demand run that installs a newly declared package while skipping the already-installed one), serial, each with a separate startup-sized budget
  • phaseE_restore — reprovision guard followed by the backup/restore lifecycle suites, resolved from test/e2e-backup/ (serial, startup-sized budget). Skipped in GitHub Actions by default because it drives multi-minute cold boots and ~250 MB R2 snapshots; run it locally, or force it in CI with E2E_RUN_RESTORE_PHASE=1. For a standalone local run of just these suites use bun run test:e2e:backup.

Phase caps. phaseD_lifecycle is bounded by PHASE_D_TIMEOUT_SECS (default 1200s / 20 min; the recovery suites plus provision_and_dispatch now legitimately exceed the old 7-minute budget, and killing the group mid-dispatch left a restore in flight). The cold-start concurrency group has its own PHASE_D_STARTUP_TIMEOUT_SECS budget (default STARTUP_TIMEOUT_SECS + 180s); backup/restore suites use PHASE_E_TIMEOUT_SECS (default STARTUP_TIMEOUT_SECS + 300s). The full run’s initial readiness budget is 1200s (basic) or 1500s (production). On a group timeout cctr fails (exit 124) and teardown still runs.

cctr 0.31.0 has no # comments. cctr silently drops a corpus file’s tests at the first top-level # line and reports the tests collected so far (often zero) with no error. Corpus files must begin with === (or a lone file-level %skip/%shell/%platform followed by the first ===), and a # comment must never sit before the first ===, between a file-level %skip and the first ===, or between two tests. Put rationale in test descriptions or docs, never in a # line (comments inside a test body/expected/where block are fine). node scripts/check-corpus-comments.mjs — run by the unit CI job — fails on any violating file, and cctr -l <file> prints the parsed test count.

Selective retry (retry_failed). A slow container can make a single backup call exceed its curl budget, failing a suite that would pass on a warm run. Rather than inflate the curl timeouts (which hides real hangs), the workflow records the failed suites — the Record failed e2e suites step writes ✗ <group>/<suite> markers to /tmp/moltworker-e2e-failed.txt, saved to the e2e-failed-<ref>-<shard>-<run_id> actions cache. Re-run only those suites with a manual workflow_dispatch and the retry_failed input checked: run-parallel.sh reads the restored file, links only the failed suites into their groups, skips every other group, and still provisions a single worker. The flag forces the suite to run (it overrides full_tests/boot-only). A phaseX_group/* entry (written when a group is killed by its phase cap) retries every suite in that group. The cache is saved only when something failed, so the restore always resolves to the most recent failing run.

Single-owner startup concurrency regression. ac_gateway_startup_concurrency.txt is a cloud-only regression for the OpenClaw 2026.9.8 cross-isolate launch race: it checkpoints tenant state, destroys the container, fires several concurrent cold non-passive /api/health probes (the cold-boot fan-out across Worker isolates), then asserts the gateway reaches startup.ok + readiness.ok, that exactly one running Gateway owner exists — the process may appear as start-openclaw.sh start before the atomic wrapper execs, or as openclaw gateway run after — and that the observable gateway output never shows either OpenClaw migration inputs changed during startup or state.sqlite is undergoing offline maintenance. Because it destroys the shared worker’s container it runs in phaseD_startup, after the ordinary lifecycle group and before restores, with its own startup-sized budget. It runs serially — never alongside the parallel phase A/B suites. Run it directly with cctr apps/fouria/test/e2e/ac_gateway_startup_concurrency.txt; it self-skips under E2E_LOCAL=true.

Each group is a generated cctr suite whose test files symlink the originals in test/e2e/ and whose _setup.txt attaches to the shared worker (copy credentials → wait readiness → warm the CLI/device identity). It forces E2E_NO_CACHE=1 (full teardown, no container left billing) and self-skips under E2E_LOCAL=true.

Readiness wait. The driver’s top-level wait uses the shared e2e_wait_for_health helper (immediate first probe, no fixed grace, bounded by STARTUP_TIMEOUT_SECS). The runner defaults that budget to 1200s; the workflow overrides it per profile (basic 1200s, production 1500s). The generated per-group _setup.txt do not call the shared helper — each keeps its own grace-free, budget-bounded /api/health loop that attaches to the already-warm worker. Because the phases run concurrently against one worker, the per-group readiness savings do not sum to the same job-time reduction — wall time tracks the slowest phase, not the total of the individual waits.

parallel-probe + a_parallel_spike.txt remain as a raw-concurrency diagnostic (how many concurrent CLI calls one worker sustains); run the probe suite directly with cctr -vv test/e2e-parallel/.

Logging policy. Verbose transcripts — the provisioning output (terraform/wrangler), each group’s cctr -vv output, and the teardown/gateway dumps — are written to /tmp/moltworker-e2e-logs/*.log and uploaded to the e2e-artifacts-<profile> branch, so the CI job console only shows concise [parallel] progress plus per-suite pass/fail markers and the summary. The serial test/e2e _teardown.txt follows the same rule: gateway/doctor dumps are written to disk only (never tee’d into the console).

See apps/fouria/test/e2e-parallel/README.md for the per-file parallelizability classification, the ordering graph, and the observed concurrency results.

Backup Semantics E2E

apps/fouria/test/e2e-backup/zzz_backup_semantics.txt and .../zz_reprovision_guard.txt prove the backup invariants against a real container (local Docker or cloud) and real R2. They are registered in phaseE_restore (serial; reprovision guard runs first) and are the E2E counterpart of the route/unit tests (src/routes/public.test.ts, src/routes/debug.test.ts, src/index.test.ts) and the dispatch fan-out unit test (workers/fouria-dispatch/test/unit/cron-fanout.test.ts).

Run the local-safe cases standalone (excluded from the default test/e2e corpus):

cd apps/fouria
bun run test:e2e:backup   # build:docker + E2E_LOCAL=true + browser skipped

See apps/fouria/test/e2e-backup/README.md.

Declared-software restore E2E (softwareProvisioning)

apps/fouria/test/e2e-backup/zzz_software_binaries_restore.txt is the runnable end-to-end example of a gateway whose agent skills need binaries that are not in the image. The manifest (apps/fouria/test/e2e/fixture/software/gog-goplaces.json) declares gogcli (the gog binary for the gog skill) and goplaces, the two skills are installed with openclaw skills install skills-sh:openclaw/openclaw/{gog,goplaces} --global, and after a backup (?inline=1) → /debug/destroy-container → restore it asserts that both binaries are back on PATH because the next provisioning cycle re-installed them (the report startedAt advances), while the skills come back from the snapshot rather than being reinstalled. It is local-only (self-skips unless E2E_LOCAL=true) and needs container network access (GitHub release downloads plus the ClawHub skills.sh resolver).

cd apps/fouria
E2E_LOCAL=true cctr -vv test/e2e-backup/zzz_software_binaries_restore.txt

The dependency-free equivalent — the same contract driven directly against the sandbox image, without workerd/Miniflare — is bun run --cwd apps/fouria test:software:smoke (scripts/software-restore-smoke.sh): provision → install skills → snapshot → fresh container → restore → re-provision, with five PASS: checks.

CaseInvariantMode
health probes leave the catalog unchangedI1 health is observability-onlylocal + cloud
POST /control X-FourIA-Control: backup on a running instanceI2local + cloud
async sync commits a durable job and promotes a stable pointerI12 durable job + stable pointlocal + cloud
idle instance returns skipped:'not-running' without wakingI2/I3cloud
control-backup → destroy → cold restore preserves a sentinelI4cloud
encryption at rest (data.sqsh.enc, no plaintext)I8cloud (opt-in)
retention prunes catalog + objects to BACKUP_MAX_VERSIONSI9cloud
worker is bound only to its per-run bucketI11local + cloud
reprovision 409 / force / running checkpointI5/I6/I7cloud

Promotion (dev → production) coverage

The former test/e2e-push/ suite drove the HTTP Worker → Worker push (POST /api/admin/releases/promote fetching /api/admin/releases/import). That transport is gone: promotion is a private R2 stage/pull (the dev instance writes promotion/staged/<id>.tar.gz + promotion/pending.json to its own bucket; production reads it through PROMOTION_SOURCE_BUCKET on boot, or on POST /api/admin/releases/apply). There is no import route to push to, and two local Workers cannot share a miniflare R2 bucket, so the two-worker local harness could not validate the new transport.

The transport is covered by:

  • src/services/promotion/r2-source.test.ts — stage → pending → apply → applied pointer, content hash + commit message, staged/applied history entries, the merged dev+production history view, plus target/environment/tenant gating, already-applied, no tenant state, parked, busy, missing bundle, and idempotency.
  • src/routes/api/releases.test.ts — 202 stage (message/author pass-through), production 409, apply + checkpoint, status available, history, and RBAC.
  • src/index.test.ts / src/gateway/process.test.ts — the bridge registration and the boot-time apply hook.

The dev/production axis remains the -dev hostname (isDevAgentsEnabled), not DEV_MODE.

Cloud helpers: test/e2e/fixture/r2-list and r2-clear list/empty the per-run bucket via the R2 REST API (needs CLOUDFLARE_TEARDOWN_API_TOKEN or CLOUDFLARE_API_TOKEN). The E2E deploy sets deterministic BACKUP_INTERVAL_MINUTES=1 / BACKUP_MAX_VERSIONS=2; set E2E_BACKUP_ENCRYPTION=true to run the encrypted-at-rest case. Note that the configured interval is a floor: the Worker stretches the effective scheduled cadence to max(BACKUP_INTERVAL_MINUTES, 2 × the last snapshot duration), so a slow snapshot cannot run back-to-back. Explicit snapshots (POST /api/admin/storage/sync, checkpoints) bypass that gate and stay immediate.

The same invariants are asserted at the fast layer by workers/fouria-dispatch/test/unit/cron-fanout.test.ts (the cron forwards backup only to non-frozen tenants) and src/persistence.test.ts (the “stable recovery point promotion” block: the recovery pointer advances only, fails closed on a malformed pointer, and retention skips the prune when the pointer is unreadable). Manual-restore hardening is unit-tested in src/persistence.test.ts too: a setRecoveryPoint: false restore leaves the pointer untouched, an explicit restore opens the maintenance window and stops the gateway while it runs, and restoreIfNeeded defers while that window is active. The route’s ?setCurrent=0 mapping is covered by src/routes/api/storage.test.ts, the durable-flag round-trip by src/activity/backup-jobs.test.ts / src/activity/scheduler.test.ts, and the Backups-page checkbox by src/client/components/__tests__/BackupHistoryPanel.test.tsx, plus the Cloudflare @cloudflare/vitest-plugin route tests planned in apps/fouria/test/e2e/BACKUP-E2E-DESIGN.md.

Restore legibility (which backup/date is loading) is covered by src/persistence.test.ts (readRecoveryState; the restore target is written during sandbox.restoreBackup and cleared on success), src/assets/loading.test.ts (the loading page consumes data.recovery / recovery.restoring / recovery.committed), and src/client/components/__tests__/BackupHistoryPanel.test.tsx (the recovery-point badge + summary). The local observability canary test/e2e/ad_observability_canary.txt shows how to query the local dev server’s Local Explorer SQL API (test/e2e/fixture/observability-query) to confirm the restore-loop guards (Restore deferred — tenant maintenance in progress, GatewayMaintenanceError) fire from the worker’s own logs.

Vitest 5 caveat. The Cloudflare route-integration layer is deferred: both @cloudflare/vitest-pool-workers and the renamed @cloudflare/vitest-plugin peer vitest ^4.1.0, and under Vitest 5 the pool fails with SyntaxError: Unexpected identifier 'file' (known open issue cloudflare/workers-sdk#15618; workaround: pin vitest ~4.1.11).

Restore sequencing. Cold‑restore survival (I4) is opt‑in via E2E_BACKUP_RESTORE=1 (zzz_backup_cold_restore.txt) because restoring the ~250 MB local‑binding archive is far too slow for CI (observed >13 min without completing). The default phaseE_restore cases (catalog, health‑observability, running‑backup, idle‑skip, bucket isolation, encryption, retention) do not restore and use fixture/wait-ready (one warm request, then ?passive=1 polls) so overlapping restores cannot abort each other with OperationInterruptedError: … runtime changed (root cause: BACKUP-E2E-DESIGN.md §9).

cctr corpus gotcha. cctr 0.31 silently registers zero tests for a *.txt corpus file whose first line is a # comment, or when comments sit between a file-level %skip and the first ===. Corpus files must begin with === (or a lone directive followed by ===); keep prose in test bodies or at the end of the file. cctr -l <file> prints the parsed test count.

Business Rules Corpus

Contract tests at apps/fouria/test/business-rules/ that validate cross-cutting concerns. These form the “T0 validation gate” per issue #137.

cd apps/fouria
cctr test/business-rules/
Corpus FileValidates
pricing.txtPer-action billing, success-only, BYOL discounts, VISION.md alignment
rbac.txtROOT/ADMIN/BASE_USER permission matrix, route guards, and dashboard + loading-page endpoint coverage
tenant-isolation.txtCross-tenant access controls, canAccessCustomer
operational-modes.txtDEV_MODE, E2E_TEST_MODE, DEBUG_ROUTES, DEMO_MODE, running-only backups, health observability-only, late-ready gateway adoption, health is not a routing gate (ROOT-revivable frozen), reprovision fail-closed guard, automatic doctor --fix repair for a launch blocked by a corrupt state database
cost-tracking.txt6 Analytics Engine event types, field schemas
dashboard-api-contract.txtAdmin API endpoint existence
agent-visibility.txtBASE_USER agent visibility rules
credits-and-plans.txtCredit model, plan limits, BYOL exclusion, p95 pricing

apps/lerma — Phoenix Web App

Unit Tests

Mirror lib/ structure under test/lerma/ and test/lerma_web/. Run with ExUnit via mix.

cd apps/lerma

# All unit tests (creates database, runs migrations)
mix test

# Single file
mix test test/lerma/clients_test.exs

# From the repository root: compile + deps.unlock + format + Credo + tests
cd ../..
mix precommit

Key test files:

FileTests
lerma/clients/client_test.exsClient CRUD, validation, Ecto schema
lerma/clients/user_test.exsUser model, invitation lifecycle fields
lerma/clients_test.exsInvite/accept/reject, instance-scoped Access sync, reserved staging-client protections
lerma/applications_test.exsApplication CRUD, one-time provisioning key generation/rotation/revocation and digest-only storage
lerma/staging_tenant_test.exsStaging-only client bootstrap, production isolation, and database delete/rename/truncate guards
lerma/emails_test.exsConfirmation email content and links
lerma/invitations_test.exsBackground invitation email delivery and PubSub alert
lerma/instances/instance_test.exsInstance lifecycle
lerma/instances_upgrade_test.exsUpgrade version guards, per-application filtering, and repeat-promotion idempotency
lerma/provisioning/tenant_provisioner_test.exsTenant provisioning logic
lerma/billing_test.exsBilling calculations
lerma/billing_plans_test.exsGlobal plan templates, assign/clone, autobuy toggle
lerma/billing_rules_corpus_test.exsBilling business rules (a–m) incl. production gating
lerma/billing_local_workflow_test.exsLocal workflow: AE rows → reconcile → consume; configured/unconfigured/duplicate events, watermark ownership
lerma/billing_becaria_trace_test.exsBecarIA flow: AE rows → reconcile → exact per-action credits + sums
lerma/credits/credit_pool_test.exsCredit pool management
lerma_web/live/billing_plans_live_test.exsGlobal plan library + designer
lerma_web/components/billing_plan_component_test.exsClient board plan assignment + summary
lerma/analytics/cf_analytics_test.exsCloudflare Analytics Engine queries
lerma/observability/otlp_test.exsOTLP/HTTP JSON parsing, allowlist privacy guard, child-span drop, caps, stable dedupe keys
lerma/observability/telemetry_test.exsIdempotent destination ingestion, uptime/failure ratios (unknown ≠ downtime), window helpers, retention prune
lerma/cloudflare/observability_test.exsWorker upload metadata: destination names, per-signal sampling, persist: false, disabled switch
lerma/cloudflare/observability_destinations_test.exsDestination reconcile: create/PATCH staging + production pairs, 403/500 handling, env-specific names
lerma_web/controllers/api/otlp_ingest_controller_test.exsDestination ingest auth (fail closed), 415 for protobuf, gzip delivery, privacy drop, replay idempotency
lerma_web/plugs/otlp_body_reader_test.exsGzip/JSON body decoding, size caps, malformed-body 400, path/method passthrough
lerma_web/controllers/api/observability_controller_test.exsUptime endpoint ratios/status/windows, unknown when no observations, sampling rate surfaced
lerma_web/components/worker_telemetry_widget_test.exsWorker telemetry widget unknown/healthy/degraded render
lerma_web/components/health_events_widget_test.exsHealth events timeline empty state, recorded transitions, dev-instance rendering
lerma_web/controllers/api/instance_controller_test.exsInstance API endpoints
lerma_web/controllers/api/provisioning_controller_test.exsGlobal/application-key auth, application-scoped upgrade and status
lerma_web/controllers/api/ai_gateway_controller_test.exsGlobal provisioning-key auth; fails closed when unset
lerma_web/controllers/api/credit_usage_controller_test.exsBilling reconcile + credit-usage API (bearer auth)
lerma_web/live/instance_live_test.exsInstance LiveView
lerma/client_docs_test.exsClient docs sources (proposal + per-env stores), R2 listing/reading, default document formats, path traversal
lerma/client_docs/markdown_test.exsSafe Markdown preview: frontmatter, headings, tables, lists, code, relative images, unsafe links, HTML escape
lerma_web/controllers/client_docs_controller_test.exsRaw docs responses: content type, inline/download disposition, sandbox CSP, not-found/invalid-path errors
lerma_web/live/client_live_test.exsInvitation UI, public acceptance flow, and the lazy documentation widget tree/preview/live
lerma_web/live/application_live_test.exsROOT-only application key generation and one-time reveal
lerma_web/plugs/cf_access_auth_test.exsCF Access auth plug

Per-test tags:

  • @tag cf_auth: %{email: "...", name: "..."} — injects CF Access authentication
  • @tag :integration — excluded by default, must run explicitly via mix test --only integration

Staging Fouria build promotion

.github/workflows/promote-staging.yml listens for completed Test runs on develop, then queries the source run’s Actions jobs and proceeds only when build-and-push concluded success. The script promotes the immutable unstable-<sha> image/bundle pair, no-ops when the staging client tenant is already running that version, and polls Lerma’s status endpoint until the running application instances converge. The GitHub secret STAGING_PROVISIONING_API_KEY must match the one-time generated key for the Fouria application in staging Lerma. This cloud workflow is not run by local mix test; the application-scope/auth/idempotency contracts are covered by the ExUnit tests above.

E2E Tests (cctr)

Sequential cctr tests against real Cloudflare dispatch namespace infrastructure.

cd apps/lerma
mix test.e2e

Setup (test/e2e/_setup.txt):

  1. Validates jq, uuidgen tools
  2. Sources .dev.vars for Cloudflare credentials
  3. Validates CLOUDFLARE_API_TOKEN, CF_ACCOUNT_ID, WORKERS_SUBDOMAIN, CF_ENVIRONMENT
  4. Computes dispatch namespace (fouria-tenants vs fouria-tenants-staging)
  5. Starts Phoenix on port 4002 via mix phx.server

Teardown (test/e2e/_teardown.txt):

  1. Reads client/instance IDs from fixture directory
  2. Deletes instance (tears down CF worker + container application) via API
  3. Deletes client via API — fully decommissions the tenant (remaining instances and container applications, Access app/group, dedicated AI Gateways, R2 backup buckets) and archives the client record
  4. Kills Phoenix process

Test files:

FileTests
client_provisioning.txtCreate client → create instance → wait for running → verify gateway → verify CF metadata
env_isolation.txtCorrect namespace, wrong namespace rejection, environment tags, Analytics Engine dataset names

Billing trace: deterministic + local

The former staging billing-trace suite (test/e2e_billing_trace/) drove a deployed worker’s public POST /api/otel/v1/traces. That endpoint no longer exists: OTLP ingest is the sandbox bridge (http://otel.internal/v1/traces), resolved by the Sandbox SDK inside the Worker, so it is only reachable from inside the tenant container. There is no public ingest route and no Access path app to configure.

The five BecarIA actions (telegram_query, queue_reevaluation, signal_analysis, email_draft, review_analysis, 1 + 1 + 3 + 5 + 10 = 20) are still covered end to end by:

  • test/lerma/billing_becaria_trace_test.exs (deterministic, stubbed Analytics Engine stream, runs in CI), and
  • test/e2e_billing_local/ (mix test.e2e.billing:local, fake Analytics Engine, no Cloudflare account).

For a manual replay inside a container, run apps/fouria/scripts/emit-billing-actions.ts with --base-url http://otel.internal (the bridge keeps OTEL_SHARED_SECRET). ANALYTICS_ENGINE_API_TOKEN is Worker-only and irrelevant to emission — the Worker writes the billable row with its ANALYTICS binding; the token only powers the dashboard read-back and is never given to a container.

Local Billing-Workflow E2E (fake Analytics Engine)

test/e2e_billing_local/ (opt-in, run separately from mix test.e2e) exercises the billing consume path entirely locally — no Cloudflare account, no staging DB, no deployed worker:

fake Analytics Engine (bun)
  └─ POST /accounts/:id/analytics_engine/sql   ← BillingReconcile reads here
       └─ lerma (mix phx.server, MIX_ENV=test)
            └─ POST /api/credits/v1/reconcile → BillingConsumption → GET /usage

It uses a mock billing definition (fixture/seed_plan.exs, not BecarIA) and feeds synthetic billable-action AE rows directly (OpenClaw’s OTel emission is owned/tested by the OpenClaw team). Cases: configured + match_rule events, unconfigured/unmatched/blank events, duplicate action_id redelivery, AE outage (502, no billing), and tenant isolation.

cd apps/lerma
mix test.e2e.billing:local

Why a fake AE and not Cloudflare Local Explorer? Local Explorer (docs) does not support Analytics Engine — it is absent from the supported-bindings table and from the Local Explorer OpenAPI, and local traces emit no writeDataPoint span. A fake of the AE SQL API is therefore the only local observability surface for the billable-event stream. Local Explorer is used on the Worker side for logs/spans (apps/fouria/test/e2e/ad_observability_canary.txt). See apps/lerma/test/e2e_billing_local/README.md.

Test Infrastructure Configuration

apps/lerma/mix.exs:

test: ["ecto.create --quiet", "ecto.migrate --quiet", "test"]
"test.e2e": ["cmd cctr test/e2e/ --sequential -v"]

Root mix.exs: The credo alias forwards all arguments to each Elixir workspace project, so mix credo --strict works from the repository root without running Credo against the root workspace shell project.

credo: &run_credo/1,
check: [
  "workspace.run -t format -- --check-formatted",
  "credo --strict"
],
precommit: [
  "workspace.run -t compile -- --warnings-as-errors",
  "workspace.run -t deps.unlock -- --unused",
  "workspace.run -t format",
  "credo --strict",
  "license_audit",
  "workspace.run -t test"
]

# Private alias helper:
defp run_credo(args) do
  args = if "--strict" in args, do: args, else: ["--strict" | args]
  Mix.Task.run("workspace.run", ["-t", "credo", "--" | args])
end

test_helper.exs: Excludes :integration tests, starts SQL sandbox.

Mocking: Uses Bypass for HTTP mocking and Meck for Elixir mocking. Not available outside :test env.


packages/moltlazy — Config Module + SDK

Unit Tests (Vitest)

cd packages/moltlazy
bun run test          # vitest run
bun run test:watch    # watch mode

Config module tests (tests/config/):

FileTests
gateway.test.tsGateway config section (port, mode, trustedProxies, allowedOrigins)
session.test.tsSession config (dmScope)
tools.test.tsTool search configuration
logging.test.tsLogging section (level, format, transports)
validate.test.tsCLI validation-error formatting (object-shaped openclaw errors)
include.test.ts$include directive injection

SDK module tests (tests/):

FileTests
sdk.test.tsSDK client initialization
sdk-cli.test.tsSDK CLI fallback mechanisms
sdk-gateway-status-ws.test.tsGateway WebSocket status checks
sdk-ws-client.test.tsWebSocket client connection lifecycle

Agent tests (tests/):

FileTests
agents.test.tsAgent defaults, list generation
index.test.tsMain module (patch flow, section ordering)
schema.test.tsZod schema validation for moltlazy config
obsidian-sync.test.tsObsidian sync skill configuration
coverage-gaps.test.tsCoverage gap tracking across modules

E2E Tests (cctr)

cd packages/moltlazy
bun run test:e2e        # Build + cctr
Test SuiteScenarios
cli/basic.txtmoltlazy patch from empty, --help displays man-style docs, -h
cli/flags.txt--file override, --help, unknown commands
cli/errors.txtValidate pass/fail, nonexistent file, filesystem errors
full-flow/roundtrip.txtFull patch → verify output (gateway, $include, agents, session)
skills/discovery.txtSkills directory discovery, empty skills dir, re-patching
workspace/seeding.txtSOUL.md creation, fouria-builder workspace, safe mode, immutability

Setup: Validates that dist/cli.js exists (pre-build check).

OpenClaw CLI version: the moltlazy-integration CI job validates the generated config with the OpenClaw CLI pinned to the version baked into the container (root package.json catalog, currently 2026.7.1-2) instead of openclaw@latest. This keeps the config-schema contract aligned with what is actually deployed and avoids flaky failures from upstream schema drift. Bump the pin in .github/workflows/test.yml and this doc together whenever the container’s OpenClaw version changes.

Smoke Tests (Vitest)

cd packages/moltlazy
bun run test:all       # Unit + E2E

OpenClaw plugins (external)

The cloudflare-unified-billing plugin — its unit tests, live AI Gateway integration tests (infer-smoke, provider-smoke, cf-cost-api), and CI — moved to Paso4/openclaw-plugins. Run them from a checkout of that repo:

cd plugins/provider/cloudflare-unified-billing
bun run test              # unit
bun run test:integration  # live (requires AI Gateway creds + PLUGIN_LIVE_TESTS=true)

See OPENCLAW-PLUGINS.md.


CI Pipeline

The .github/workflows/test.yml workflow defines the CI pipeline (see reference table in AGENTS.md for the full job graph):

unit ──┬─ build-and-push → e2e   (PR: single shard · develop: sharded matrix)
       └─ moltlazy-integration   (only when packages/moltlazy changed)

changes ── iac-test              (only when apps/*/iac changed)

The changes job git-diffs the event SHAs and gates iac-test and moltlazy-integration, so unrelated changes no longer spin them up. The local-integration job was removed.

The Changesets version pull request runs no CI. The “Version Packages” PR is opened from the changeset-release/develop branch by .github/workflows/version-packages.yml; it only bumps versions and writes changelogs, so every PR-triggered workflow exits immediately for that head branch — test.yml (changes, unit, validate-secrets, build-and-push), lerma.yml (quality, test), staging-validation-ci.yml and changeset-status.yml. The guard is the head-ref prefix !startsWith(github.head_ref, 'changeset-release/'), which is empty on push/workflow_dispatch, so those paths are unaffected. Skipped jobs report as skipped (not pending) to branch protection, and no CD workflow can be affected: both promote-staging.yml and deploy-dashboard.yml only ever act on a successful run whose head_branch is develop.

build-and-push builds the worker, validates the wrangler multipart bundle via scripts/validate-bundle.mjs, then builds the container image and pushes it to the Cloudflare registry as fouria:unstable-<sha> and uploads the worker bundle to the fouria-bundles R2 bucket as unstable-<sha>/fouria.mjs. On develop, e2e consumes that image (via IMAGE_REF) instead of rebuilding it. The deploy-dispatch.yml workflow also deploys workers/fouria-dispatch (staging on push to develop, production on a stable CalVer release or a manual workflow_dispatch). deploy-fouria.yml is the fouria CD: on a stable CalVer release (vYYYY.M.D / same-day vYYYY.M.D-N, classified from the tag name, not the GitHub prerelease flag) it promotes the CI-built unstable-<sha> artifacts — retagging that image to latest + the release tag and copying the bundle to R2 under the release tag + latest. Nothing is rebuilt. Release commits are pushed with [skip ci], so they never re-trigger this CI pipeline.

Production deploys are stable-tag-only: lerma filters the deployable versions to stable release tags and Instances.update_instance_version/3 rejects unstable ones (staging keeps the unstable channel). The cleanup-stale-images.yml cron never deletes an image (or matching R2 bundle) still referenced by a container application — including in-progress rollout versions — and fails closed if that in-use set cannot be resolved. Covered by apps/lerma/test/lerma/cloudflare/image_registry_test.exs, apps/lerma/test/lerma/cloudflare/registry_bundle_test.exs, apps/lerma/test/lerma/instances_test.exs, and apps/lerma/test/lerma_web/live/instance_live_test.exs.

Container image upgrades roll out with strategy: "rolling" (a single step_percentage: 100 step) — see Lerma.Cloudflare.Containers. The tenant container is Durable-Object-managed, and a new_instances rollout does not replace a DO-bound instance: the platform stages the target as a rollout version at 0% and the application keeps its previous image, so new_instances silently left promoted tenants running the old image. Promotion convergence is also sourced from the live container image tag (Containers.fetch_deployed_config/1) in Instances.at_requested_version?/2, not the recorded deployed_version, so a promotion cannot report success while the container still runs the previous image (and a genuine miss can be re-promoted instead of being skipped). Covered by apps/lerma/test/lerma/cloudflare/containers_test.exs and apps/lerma/test/lerma/instances_upgrade_test.exs.

Plugin Integration (unified billing e2e) — moved

The cloudflare-unified-billing e2e tests and their dedicated workflow moved to Paso4/openclaw-plugins. That repo runs live AI Gateway integration tests on develop only (they cost real credits), decoupled from this monorepo’s test.yml.

Key env vars for CI:

  • SKIP_CLI_E2E=true — used in unit job to skip CLI-dependent tests
  • E2E_SKIP_BROWSER_TESTS=1 — skip Playwright browser tests
  • RUN_REASONING_TESTS, RUN_VISION_TESTS — gate expensive integration tests

Writing Tests

TypeScript (Vitest)

  • Colocate: Place *.test.ts next to source files in apps/fouria/src/ and packages/moltlazy/tests/
  • Strict mode: Tests use strict: true TypeScript
  • Mocking: Use vitest’s vi.mock() for module-level mocking
  • Lifecycle: beforeAll/afterAll for setup/teardown; beforeEach/afterEach for per-test state
  • Pattern: Arrange → Act → Assert (AAA)
  • Coverage: ≥80% for new code (≥95% for API routes in apps/fouria)

Elixir (ExUnit)

  • Mirror structure: Test files mirror lib/ paths under test/
  • Database: Use Ecto.Adapters.SQL.Sandbox for concurrent DB tests
  • HTTP mocking: Use Bypass for external HTTP requests
  • Auth helpers: Use @tag cf_auth in conn_case.ex to inject CF Access headers
  • Integration tests: Tag with @tag :integration (excluded by default)
  • Fixtures: Define in test/support/fixtures/
  • Convention: describe "function_name/arity" blocks; test names as strings

cctr (Corpus Contract Tests)

  • Format: Plain text test files with === (description) → script → --- → assertion sections
  • %require: First test case in a file with %require cascades failures to skip remaining tests
  • Variables: Capture dynamic output with {{ name }} — used for tokens, URLs, etc.
  • Assertions: Use where clauses (* s contains "ready", * status == "200")
  • Fixtures: The fixture/ directory is mirrored to $CCTR_FIXTURE_DIR
  • Setup/teardown: _setup.txt runs before the suite, _teardown.txt after
  • Shared helpers: apps/fouria/test/lib/e2e-helpers.sh provides e2e_wait_for_health (immediate, bounded readiness probe) and e2e_classify_ws_upgrade (WS-upgrade verdict), unit-tested in apps/fouria/src/e2e-helpers.test.ts. e2e_wait_for_health is called by the serial (test/e2e) and agent (test/e2e-agent) _setup.txt corpora and by the parallel driver’s top-level wait (test/e2e-parallel/run-parallel.sh). ab_gateway_ws_origin.txt sources the library only for e2e_classify_ws_upgrade and keeps its own health poll; the parallel driver’s generated group _setup.txt also keeps its own grace-free bounded health poll.
  • Local mode guard: Tests that require cloud infrastructure use:
    if [ "${E2E_LOCAL:-}" = "true" ]; then
        echo "SKIP: ..." >&2
        exit 1
    fi

Business Rules

  • Location: apps/fouria/test/business-rules/
  • Format: cctr .txt files
  • Purpose: Cross-cutting behavioral contracts (pricing, RBAC, tenant isolation, etc.)
  • When to update: Any change affecting pricing, RBAC, tenant isolation, operational modes, cost tracking, dashboard API, agent visibility, or credits/plans
  • Run: cctr apps/fouria/test/business-rules/

Validation Checklist

Before submitting changes that modify test infrastructure or add new tests:

  • bun run typecheck passes
  • bun run lint passes
  • bun run test passes (all JS/TS packages)
  • For apps/fouria: bun run test:coverage passes
  • For apps/lerma: mix test passes
  • New code has ≥80% unit test coverage (≥95% for API routes)
  • If adding routes: add tests
  • If changing env vars: update .dev.vars.example, ENV-VARIABLES.md, and apps/fouria/src/gateway/env.ts
  • If changing business rules: update corresponding corpus file and run cctr apps/fouria/test/business-rules/
  • For apps/fouria local E2E: bun run test:e2e:local:cli passes (cloud-only tests self-skip)