F FourIA staging 2026.10.29 GitHub ↗

Fouria

The multi-tenant Cloudflare Worker that runs OpenClaw inside a Sandbox container.

Cloudflare Worker that manages OpenClaw instances inside Cloudflare Sandbox containers. This is the core component of FourIA.

Dev guide: docs/WORKER-DEVELOPMENT.md — deep-dive architecture, frontend/backend patterns, RBAC Cloudflare setup & operations: docs/OPERATOR-GUIDE.md — Access, R2, AI Gateway, Tunnel, CDP, backups, rescue/restart, troubleshooting Project docs: ../../docs/DEVELOPMENT.md — full onboarding guide Env vars: ../../docs/ENV-VARIABLES.md — complete reference Operating & recovery: docs/OPERATOR-GUIDE.md — health, debug options, backups, rescue/restart, troubleshooting


Architecture

Browser → Worker (Hono) → Sandbox Container (OpenClaw Gateway on :18789)
                              │
                    start-openclaw.sh:
                      1. OpenClaw onboard (auth setup)
                      2. moltlazy patch (config injection)
                      3. OpenClaw gateway run

The Worker proxies HTTP/WebSocket to the gateway, provides the admin dashboard at /_admin/, and exposes a REST API at /api/*.


Development

# Install dependencies (from repo root)
bun install

# Build shared packages (moltlazy + plugin)
bun run build:all

# Create .dev.vars (from root of this package)
cp .dev.vars.example .dev.vars
# Add at minimum: CLOUDFLARE_AI_GATEWAY_API_KEY (+ CF_AI_GATEWAY_ACCOUNT_ID + CF_AI_GATEWAY_GATEWAY_ID), MOLTBOT_GATEWAY_TOKEN
# For local unified-billing REST requests, also set CLOUDFLARE_API_TOKEN to the same AI-scoped credential.

Running Locally

# Terminal 1: Frontend (Vite hot-reload on :8787)
bun run dev

# Terminal 2: Backend Worker (Cloudflare Vite plugin, workerd on :8788)
bun run dev:worker

# Full local stack (build all + Vite dev server)
bun start

Frontend and Worker changes both hot-reload. The Worker dev server runs in workerd through the Cloudflare Vite plugin — the same plugin used by vite build, so local and deployed behavior stay aligned.

Exposing the dev server through a tunnel (Telegram webhooks, external clients): Vite rejects unknown Host headers by default. Pass the public hostname (comma-separated; a leading dot allows a whole domain):

DEV_ALLOWED_HOSTS=foo.trycloudflare.com bun run dev:worker
# or: DEV_ALLOWED_HOSTS=.trycloudflare.com bun run dev:worker

Use the Worker port (:8788), not the SPA port (:8787), for webhook ingress.

Vite Proxy

bun run dev (vite.ui.config.ts) proxies /api/*, /debug/*, /cdp/*, and static assets to the Worker on port 8788. The Worker itself is served on port 8788 by bun run dev:worker (vite.config.ts + @cloudflare/vite-plugin, which also builds/launches the sandbox container image through Docker).


Key Files

FilePurpose
src/index.tsMain Hono app: middleware pipeline, WS proxy, route mounting
src/routes/api.tsAPI route aggregator (delegates to domain modules)
src/gateway/process.tsContainer lifecycle: find, start, restart, destroy
src/gateway/env.tsBuilds env vars passed to container
src/auth/CF Access JWT auth + RBAC (ROOT/ADMIN/BASE_USER)
src/client/React SPA (Vite 6, React 19, react-router-dom 7)
start-openclaw.shContainer entrypoint: onboard → moltlazy patch → gateway
DockerfileSandbox container image
wrangler.jsoncCloudflare Worker config (bindings, envs, DO migrations)
iac/OpenTofu infrastructure-as-code

Frontend (src/client/)

React 19 SPA built with Vite 6, served from /_admin/.

Pages

PageRoutePurpose
Dashboard Home/_admin/Overview and status
Agent Studio/_admin/agentsRPC agent workbench
Connections/_admin/connectionsChannel integrations
Knowledge/_admin/knowledgeMemory wiki and vault
Integrations/_admin/integrationsIntegration configuration
Secrets/_admin/secretsVault management
Backups/_admin/backupsBackup history and restore
Diagnostics/_admin/diagnosticCosts, usage, OTel metrics
Audit Logs/_admin/audit-logsActivity log viewer
Automation/_admin/automationWorkflow editor

Agent Studio is the all-in-one, read-first agent workbench: roster, policy, tool catalog + session-effective surface, skills, MCP servers, memory/bootstrap budget, automations/runs, bindings, and health. It talks to the Gateway over Admin HTTP RPC and one Gateway WebSocket per request — no openclaw CLI and no shell — and it never wakes a sleeping instance (it reports available: false, reason: "sleeping" instead). Bootstrap edits use agents.files.set with optimistic hashes; bindings use config.patch.

API Client (src/client/api.ts)

The frontend API client calls /api/admin/* endpoints. Most routes wrap OpenClaw CLI calls inside the Sandbox container; the Agent Studio routes (/api/admin/agent-studio*) are RPC-only and read through config.get, agents.list, agents.files.*, agents.workspace.*, tools.*, skills.*, plugins.list, hooks.status, cron.*, sessions.*, memory.status and system.info.


Backend (src/)

API Routes

All API routes are at /api/admin/*, protected by CF Access auth + RBAC:

ModuleEndpoints
misc/whoami, /version, /logs
devicesDevice pairing approval
gatewayStatus, restart, lifecycle
configGet/set/validate/fix config
agentsAgent CRUD, bindings, persona
modelsModel listing and selection
channelsChannel status and accounts
integrationsIntegration config
knowledgeKnowledge tree, wiki management
secretsVault audit and reload
dreamingMemory dreaming config
costsCost pricing and metrics
cronCron job management
vaultEncrypted secrets store
audit-logsActivity log retrieval
usersTeam member management

RBAC Roles

RolePermissionsAccess Scope
ROOTadmin:platform, admin:customer, api:allAll tenants, debug endpoints
ADMINadmin:customer, api:allOwn tenant only
BASE_USERapi:ownOwn tenant, read-only

Operational Modes

VariablePurpose
DEV_MODESkips CF Access auth + bypasses device pairing
E2E_TEST_MODESkips CF Access auth but keeps device pairing
DEBUG_ROUTESEnables /debug/* endpoints (ROOT role only)
DEMO_MODEEnables read-only dashboard with sample data
BILLABLE_EVENTSOpt-in for billable-action emission on a dev instance (TRUE); production and staging always emit
STARTUP_LOG_VERBOSITYClean boot-log verbosity for the loading page: steps | detailed (default) | verbose

Health Checks

Every User Worker exposes a single, unified health endpoint that all internal and external consumers share.

GET /api/health — unified snapshot (public, no auth)

This is the source of truth for instance health. It runs Kubernetes-style probes against the OpenClaw gateway inside the sandbox container and returns a structured snapshot. lerma’s HealthMonitor/HealthPoller poll it to keep clients.status / instances.status accurate, the loading page polls it during cold start, and the admin dashboard (/_admin) streams it.

{
  "ts": "2026-08-15T12:00:00.000Z",
  "startup": { "ok": true, "status": "success", "latencyMs": 12 },
  "readiness": { "ok": true, "status": "success", "latencyMs": 8 },
  "liveness": { "ok": true, "status": "success", "latencyMs": 5 },
  "restoreStatus": { "hadBackup": true, "restored": true },
  "status": "running",
  "processId": "proc_123",
  "processType": "gateway"
}
FieldTypeDescription
startupProbeResultOne-time init: liveness + readiness + backup-restore confirmation
readinessProbeResultReady to serve traffic: script exited, gateway binary running, /health up
livenessProbeResultGateway /health responds with {"ok": true}
restoreStatus{ hadBackup, restored }Whether an R2 backup existed and was restored
statusrunning|starting|stoppedCanonical gateway status derived from the probes
processIdstring?Detected gateway/startup-script process id, if any
processTypegateway|script?Whether the detected process is the gateway binary or the startup script

ProbeResult = { ok: boolean, status: "success"|"failure"|"unknown", latencyMs: number, error?: string }.

Status derivation:

Conditionstatus
readiness.okrunning
Readiness fails, a process existsstarting
Readiness fails, no processstopped

Activating vs passive. By default a call to /api/health warms the container — if the gateway is not running it triggers ensureGateway + restoreIfNeeded (this is how lerma keeps sleepers alive and recovers stopped instances). Pass ?passive=1 to fail fast instead of warming; passive probes use a short timeout and never start the container. The ROOT diagnostics dashboard uses passive probes so an operator can debug an instance whose container is down.

curl -s https://{slug}.fouria.io/api/health           # activating (warm + recover)
curl -s https://{slug}.fouria.io/api/health?passive=1  # passive (fail fast, ROOT debug)

GET /healthz — fast liveness probe (non-waking)

A Kubernetes-style liveness probe that returns immediately and never starts a container or probes the gateway. It does one non-waking isRuntimeActive() read and answers 200 with the Worker liveness plus the container-runtime state:

curl -s https://{slug}.fouria.io/healthz
# {"ok":true,"ts":"…","status":"active"}   # or "idle" (sleeping/stopped)

Use this for uptime checks / load-balancer liveness where an answer must arrive in well under a second. Without a dedicated route /healthz fell through to the gateway proxy, which warms the container and waits for the gateway port — ~108 s on a cold instance, long enough for the platform to kill the request before it answered. Gateway readiness is reported by /api/health (503 while starting) and /api/health-snapshot.

/healthz sits on the tenant hostname, so it is covered by the host Access application (ROOT operators, client users, or a host-app service token) — it is not a public endpoint.

GET /api/health-ws, GET /api/health-snapshot, GET /api/health-components — Browser-facing health endpoints

On tenant hostnames /api/health and /api/health/* are covered by the path-scoped FourIA Health - <slug> Cloudflare Access application — a different Access app (with its own immutable AUD) than the host app the browser logged into. Access answers requests there with a 302 redirect to https://<team>.cloudflareaccess.com/cdn-cgi/access/login/…:

  • A WebSocket upgrade cannot follow a 302 redirect at all.
  • A browser fetch() cannot follow a cross-origin redirect to *.cloudflareaccess.com because the Access login endpoint sends no Access-Control-Allow-Origin header (CORS error).

Browser clients (the dashboard and loading.html) therefore use sibling paths outside /api/health/*, which fall through to the host Access app whose CF_Authorization session the browser already holds:

wss://{slug}.fouria.io/api/health-ws?passive=1          # WebSocket stream (mirrors /api/health/ws)
GET https://{slug}.fouria.io/api/health-snapshot?passive=1  # HTTP snapshot (mirrors /api/health)
GET https://{slug}.fouria.io/api/health-components          # Cloudflare component states (mirrors /api/health/components)

GET /api/health/ws — legacy WebSocket stream (non-browser clients)

Identical stream, kept mounted for clients that authenticate to the health app itself (e.g. the lerma health-monitor service token). Browser clients must use /api/health-ws above.

wss://{slug}.fouria.io/api/health/ws?passive=1

GET /api/status — legacy alias (public, no auth)

Kept for backward compatibility (E2E scripts, server/wait-ready, dispatch e2e). Delegates to the same snapshot and returns the legacy shape { ok, status, processId, e2eMode, pairing }; pairing is { enabled: true } when healthy.

GET /api/startup-progress — boot stepper + log (public, no warm-up)

Drives the loading page during cold start. Returns the boot-phase stepper and curated boot log emitted by start-openclaw.sh (see STEPS/mark_step in the script and WORKER-DEVELOPMENT.md). Fail-fast and never warms the container (no ensureGateway/restoreIfNeeded), so it adds no load to the health path.

{
  "available": true,
  "steps": [
    { "id": "init", "label": "Preparing container" },
    { "id": "onboard", "label": "Configuring gateway" }
  ],
  "currentStep": 1,
  "currentStepId": "onboard",
  "totalSteps": 7,
  "skipped": false,
  "ts": 1787052449,
  "log": "[startup] (1/7) Preparing container\n..."
}

Returns { "available": false } when the container is cold/off or the image hasn’t emitted progress yet.

Protected status endpoints (CF Access)

GET /api/admin/gateway/status/polling and GET /api/admin/gateway/status/ws are thin aliases over the same snapshot for the authenticated admin API and the moltlazy SDK. The POST /api/admin/gateway/restart handler performs an in-process soft reload via SIGUSR2; it does not create a snapshot. (SIGUSR1 is not a restart signal in current OpenClaw — it starts Node’s inspector on 127.0.0.1:9229.) Use POST /api/admin/storage/sync for an explicit checkpoint.

For a checkpointed hard restart (checkpoint → kill → restore marker → reprovision through start-openclaw.sh), use POST /debug/restart-openclaw (requires DEBUG_ROUTES=true; button in the diagnostic dashboard). If the checkpoint fails, the gateway is left running.

See ../../docs/WORKER-HEALTH.md for the full probe contract and lerma consumer semantics.


Cloudflare API Token Permissions

This project does not use a Cloudflare API token at runtime — per-tenant secrets are provisioned by lerma. The following tokens are used during deployment and E2E testing:

TokenPermissionPurpose
DEPLOY_WORKER_API_TOKENWorkers Scripts:Editwrangler deploy and wrangler secret bulk
LERMA_API_TOKEN (CI only)Full provisioning scopeE2E test infrastructure (Tofu init)

All credentials are documented in docs/schemas/secrets-manifest.schema.json.

Testing

Unit Tests (Vitest)

bun run test             # Unit tests (Vitest)
bun run test:coverage    # With coverage report
bun run test:watch       # Watch mode
bun run typecheck        # TypeScript check
bun run lint             # oxlint
bun run format:check     # Formatting check

E2E Tests

Two E2E test tiers exist: cloud-deployed (cctr, real CF infra) and local-API (Vitest, mocked deps).

bun run test:e2e          # Full E2E: cctr + Playwright (deploys real infra)
bun run test:e2e:cli      # CLI-only E2E: cctr (skips browser tests)
bun run test:e2e:parallel # Parallel E2E (one worker, dependency-grouped groups)

E2E Test Design Guide

Tests are in test/e2e/. Each .txt file is a cctr corpus — a plain-text test script where each section is a shell command with assertions.

cctr File Structure
===                           ← section header (test name or empty)
command description           ← %require means: if this fails, skip rest
%require                      ← optional: block subsequent tests on failure
===
shell command(s) here          ← executed with /bin/bash
---
{{ output_name: type }}        ← capture command output as typed variable
---
where                         ← assertion block
* output_name contains "text"  ← substring check
* output_name matches "^\\d+"  ← regex check
* output_name != ""            ← inequality
Naming Convention

Files run in alphabetical order. Prefix determines execution sequence:

PrefixPurposeExample
_setupDeploy infra, start browser, wait_setup.txt
_teardownDump logs, stop browser, destroy_teardown.txt
a_cli_*CLI integration tests (core APIs)a_cli_misc.txt, b_cli_devices.txt
f_cli_*CLI integration tests (advanced)f_cli_agents.txt, g_cli_secrets.txt
pairing_*Browser-based (plwr) testspairing_and_conversation.txt
z_cli_*Staging/prod-specificz_cli_staging.txt
zzz_*Must run lastzzz_cron_wake.txt
cctr Features
FeatureSyntaxExample
Section header=== on its own lines, with description between=== / wait for gateway / ===
Block on failure%require on line after ===%require → if fails, skip all later tests
Output capture{{ name: type }}{{ result: json object }}
Assertionswhere block after output* result contains "pending"
Regex assertion* name matches "pattern"* version matches "^v\\d+"
Equality* name != ""* output != ""
Complex objectsParenthesized boolean expressions* (status.ok == true) or (status.ok == false)
Variable Types
TypeUse case
stringText output (version numbers, status codes)
json objectJSON {} response (devices, config, agents)
json arrayJSON [] response (list endpoints)
Fixtures

The fixture/ directory contains helper scripts that get copied to $CCTR_FIXTURE_DIR at runtime:

ScriptPurpose
debug-cliExecutes OpenClaw CLI commands in sandbox via /debug/cli endpoint. Extracts JSON from mixed stdout.
curl-authAuthenticated curl against the deployed Worker URL
server/startDeploys Worker + Terraform infra to Cloudflare
server/stopDeletes Worker + destroys Terraform resources
server/deployDeploys worker code + builds all dependencies
server/wait-readyPolls /api/health until the gateway is up
start-browserLaunches Playwright Chromium for plwr tests
stop-browserStops Playwright and saves video
Example: CLI E2E Test
# f_cli_agents.txt — Agent CRUD via debug CLI endpoint
===
list agents from openclaw.json
===
WORKER_URL=$(cat "$CCTR_FIXTURE_DIR/worker-url.txt")
./curl-auth -s "$WORKER_URL/debug/container-config" | jq '.config.agents'
---
{{ result: json object }}
---
where
* result contains "list"
* result contains "defaults"

===
create a test agent via CLI
%require
===
WORKER_URL=$(cat "$CCTR_FIXTURE_DIR/worker-url.txt")
AGENT_ID="e2e-test-agent-$(date +%s)"
./debug-cli "openclaw agents add $AGENT_ID --workspace ... --json"
---
{{ result: json object }}
---
where
* result contains "id" or result contains "success"

===
delete test agent
===
./debug-cli "openclaw agents delete $AGENT_ID --force --json"
---
{{ result: json object }}
---
where
* result contains "id" or result contains "success" or (result contains "error" and result contains "not found")
Example: Browser E2E Test (plwr)
# pairing_and_conversation.txt — Device approval + chat via browser
===
navigate to admin page
%require
===
if [ "${E2E_SKIP_BROWSER_TESTS:-}" = "1" ]; then
    echo "Skipping browser test"
    exit 0
fi
TOKEN=$(cat "$CCTR_FIXTURE_DIR/gateway-token.txt")
WORKER_URL=$(cat "$CCTR_FIXTURE_DIR/worker-url.txt")
plwr -S moltworker-e2e open "$WORKER_URL/_admin/?token=$TOKEN" -T 120000

===
wait for pending devices and approve
%require
===
plwr -S moltworker-e2e wait 'button:has-text("Approve All")' -T 120000
plwr -S moltworker-e2e click 'button:has-text("Approve All")' -T 10000
plwr -S moltworker-e2e wait-not 'button:has-text("Approv")' -T 30000

===
send math question and verify answer
===
plwr -S moltworker-e2e fill textarea 'What is 847293 + 651824? Reply with just the number.'
plwr -S moltworker-e2e press Enter
plwr -S moltworker-e2e wait 'text=1499117' -T 120000
Running E2E via act (CI Simulation)

NixOS users can run CI workflows locally with act. The E2E job deploys real Cloudflare infrastructure (Worker + Terraform), runs tests, and destroys it.

# Prerequisites: create a GitHub personal access token (classic) with repo scope
# Store it and Cloudflare credentials in a .secrets file:
cat > .secrets << 'SECRETS'
GITHUB_TOKEN=ghp_your_personal_access_token
LERMA_API_TOKEN=your_lerma_api_token
E2E_CF_ACCOUNT_ID=your_account_id
E2E_WORKERS_SUBDOMAIN=your_subdomain
E2E_CF_ACCESS_TEAM_DOMAIN=yourteam.cloudflareaccess.com
E2E_R2_ACCESS_KEY_ID=your_r2_access_key
E2E_R2_SECRET_ACCESS_KEY=your_r2_secret_key
CLOUDFLARE_AI_GATEWAY_API_KEY=your_ai_gateway_key
CF_AI_GATEWAY_ACCOUNT_ID=your_account_id
CF_AI_GATEWAY_GATEWAY_ID=your_gateway_id
SECRETS

# Run the unit + lint workflow (no Cloudflare credentials needed beyond GITHUB_TOKEN)
act -W .github/workflows/test.yml -j unit

# Run the full E2E job (requires Cloudflare credentials)
act -W .github/workflows/test.yml -j e2e

# Run specific event (e2e only runs on develop branch by default, so simulate push to develop)
act -W .github/workflows/test.yml -j e2e -e <(echo '{"ref":"refs/heads/develop"}')

The GITHUB_TOKEN is needed because act clones GitHub Actions (actions/checkout, etc.) via HTTPS. Without a valid token, git operations fail with “authentication required.” Create a classic PAT with repo scope.

Running E2E Tests Directly (no act)

If you have Cloudflare credentials in a .dev.vars file, run cctr directly:

# 1. Copy E2E credentials
cp test/e2e/.dev.vars.example test/e2e/.dev.vars
# Edit .dev.vars with your Cloudflare credentials

# 2. Install tools
npx playwright install chromium  # For browser tests
# cctr: cargo install cctr or brew install andreasjansson/tap/cctr
# plwr: see https://github.com/andreasjansson/plwr

# 3. Run tests
cctr test/e2e/                           # All tests
cctr test/e2e/ -p pairing                # Specific test file
cctr test/e2e/ -vv                       # Verbose (stream output)
PLAYWRIGHT_HEADED=1 cctr test/e2e/       # See browser during tests
E2E_SKIP_BROWSER_TESTS=1 cctr test/e2e/  # CLI tests only

Vitest E2E Pattern (for API routes)

For tests that don’t need real Cloudflare infra, use Vitest with a real Hono app instance:

import { Hono } from 'hono';
import { createMockEnv, suppressConsole } from '../src/test-utils';

function createTestApp() {
  const app = new Hono<AppEnv>();
  app.use('*', async (c, next) => {
    c.set('userRole', 'ADMIN' as any);
    c.set('customerSlug', 'test-customer');
    c.env = { FOURIA_API_URL: 'https://api.example.com', ...c.env } as any;
    await next();
  });
  app.route('/', myRoutes);
  return app;
}

function makeRequest(app: Hono<AppEnv>, path: string, opts = {}) {
  const env = createMockEnv(opts.env);
  return app.request(path, { method: opts.method || 'GET' }, env);
}

describe('my endpoint', () => {
  beforeEach(() => suppressConsole());

  it('returns expected structure', async () => {
    const res = await makeRequest(createTestApp(), '/');
    expect(res.status).toBe(200);
    const body = await res.json();
    expect(body).toHaveProperty('expectedKey');
  });

  it('forwards to FourIA API with correct headers', async () => {
    vi.stubGlobal(
      'fetch',
      vi.fn().mockResolvedValue(new Response(JSON.stringify({ result: 'ok' }), { status: 200 })),
    );
    const res = await makeRequest(createTestApp(), '/endpoint?param=value');
    expect(res.status).toBe(200);
    vi.unstubAllGlobals();
  });
});
Design Principles
  1. Tests deploy real infra, run against it, destroy it. Each cctr suite provisions a live Worker and container. No mocks — tests validate against the actual Cloudflare environment (R2 mounts, cold starts, Access auth, network latency).

  2. Sequential naming enforces order. Files run alphabetically. Use numeric prefixes (a_, b_, c_) to build dependencies (gateway must be up before agent tests run).

  3. %require for hard gates. If the gateway isn’t reachable, skip all downstream tests. Use sparingly — only for prerequisites that invalidate all later tests.

  4. fixture/ scripts provide a stable CLI surface. The debug-cli and curl-auth scripts abstract away URL/token management so test files focus on the scenario.

  5. Browser tests are skippable. Set E2E_SKIP_BROWSER_TESTS=1 to run only CLI tests. Every browser section checks this flag and exits 0 if set.

  6. Setup/teardown are tests too. _setup.txt and _teardown.txt are full cctr files with assertions. If setup fails, the suite doesn’t run. Teardown always runs.

  7. Vitest E2E supplements cctr. For API-level integration tests that don’t need a live container (cost pipelines, OTel ingestion, RBAC enforcement), use Vitest with createTestApp() and mocked fetch. These run fast, don’t require credentials, and validate the full middleware pipeline.


Deployment

bun run deploy:staging   # Deploy to staging
bun run deploy           # Deploy to production

Quick Debug Reference

# View live logs
npx wrangler tail

# Enter running container
docker exec -it <container_id> bash

# Inside container: check gateway
openclaw status
openclaw devices list --json
tail -f /tmp/openclaw-logs/*.log

# Check secrets
npx wrangler secret list

Normal Boot Warnings

During cold start (first request), these errors are normal:

[ERROR] [api/status] Port wait failed: Process did not become ready within 5000ms
[ERROR] Uncaught ProcessReadyTimeoutError: Waiting for: port 18789 (TCP)

The gateway takes up to 3 minutes to become ready. Retries are automatic. If errors persist beyond 3 minutes, check container logs.