# Lerma

Cloudflare-native platform for managing isolated application provisioning, cost analytics, and billing reporting. Lerma provisions and monitors tenant applications — each running in a fully isolated environment — through a Phoenix LiveView dashboard.

## Goal

Lerma is a self-serve platform that allows administrators to:

1. **Provision** — deploy applications using Workers for Platforms dispatch namespaces, with each application instance running in an isolated environment
2. **Monitor** — track health, usage metrics, and cost analytics across all deployed application instances
3. **Bill** — manage credit-based billing, invoicing, and subscription plans for tenants

Lerma serves on its own subdomain (`lerma.paso4.io`) on the `paso4.io` zone.

## Documentation map

| Need                             | Where                                                                 |
| -------------------------------- | --------------------------------------------------------------------- |
| Get started                      | [docs/GETTING-STARTED.md](/llms/platform/apps/lerma/getting-started/index.md) (hosted: `/docs/getting-started.html`) |
| First steps (local setup)        | [docs/FIRST-STEPS.md](/llms/platform/apps/lerma/first-steps/index.md) (hosted: `/docs/first-steps.html`) |
| API documentation (HexDocs)      | <https://lerma.paso4.io/docs> (see [API documentation](#api-documentation)) |
| Deployment runbook               | [DEPLOY.md](/llms/platform/apps/lerma/deploy/index.md)                                              |
| Product vision & pricing         | [docs/VISION.md](/llms/platform/reference/vision/index.md) (symlink to the monorepo vision)         |
| UI/design tokens                 | [docs/STYLE.md](/llms/platform/apps/lerma/style/index.md)                                      |
| Infrastructure as Code           | [Infrastructure as Code](#infrastructure-as-code) below               |
| Environment variables            | [Configuration](#configuration) below + [`docs/ENV-VARIABLES.md`](/llms/platform/reference/env-variables/index.md) |
| AI agent conventions             | [AGENTS.md](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/AGENTS.md)                                              |

## Architecture

Lerma is a **standalone Phoenix application** (Elixir 1.20 / OTP 29) running on Hetzner VMs behind Cloudflare:

- **Phoenix LiveView** — real-time dashboard UI, controllers, and JSON API under `/api`.
- **Ecto / PostgreSQL** — the multi-tenant data layer. Production uses a dedicated Supabase PostgreSQL database per environment; local development runs PostgreSQL in Docker Compose.
- **Cloudflare Access** — every browser request is gated by an Access JWT verified in `LermaWeb.Plugs.CfAccessAuth` (see [Local Auth Setup](#local-auth-setup-cloudflare-access)). Invitation links (`/invitations/*`) plus the static assets the page needs (`/assets/*`, `/images/*`, `/favicon.ico`) are the deliberate exception: they are served outside Access so invitees who are not FourIA operators can review and accept an invitation.
- **Cloudflare API** — Lerma provisions per-tenant Workers, AI Gateways, Access applications, R2 buckets, and service tokens at onboarding time using `LERMA_API_TOKEN`.

```
Browser
   │  HTTPS (proxied)
   ▼
Cloudflare DNS (paso4.io zone)
   │  ├── lerma.paso4.io      ──► Production Hetzner VM
   │  └── lerma-dev.paso4.io  ──► Staging Hetzner VM
   │                                  │
   │                                  ▼
   │                            Docker Compose ──► Phoenix release ──► Supabase (per-env)
   │
   └── lerma-host.paso4.io (DNS-only) ──► Production VM :22   (maintainer SSH)
       lerma-staging-host.paso4.io (DNS-only) ──► Staging VM :22
```

Tenant traffic is routed independently by `fouria-dispatch` on `*.fouria.io`; Lerma only hosts the dashboard on the `paso4.io` zone. The dashboard JSON API is served at `/api` on the dashboard domain — there is no separate API subdomain.

## Environments

| Environment | Domain               | Server       | Image tag     | Database             |
| ----------- | -------------------- | ------------ | ------------- | -------------------- |
| Production  | `lerma.paso4.io`     | Hetzner CX23 | `:latest`     | Supabase (prod)      |
| Staging     | `lerma-dev.paso4.io` | Hetzner CX23 | `:latest-pre` | Supabase (staging)   |
| Local       | `localhost:4000`     | Docker       | dev build     | `postgres:16-alpine` |

Each environment also publishes a **DNS-only** host record for maintainer SSH access (`lerma-host.paso4.io`, `lerma-staging-host.paso4.io`) — see [DEPLOY.md](/llms/platform/apps/lerma/deploy/index.md#maintainer-access).

## Instance types and sizing

The platform default is **`standard-1`** (4 GiB memory, 0.5 vCPU, 8 GB disk).
`basic` is selectable but **cannot host a tenant**: its 1 GiB is below the
OpenClaw gateway's ~1.1 GiB steady-state RSS (~2.6 GiB peak), so it OOM-loops on
first use. `lite` cannot even unpack the image (2 GB disk). The default used to
be `basic`; any instance provisioned without an explicit type before that fix
keep their stored type.

| Type | Memory | vCPU | Disk | Guidance |
| --- | --- | --- | --- | --- |
| `lite` | 0.25 GiB | 0.0625 | 2 GB | Not usable (image does not fit). |
| `basic` | 1 GiB | 0.25 | 4 GB | Not usable (below gateway RSS). |
| `standard-1` | 4 GiB | 0.5 | 8 GB | Default; practical floor for an agent-running tenant. |
| `standard-2` | 6 GiB | 1 | 12 GB | Cheap headroom (1.5× memory). |
| `standard-3` | 8 GiB | 2 | 16 GB | Deliberate headroom (2× memory, 4× core). |
| `standard-4` | 12 GiB | 4 | 20 GB | Heavy Builder workloads. |

Changing an instance type is a Cloudflare **rollout**, not a PATCH:
`Lerma.Cloudflare.Containers.change_instance_type/2` creates a `full_auto`
rollout with `target_configuration`, which **replaces the container** — plan one
cold start plus an R2 restore per tenant. The Worker's dev-gated memory watchdog
reads the container's cgroup limit and uses 75% of it (3 GiB on `standard-1`,
6 GiB on `standard-3`), so the threshold follows a resize without a Worker
deploy. See `docs/plans/sandbox-memory-usage.md` for the full analysis.

## Paso4-exclusive instances

Besides client tenants, Lerma can provision **Paso4-exclusive instances**: internal
platform instances owned by Paso4 itself. They:

- have **no associated client** and **no billing plan** (the billing gate is
  bypassed entirely),
- keep the same per-instance `dev`/`production` environment axis (routable at
  `paso4.fouria.io` / `paso4-dev.fouria.io`), and
- are reachable **only by `@paso4.io` identities** — the provisioner creates a
  domain-scoped Cloudflare Access group (`Paso4 Exclusive Users - paso4`) plus
  the shared root group, and the Dispatch Worker re-checks the email domain as
  defense-in-depth.

Provision them from **Dashboard → Instances → Provision Paso4 instance**. They
also appear in the unified instances list (marked with a `Paso4` badge) and are
managed from the same instance detail page as client instances. The operator
selects the environment, instance type, and an optional agent roster, and the
provisioning pipeline is rendered inline on the instances page. The roster is
deployed to the
container as `MOLTLAZY_AGENTS`; Paso4 instances additionally set
`MOLTLAZY_PASO4_AGENTS=true`, which makes the opt-in **`paso4-dev`** agent
eligible. `paso4-dev` picks up open issues from the Paso4 GitHub organizations
and the FourIA repository, using the bundled `github` skill (`gh` CLI)
authenticated by the platform `GITHUB_TOKEN` env var.

On the data model, Paso4 instances are `instances.scope = "paso4"` with a
nullable `client_id`; runs use `provisioning_runs.scope` and a dedicated
partial unique index (`one_active_run_per_paso4`). See
`Lerma.Instances.Owner` for the owner abstraction the provisioner works against.

Lerma can also authorize an operator's `ssh-ed25519` public key on **dev
Container applications** via the optional `DEV_INSTANCE_SSH_PUBLIC_KEY` secret.
SSH uses Wrangler and Cloudflare account access; it does not expose a public
port or pass the key into a tenant Worker/container environment. See the
[deployment guide](/llms/platform/apps/lerma/deploy/index.md#dev-container-ssh-access) for setup and connection
commands.

## Container-backed tenant deploys

Every tenant script is uploaded into a dispatch namespace and bound to a
per-tenant container application whose `durable_objects.namespace_id` is that
script's `Sandbox` Durable Object namespace. The upload metadata's `containers`
entry **must carry the full container configuration on every deploy**, not only
the class name:

The provisioner also deploys two non-secret bindings used by the Sandbox v1
migration gate: `WORKER_PLATFORM` (from Lerma's `cf_environment`, so only the
staging platform may migrate — contract C1) and `SANDBOX_V1_CUTOVER` (default
`false`, so unselected tenants stay on the legacy runtime byte-identically).

```elixir
%{
  class_name: "Sandbox",
  scheduling_policy: "default",
  image: image,
  instance_type: instance_type,
  max_instances: max_instances
}
```

A class-name-only entry is enough for the **first** upload (it creates and
container-enables the class). On an **update**, Cloudflare drops the class's
container enablement when the configuration is missing, so the next Durable
Object instantiation throws `Containers have not been enabled for this Durable
Object class` and the tenant can no longer start its sandbox. Because Durable
Objects keep running the code they were instantiated with, this stays latent
until the object is evicted — which is what happened to `tenant-tarjetero-dev`
after the `v2026.10.29` upgrade.

`Lerma.Cloudflare.DispatchNamespace` accepts either class names or full
configuration maps, and `Lerma.Cloudflare.Containers.deploy_script_and_container/1`
always sends the full entry. Regression tests:
`test/lerma/cloudflare/dispatch_namespace_test.exs` and
`test/lerma/cloudflare/containers_test.exs`.

### Image preparation and the rollout health gate

A rollout only *asks* Cloudflare to replace the instances; it does not mean the
new image can start. Cloudflare Containers build a **runtime artifact** (a rootfs
snapshot) per image **digest**, and until it exists an instance sits in
`starting` and is marked `failed` with an `ImagePullError` after a 10-minute pull
budget. The artifact is not retained indefinitely, and every CI build produces a
new digest, so a freshly promoted tag usually has no artifact yet.

Two guards prevent a promotion from pointing a tenant at an image it cannot run
(the 2026-11-03 staging outage: both tenants dropped to zero healthy instances
and the promotion still reported success in 40 s):

- **`Lerma.Cloudflare.ImagePreparation`** resolves the tag to its digest
  (short-lived pull-only registry credential) and polls
  `POST /containers/image-preparations` until the platform reports `ready`.
  `Containers.deploy_script_and_container/1` runs it **before** the script
  upload (so a failure cannot trigger the atomic-deploy rollback that deletes a
  live tenant's script), and the restart / instance-type rollouts run it before
  recreating instances. An image outside `registry.cloudflare.com` is skipped
  (the API supports the managed registry only), as is an unconfigured
  environment.
- **Live health, not the declared image.** `Containers.fetch_deployed_config/1`
  reports the image a rollout was *asked* to apply, which is set the moment the
  rollout is created. `Containers.fetch_application_health/1` reads
  `health.instances` (plus the error count and `active_rollout_id`) instead.
  `Instances.do_upgrade/3` waits for a finished rollout with a healthy instance
  before marking the instance `running`, `GET /api/provisioning/upgrade/:slug`
  exposes it as `container_health`, and
  `.github/scripts/promote-staging.sh` gates convergence on it and fails fast on
  a recorded provisioning error.

An instance already at the requested version but with no healthy instance is
re-deployed rather than skipped, so re-running a promotion repairs a tenant that
failed to start.

A promotion also **retries `error` instances**: `POST /api/provisioning/upgrade/:slug`
drives every running/sleeping instance *plus* any instance left in `error` by an
earlier rollout, as long as it does not already serve the requested **bundle**.
`Instances.at_requested_version?/2` verifies the live container image **and** the
deployed script content (the requested main module must be present) before
skipping, so a failed script upload — image at target, script still on the
previous bundle — is repaired by re-promoting. A genuinely broken container that
already serves the target bundle is still not looped.
Correspondingly, the validation gate (`.github/scripts/promote-validation.sh`)
refuses to report convergence while **any** instance is in `error`, whatever
version it declares — previously an `error` instance was filtered out of the
gate, so a promotion could report success while one tenant was broken at an old
version (2026-11-06).

### Choosing the sandbox runtime when deploying an instance

A new instance is deployed on one of two runtimes, chosen **once**, at
provisioning time:

| `sandbox_runtime` | Runtime                                                                  | Availability               |
| ----------------- | ------------------------------------------------------------------------ | -------------------------- |
| `legacy` (default) | the v0 `Sandbox` class (`.sqsh` archives) — what every existing tenant runs | every environment/platform |
| `v1`              | the Sandbox SDK 1.0 `SandboxV1` class (`DirectoryBackup` records)          | **`dev` environment on the staging platform only** |

`v1` is refused for a production-environment instance (schema invariant) and off
the staging platform (`cf_environment == "staging"`), for the same reason the
conversion track is staging-only: the v1 track has no production cutover. The
column is immutable — `update_instance/2` and the changeset reject a change, so a
runtime change is a migration, never an edit.

Surfaces:

```elixir
# API — per instance
POST /api/clients/:client_id/instances
  { "instance": { "environment": "dev", "sandbox_runtime": "v1" } }
  → 202 { run_id, sandbox_runtime: "v1", ... }
  → 422 { errors: { detail: "v1 sandbox runtime is available for dev instances only ..." } }
  → 422 { errors: { detail: "v1 sandbox runtime is available on the staging platform only" } }
```

```elixir
# Elixir — one runtime per environment (the UI sends the same shape)
Instances.create_instances(client.id, %{
  "environments" => ["dev"],
  "sandbox_runtimes" => %{"dev" => "v1"}   # a single "sandbox_runtime" still works
})
```

The client page's provision form renders a runtime select per environment row
(`sandbox_runtimes[env]`); the `v1` option is disabled for production with the
hint "v1 is dev-only — no production cutover", and the instance list shows the
runtime as a badge. The unified **Instances** dashboard (`/dashboard/instances`)
resolves the *effective* runtime with `Instances.runtime/1`: a tenant
**converted** from legacy to v1 keeps `sandbox_runtime: "legacy"` on its row,
so the stable pointer decides the badge (`v1`), and an unreadable pointer is
shown as `unknown` instead of claiming either runtime. Pointers are only read
on the staging platform (the v1 track has no production cutover) and when the
Cloudflare client is configured, so a production or local dashboard never
reaches the network.

#### What a fresh v1 instance does

There is **no conversion**. The provisioner writes a durable **runtime marker**
before the script deploy, and the Worker resolves a fresh v1 tenant from it:

```text
s3://fouria-backup-<hostname-label><platform-suffix>/sandbox-migration/<canonical-slug>/runtime.json
{ "version": 1, "tenant": "<canonical-slug>", "runtime": "v1", "mode": "fresh",
  "provisionedAt": "<ISO8601>" }
```

- **Canonical slug** is what the tenant Worker derives from its own env — the
  worker name with the dispatch prefix and `-dev` suffix stripped
  (`tenant-acme-dev` → `acme`), from `Lerma.Instances.Naming.canonical_tenant_slug/1`.
- **Create-only**, and written as the `:sandbox_runtime_marker` pipeline step,
  i.e. immediately **before** `:dispatch_deploy`. A tenant whose marker cannot be
  written is never deployed (the run rolls back and the step fails).
- The Worker also gets `SANDBOX_RUNTIME` (`"legacy"` / `"v1"`, re-asserted on
  every re-deploy) and `SANDBOX_V1_CUTOVER=true` for v1. A `v1` binding **without**
  a marker fails closed on the Worker (`pending`), which is why the marker is a
  hard step rather than a best-effort write.
- On boot the tenant **onboards inside the v1 container** (no restore, no legacy
  state), and the idle snapshot cadence takes the first `DirectoryBackup` +
  `stable.json` pointer — that record becomes the recovery point for
  park/restore/cold boot.

The first boot is also self-healing. A born-v1 first provision can leave the
`SandboxV1` container application with **zero instances**: the DO reports
running, every `containerFetch` times out and the gateway never boots, which
used to require deprovision + re-provision. The background health check for a
**fresh** v1 provision with **no committed recovery point** now recreates that
application (`Containers.recreate_v1_application/1`) — scoped to the app, so the
deployed script and its Durable Object state are untouched — and retries the
probe once. An upgrade never opts in, and a tenant with a recovery point is
never reconciled.

Refusals in the marker step:

| Outcome                        | Meaning                                                                                       |
| ------------------------------ | --------------------------------------------------------------------------------------------- |
| `{:ok, :existing}`             | a matching marker is already there — idempotent retry, and a rollback leaves it alone           |
| `{:error, :already_converted}` | `sandbox-migration/<slug>/intent.json` exists: the tenant is being/was **converted** from legacy |
| `{:error, {:runtime_mismatch, r}}` | the marker names another runtime; it is never overwritten                                  |

A rollback deletes the marker **only if that run created it**.

#### Inspecting a tenant's runtime

```bash
# What Lerma thinks the instance runs on
psql -c "select cf_worker_name, environment, sandbox_runtime from instances"

# The authoritative marker the Worker reads (canonical slug, no `tenant-`/`-dev`)
curl -s -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/r2/buckets/fouria-backup-acme-dev-staging/objects/sandbox-migration%2Facme%2Fruntime.json"
```

The column is the runtime **chosen at provisioning**, not necessarily what a
tenant serves: a converted tenant keeps `legacy` there while its stable pointer
makes the Worker serve v1. Resolve the effective runtime in Elixir when the
answer must include conversions:

```elixir
Instances.runtime(instance)   # :legacy | :v1 | :unknown
Instances.runtimes(instances) # %{instance_id => :legacy | :v1 | :unknown}
```

A `legacy` instance has no marker file — absence is the normal legacy state, and
the Worker keeps serving it on the legacy class. Never create or edit a marker by
hand; it selects the runtime of a live tenant.

#### A v1 instance is never a conversion candidate

`POST /api/provisioning/migrate/<slug>` (and its dry run) treats an instance with
`sandbox_runtime == "v1"` as **already v1**: the dry run reports
`eligible: false`, reason `sandbox_runtime_v1`, `already_v1: true`, and a real
request answers `200 {state: "committed", already: true, runtime: "v1"}` with no
control call at all. Converting a tenant that is already v1 would have nothing to
restore, so it is never attempted. The `already_converted` pointer check stays
for tenants that were **converted** from legacy.

### Migrating a legacy (v0) tenant to Sandbox v1

The `v1-sandbox` track can convert a tenant from the legacy `Sandbox` runtime
(v0, `.sqsh` archives) to `SandboxV1` (SDK 1.0, `DirectoryBackup` records). The
conversion runs as a **durable job** on the tenant Worker's `SandboxV1` Durable
Object (`POST /control` with `X-FourIA-Control: migrate`): the tenant's committed
legacy archive is streamed into a fresh v1 container, uploaded as a v1 record,
catalogued, and promoted to the stable pointer.

Lerma only decides **whether** an operator may ask for it and reports why not. It
never waits for a conversion: the control call **enqueues** and returns the
job's current state, and a separate poll reads it.

#### Eligibility

A run is refused, before any control request is spent, when:

| Refusal (`reasons`)        | Meaning                                                                                                                                                    |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `platform_not_staging`     | This Lerma is not the staging platform (`cf_environment != "staging"`), mirroring the Worker's `WORKER_PLATFORM === "staging"` authority (contract C1).        |
| `reserved_client`          | The reserved `client` slug never migrates (contract C2).                                                                                                    |
| `production_environment`   | Only a client's **non-production (`dev`) environment** instance may be the first conversion (contract S7: no existing customer is the first conversion).     |
| `no_active_instance`       | The tenant has no `running`/`sleeping` instance.                                                                                                            |
| `instance_not_found`       | An explicit `--instance` / `instance:` names a worker the client does not own.                                                                               |
| `no_legacy_recovery_point` | The tenant's backup bucket has no `stable-backup.json`: there is nothing to convert.                                                                        |
| `already_converted`        | The tenant already has a v1 stable pointer; the run would be a no-op (reported as `committed`, see below).                                                   |
| `sandbox_runtime_v1`       | The instance was **provisioned** on v1 (`sandbox_runtime: "v1"`); it is already v1, so there is nothing to convert (short-circuits to `committed`, no control call). |

Warnings never block: `legacy_recovery_point_unknown` (the R2 read could not be
verified — missing credential or transport error) and
`checkpoint_recommended_before_conversion` (the instance is running, and the
conversion restores the **last committed** recovery point, so checkpoint first —
`POST /control` with `X-FourIA-Control: checkpoint` — to avoid losing writes made
since the last snapshot).

One instance is converted per run: the `dev` instance. A client's production and
dev tenants are separate workers with separate backup buckets (they do not share
a data plane), and the control plane is addressed per tenant slug, so the
conversion always targets exactly one of them.

#### Trigger and poll

The conversion is asynchronous, in one direction only:

```text
POST /api/provisioning/migrate/<slug>   → 202 {state: "queued"}   enqueue (never waits)
                                        → 202 {state: "running"}  already in flight
GET  /api/provisioning/migrate/<slug>   → 200 {state: "absent"}   no job recorded (read-only)
                                        → 202 {state: "queued"|"running"}
                                        → 200 {state: "committed", record_id}
                                        → 502 {state: "failed", error}
```

The `POST` returns as soon as the durable job records the request, so the old
edge-timeout caveat no longer applies to the trigger: a `524`/gateway timeout can
no longer be ambiguous about whether a conversion started, and an operator does
not hold a request open for the whole archive round-trip.

**Only the `POST` starts a conversion.** The `GET` is a **read-only** status read
(`migration-status` control action): it never enqueues, never starts work and
never wakes the container, so it is always safe to poll — including on a tenant
that is still legacy (it answers `state: "absent"`) or whose job vanished. A
conversion is started by `POST /api/provisioning/migrate/:slug`, or by
`mix lerma.sandbox_v1.migrate`, which enqueues first unless `--dry-run` is given.

#### Mix task

```bash
# Preflight only: prints the table (instance, status, environment, eligible,
# recovery point, already-v1, reasons) and stops. No control call, no container wake.
mix lerma.sandbox_v1.migrate --client acme --dry-run

# Enqueue, then poll until terminal (default: 30 minutes, every 10 seconds)
mix lerma.sandbox_v1.migrate --client acme --yes

# Enqueue only; do not wait
mix lerma.sandbox_v1.migrate --client acme --yes --wait 0

# Explore the poll cadence / budget
mix lerma.sandbox_v1.migrate --client acme --yes --wait 900 --poll-interval 5

# Target an explicit instance
mix lerma.sandbox_v1.migrate --client acme --instance tenant-acme-dev --yes
```

The task **enqueues** and then polls the read-only `migration-status` action, so
a poll can never start a conversion. It prints each transition with the elapsed
time (`state=queued (elapsed 0s)` → `state=running (elapsed 10s)` →
`state=committed … record_id=… duration_ms=…`) and ends with a `state` / record
id / duration summary. `--wait` bounds the **poll loop only** (configurable with
`:migrate_wait_timeout_ms`, 30 minutes by default); the conversion keeps running
in the tenant if the loop gives up.

Exit codes: `0` dry-run eligible or conversion committed, `1` refused/invalid
invocation/declined confirmation/enqueue-only (`--wait 0`)/poll budget exhausted,
`2` conversion failed, the job vanished (`state=absent`) or the control plane was
unreachable.

#### API

The same preflight/trigger/poll logic is available to the provisioning API, using
the same authentication as `/api/provisioning/upgrade/:slug` (the global
`PROVISIONING_API_KEY` or an application key plus `X-FourIA-Application`):

```bash
# Dry run (no control call): 200 with the eligibility report
curl -X POST "https://lerma-dev.paso4.io/api/provisioning/migrate/acme?dry_run=1" \
  -H "Authorization: Bearer $PROVISIONING_API_KEY"

# Enqueue (202 when queued/running, 200 when already committed)
curl -X POST "https://lerma-dev.paso4.io/api/provisioning/migrate/acme" \
  -H "Authorization: Bearer $PROVISIONING_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"instance": "tenant-acme-dev"}'

# Poll until terminal (read-only: this never starts a conversion)
curl "https://lerma-dev.paso4.io/api/provisioning/migrate/acme?instance=tenant-acme-dev" \
  -H "Authorization: Bearer $PROVISIONING_API_KEY"
```

The dry run always answers `200`:

```json
{
  "ok": true,
  "dry_run": true,
  "eligible": true,
  "reasons": [],
  "warnings": ["checkpoint_recommended_before_conversion"],
  "already_converted": false,
  "v1_record_id": null,
  "target": {"cf_worker_name": "tenant-acme-dev", "environment": "dev"},
  "instances": [
    {
      "cf_worker_name": "tenant-acme-dev",
      "status": "running",
      "environment": "dev",
      "active": true,
      "eligible": true,
      "reasons": [],
      "legacy_recovery_point": true,
      "already_converted": false,
      "v1_record_id": null
    }
  ]
}
```

A real run answers:

| Status | Body                                                                                                  | Meaning                                                     |
| ------ | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| `202`  | `{ok: true, state: "queued"\|"running", record_id: null, duration_ms}`                                | enqueued / already in flight — poll the `GET`                |
| `200`  | `{ok: true, state: "committed", record_id, duration_ms}`                                              | the durable job committed (or the no-op short-circuit)       |
| `200`  | `{ok: true, state: "committed", already: true, record_id, duration_ms: 0}`                            | already converted; **no control call** was made              |
| `200`  | `{ok: true, state: "absent", record_id: null}`                                                        | `GET` only: no conversion job is recorded (read-only)        |
| `502`  | `{ok: false, state: "failed", error}` (or `state: "unknown"` for a control-plane failure)             | the job failed — retrying restarts it                        |
| `503`  | `{ok: false, state: "unavailable", error}`                                                            | no `SandboxV1` binding on the tenant Worker — retry later    |
| `403`  | `{ok: false, code: "MIGRATION_NOT_ELIGIBLE", reasons: ["platform_not_staging"]}`                      | not the staging platform                                     |
| `409`  | `{ok: false, code: "MIGRATION_NOT_ELIGIBLE", reasons: [...]}`                                         | any other preflight refusal                                  |

The `GET` accepts the same `?instance=` selector and answers the same shapes plus
`200 {state: "absent"}` and `503` (the `202`/`200`/`502` rows). Both include the
eligibility report (`target`, `instances`, `reasons`, `warnings`) so a poll
explains the tenant as well as its state. The control bearer and the tenant token
are never returned or logged.

#### State machine and idempotency

| State       | Meaning                                                                                     | HTTP  |
| ----------- | ------------------------------------------------------------------------------------------- | ----- |
| `absent`    | No conversion job is recorded for the tenant; the `GET` reports it as the truthful state.     | `200` |
| `queued`    | The durable job is recorded on the DO and will convert on its next turn.                     | `202` |
| `running`   | The DO is converting (streaming, extracting, uploading the v1 record).                       | `202` |
| `committed` | The v1 record is catalogued and the stable pointer names it; the gateway boots from it.      | `200` |
| `failed`    | The conversion failed (`error` carries the bounded reason); a retry restarts it.             | `502` |

The **trigger** (`migrate`) is idempotent: a committed tenant answers `committed`
without doing work, an in-flight one answers its `queued`/`running` state, and a
failed one may be retried. Because the job lives in the tenant's Durable Object,
a dropped control call is safe to repeat: every step re-reads its own recorded
state, and the caller never has to keep a request open. The **status read**
(`migration-status`) is a separate, read-only action: it never enqueues, so
polling a tenant — even one with no job, or a job that vanished, which answers
`absent` — can never start a conversion.

An **already-converted** target short-circuits before any control call: the v1
stable pointer (`sandbox-migration/<tenant>/stable.json`) is the decisive signal
(a conversion leaves the legacy archive in R2, so `legacy_recovery_point` alone
would report it as eligible), and both the dry run (`already_converted`,
`v1_record_id`) and a real run (`state: "committed", already: true`) report it.
When the pointer is unreadable (`nil`) the idempotent control call is still made:
the Worker is the authority.

#### One-way conversion

The conversion is **one-way**. Once the intent is written (before the first
conversion step), the tenant Worker refuses to route or serve it from legacy
again (`SANDBOX_V1_RUNTIME_PENDING` while pending, then v1); contract C5 makes
this a point of no return with **no automatic rollback**. Returning a converted
tenant to the legacy archive is an explicit operator action outside this surface
(and outside this repository's automation), and the legacy archive is left in
place precisely so that the recovery point remains portable.

Restore and rollback boundaries on a converted tenant:

- routine snapshots, manual `backup`/`checkpoint`, freeze and resume operate on
  **v1 records** (`stable.json` + `records/<id>.json`), not on the legacy
  catalog; the legacy `backups/<id>/data.sqsh` path is never used again;
- a v1 restore applies a catalogued record and (by default) advances the stable
  pointer, so a later cold boot restores it;
- the legacy recovery point stays readable (it is what the conversion restored
  from), but nothing in Lerma promotes or rewrites it after conversion.

#### What the operator verifies afterwards

1. `GET /api/health` on the tenant reports readiness (the v1 boot restores the
   committed record and launches the gateway; `softwareLog`/`jobLog` show
   progress).
2. The tenant Worker's runtime gate reports the converted runtime (a subsequent
   poll answers `committed`), and any other control action is refused with
   `SANDBOX_V1_CONTROL_UNSUPPORTED` until its v1 equivalent exists.
3. The conversion's `record_id` is the recovery point a cold boot restores
   (`stable.json` → `records/<id>.json`). The admin Backups page still reads the
   legacy `.sqsh` catalog, so it does not yet list v1 records — that read model
   is separate work; use the conversion state/record id (and the Worker's
   `/api/admin/storage/*` v1 routes) to inspect v1 persistence.

#### Reading the recovery point (C8, S6)

Every migrate response (dry run, trigger and poll) carries a `recovery_point`
object per instance, reported as **distinct fields** instead of one boolean —
the two C8 conditions must be observed separately:

```jsonc
{
  "runtime": "v1",                     // v1 | legacy | unknown
  "recovery_point": "complete",        // complete | pointer_absent | record_absent |
                                       // record_invalid | pointer_unknown | record_unknown
  "pointer": { "record_id": "…", "committed_at": "…" },  // or null / { "unreadable": true }
  "record":  { "state": "present", "source": "catalog", "dir": "/home/openclaw",
               "format": "tar+zstd/1", "size": 7948131, "issues": [] },
  "readiness": { "state": "ready", "source": "health_monitor",
                 "observed_at": "…" }  // ready | down | unknown
}
```

- **pointer** is the id named by `sandbox-migration/<tenant>/stable.json`.
- **record** validates the catalogued `records/<id>.json` (or the pre-catalog
  `conversion.json` fallback) against the Worker's own shape check.
- **readiness** is the latest **real** gateway observation: `HealthMonitor`
  records `up` only when `/api/health` reports `status: "running"`, so a
  sleeping tenant is never `ready`. An instance the monitor does not cover (a
  dev tenant) reports `unknown` — never a claim.
- `recovery_point: "complete"` requires **both** a valid pointer/record and is
  reported independently of readiness; pointer-without-readiness and
  readiness-without-pointer are both incomplete.

The same fields render on the instance page under **Sandbox v1 recovery point**
(only for a v1 tenant), so an operator can tell "the pointer moved" from "the
gateway is actually serving".

#### Upgrading a v1 tenant

A v1 tenant's container is the `…-sandbox-v1-<platform>` application (a
`durable_object` instance application), not the legacy `…-sandbox-<platform>`
one. The legacy application is unused by the v1 runtime and its rollout state can
wedge independently, so `Instances.container_health/1` reads the **v1**
application for a v1 instance: the legacy application's `rollout_active` can no
longer time out a promotion that the tenant is actually serving. The promotion
gate's `serving_version` falls back to the instance's recorded
`deployed_version` for a v1 tenant (the v1 application declares no image tag —
the DO reads a digest-pinned named image from the script); the authoritative
"the deployed script is the requested bundle" check remains
`Instances.at_requested_version?/2`.

## Quick Start

```bash
# Phoenix dependencies + local database
docker compose up -d db
mix setup

# Run the Phoenix app
mix phx.server
```

Visit `http://localhost:4000` for the dashboard.

## API documentation

The HexDocs-format API documentation is generated with [ExDoc](https://hexdocs.pm/ex_doc) from module docs and the README/DEPLOY/STYLE `extras`, and served by the Phoenix endpoint at:

- Production: <https://lerma.paso4.io/docs>
- Staging: <https://lerma-dev.paso4.io/docs>

`/docs` is served from `priv/static/docs` (see `docs/0` in [mix.exs](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/mix.exs)) and stays behind Cloudflare Access like the rest of the dashboard. Generate it locally with:

```bash
mix docs          # writes priv/static/docs, open priv/static/docs/index.html
```

`mix docs` is also run during the Docker image build, so the hosted docs always match the deployed release. Add new guides by listing them under `extras` in [mix.exs](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/mix.exs).

## Local Auth Setup (Cloudflare Access)

The dashboard authenticates via Cloudflare Access JWT. In production, CF Access sits in front of the app. For local development, use a cloudflared tunnel to get real JWTs.

### One-Time Setup

Create a Terraform workspace and apply the local dev configuration:

```bash
cd apps/lerma/iac/tf

# Ensure .env has TF_VAR_deploy_environment=local
# Then create and select the "local" workspace
tofu workspace new local
tofu workspace select local

# Source the .env (which has CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, etc.)
# and apply — this only creates CF Access resources, no Hetzner/Supabase
(set +a && source ../../.env && set -a && tofu apply)
```

`CLOUDFLARE_API_TOKEN` remains Lerma's administrative provisioning credential. Set `CLOUDFLARE_AI_GATEWAY_API_KEY` separately to a real AI-scoped Cloudflare credential; the tenant provisioner copies that credential to User Workers and derives their `CLOUDFLARE_API_TOKEN` REST binding without exposing Lerma's token.

Then copy the output values into your `.env`:

```bash
tofu output -raw local_access_aud            # → CF_ACCESS_AUD
tofu output -raw local_service_token_id      # → CF_ACCESS_SERVICE_TOKEN_ID
tofu output -raw local_service_token_secret  # → CF_ACCESS_SERVICE_TOKEN_SECRET
tofu output -raw local_tunnel_token          # → CLOUDFLARED_TUNNEL_TOKEN
```

### Daily Development

Start the tunnel in one terminal:

```bash
CLOUDFLARED_TUNNEL_TOKEN=<token> bin/local-tunnel.sh
```

Start the app in another:

```bash
mix phx.server
```

Access via `https://local.lerma.paso4.io` — Cloudflare Access will prompt for OAuth login.

### Escape Hatch

To run without the tunnel (auth bypassed):

```bash
SKIP_AUTH=true mix phx.server
```

## Infrastructure as Code

All Lerma infrastructure lives under [`apps/lerma/iac/`](./iac/) and has two layers:

```
apps/lerma/iac/
├── tf/                       # OpenTofu — infrastructure provisioning
│   ├── providers.tf          # hcloud + cloudflare + supabase providers
│   ├── main.tf               # Hetzner server, firewall, SSH keys
│   ├── supabase.tf           # Supabase PostgreSQL project (prevent_destroy)
│   ├── access.tf             # Cloudflare Access app + Paso4/Partner root groups
│   ├── variables.tf          # Input variables (see terraform.tfvars.example)
│   ├── outputs.tf            # Hand-off values for Ansible + GitHub secrets
│   ├── tests/*.tftest.hcl    # `tofu test` suites (mocked providers)
│   ├── justfile              # Task aliases (plan/apply/output/destroy per env)
│   └── .aliases              # Shell-function fallback for the justfile
└── ansible/                  # Ansible — server configuration + deployment
    ├── inventory.yml         # Production inventory (SERVER_IP)
    ├── inventory.staging.yml # Staging inventory (STAGING_SERVER_IP)
    ├── playbooks/site.yml    # Master playbook
    ├── roles/                # common, firewall, docker, app-config, app
    ├── group_vars/all.yml    # Shared playbook variables
    ├── templates/            # docker-compose.yml.j2
    └── justfile              # `just deploy-staging` / `just deploy-production`
```

### OpenTofu (`iac/tf`)

OpenTofu provisions everything *outside* the application:

| File           | Manages                                                                 |
| -------------- | ----------------------------------------------------------------------- |
| `providers.tf` | `hcloud`, `cloudflare`, and `supabase` provider configuration           |
| `main.tf`      | Hetzner server + firewall + uploaded SSH keys (env-suffixed per workspace) |
| `supabase.tf`  | One Supabase PostgreSQL project per environment (`prevent_destroy`)     |
| `access.tf`    | Cloudflare Access applications for the dashboard + the Paso4/Partner Root groups, and the `/invitations/*` and static-asset (`/assets/*`, `/images/*`, `/favicon.ico`) bypass apps that keep invitation links public |
| `variables.tf` | `hcloud_token`, `lerma_api_token`, `cloudflare_account_id/zone_id`, SSH keys, `deploy_environment`, server sizing, Supabase org/region, root email lists |
| `outputs.tf`   | `server_ipv4_address`, `database_url`, `host_fqdn`, `ansible_inventory`, `dashboard_access_aud`, root group IDs, local tunnel/token values |

**Workspaces** isolate state per environment (they are *not* git-branched):

| Workspace    | Environment | Env file              | Notes                                            |
| ------------ | ----------- | --------------------- | ------------------------------------------------ |
| `default`    | Production  | `../../.env`          | The default workspace **is** production           |
| `staging`    | Staging     | `../../.env.staging`  | Set `TF_VAR_deploy_environment=staging` implicitly via workspace |
| `local`      | Local dev   | `../../.env`          | `TF_VAR_deploy_environment=local`; only CF Access + tunnel resources, no Hetzner/Supabase |

Use the `justfile` (recommended) or source `.aliases`:

```bash
cd apps/lerma/iac/tf

just init                  # tofu init
just test                  # tofu test (all mocked suites)
just plan-production       # tofu plan   (default workspace, sources ../../.env)
just apply-staging         # tofu apply  (staging workspace, sources ../../.env.staging)
just output-production     # inspect outputs
just refresh-production    # tofu apply -refresh-only (reconcile after state drift)
just fmt-production        # tofu fmt
just help                  # list all targets
```

`tofu test` runs the suites in `tests/` with `mock_provider`, so it needs no real credentials — run it before every IaC change.

### Ansible (`iac/ansible`)

Ansible configures each provisioned VM and deploys the app container. Roles run in order:

| Role         | Purpose                                                        |
| ------------ | -------------------------------------------------------------- |
| `common`     | Update apt, install base packages (curl, gnupg, ufw)           |
| `firewall`   | Open ports 22, 80, 443; enable UFW with deny-incoming default  |
| `docker`     | Install Docker CE + Compose plugin, start the service          |
| `app-config` | Copy the compose file, write `.env` with secrets               |
| `app`        | Log in to GHCR, pull the image, restart the container, health-check |

Inventories are environment-specific and read the server IP from an env var. Compose files: `docker-compose.prod.yml` (`:latest`, container `lerma`), `docker-compose.staging.yml` / `docker-compose.pre.yml` (`:latest-pre`, container `lerma-staging`).

```bash
cd apps/lerma/iac/ansible
cp .env.example .env         # fill in secrets
just deploy-staging          # deploy to lerma-dev.paso4.io
just deploy-production       # deploy to lerma.paso4.io
```

See [DEPLOY.md](/llms/platform/apps/lerma/deploy/index.md) for the complete OpenTofu + Ansible runbook, required GitHub secrets, and the full manual `ansible-playbook` invocations.

### Relationship between the two layers

```
OpenTofu (iac/tf)  ──provisions──►  Hetzner VM + firewall + Supabase + Access
        │
        └── outputs (server IP, database_url, AUDs) ──► GitHub secrets
                                                              │
Ansible (iac/ansible) ──consumes secrets + outputs──► configures VM, deploys container
```

## Deployment

```bash
# Staging (push to develop triggers CD after Lerma CI passes)
git push origin develop

# Production (publish a GitHub release; the pipeline promotes :latest-pre → :latest)
gh release create <tag>
```

See [DEPLOY.md](/llms/platform/apps/lerma/deploy/index.md) for the full deployment guide, including migrations, container lifecycle, DNS, and monitoring.

## Configuration

Lerma reads its runtime configuration from environment variables in [`config/runtime.exs`](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/config/runtime.exs) and its deployment credentials from `apps/lerma/.env` (IaC) and `apps/lerma/iac/ansible/.env` (server deploy). Never commit these files.

| Variable           | Required | Purpose                                                             |
| ------------------ | -------- | ------------------------------------------------------------------- |
| `DATABASE_URL`     | yes      | PostgreSQL (Supabase in prod; `postgres:16-alpine` locally)          |
| `SECRET_KEY_BASE`  | yes      | Phoenix cookie/session signing key (`mix phx.gen.secret`)            |
| `PHX_SERVER`       | prod     | Starts the Phoenix HTTP server                                       |
| `PHX_HOST`         | prod     | Hostname used for URL generation (`lerma.paso4.io`)                  |
| `PHX_CHECK_ORIGIN` | prod     | Allowed LiveView WebSocket origins                                   |
| `DNS_CLUSTER_QUERY`| optional | Cluster discovery query                                              |
| `SKIP_AUTH`        | dev only | Bypasses Cloudflare Access JWT verification                          |

IaC and deploy credentials (see [`iac/ansible/.env.example`](https://github.com/0xCAB0/fouria/blob/develop/apps/lerma/iac/ansible/.env.example) and the secrets manifest) include `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_ZONE_ID`, `CLOUDFLARE_API_TOKEN` / `LERMA_API_TOKEN`, `CLOUDFLARE_AI_GATEWAY_API_KEY`, `CF_ACCESS_TEAM_DOMAIN`, `CF_ACCESS_AUD`, `CF_ACCESS_PLATFORM_AUD`, `GHCR_OWNER`/`GHCR_TOKEN`, `HCLOUD_TOKEN`, and `SUPABASE_*`.

The complete cross-project env var reference is in [`docs/ENV-VARIABLES.md`](/llms/platform/reference/env-variables/index.md), and the machine-readable secrets manifest is [`docs/schemas/secrets-manifest.schema.json`](/assets/docs/schemas/secrets-manifest.schema.json).

### Health monitor service token

The health monitor authenticates its tenant `/api/health` probes with the
**`FourIA Health Monitor`** Cloudflare Access service token created by the fouria
IaC (`apps/fouria/iac/access.tf`):

| Variable                               | Source                                                                                     |
| -------------------------------------- | ------------------------------------------------------------------------------------------ |
| `HEALTH_MONITOR_ACCESS_CLIENT_ID`      | `tofu -chdir=apps/fouria/iac output -raw health_monitor_service_token_client_id`            |
| `HEALTH_MONITOR_ACCESS_CLIENT_SECRET`  | `tofu -chdir=apps/fouria/iac output -raw health_monitor_service_token_client_secret`        |
| `CF_ACCESS_HEALTH_MONITOR_TOKEN_ID`    | `tofu -chdir=apps/fouria/iac output -raw health_monitor_service_token_id` (optional pin)    |

When the client id/secret are empty (e.g. local dev) the probes are sent
unauthenticated. That is only safe when the health endpoints are not behind
Access. The dashboard CD (`.github/workflows/deploy-dashboard.yml`) passes all
three values to Ansible (as `health_monitor_access_client_id` /
`health_monitor_access_client_secret` / `cf_access_health_monitor_token_id`),
which writes them into the root-only Compose env file — so a deploy never ships
a dashboard whose tenant probes are unauthenticated.

An exact-host per-tenant Access app shadows the wildcard
`*.fouria.io/api/health` app, so the provisioner
(`Lerma.Cloudflare.AccessPolicy`) also creates a per-tenant health application at
`{slug}.fouria.io/api/health` that carries no role requirement — it allows any
authenticated user and this service token. `CF_ACCESS_HEALTH_MONITOR_TOKEN_ID`
pins the token id; when unset the token is resolved by name at provisioning time.

### Dispatch control bearer

Lerma drives the Dispatch Worker's control plane (`POST /_control/:slug/{freeze,unfreeze,backup,checkpoint,migrate}`,
`GET /_invalidate/:slug`) with a shared bearer. The Worker has **no hardcoded
fallback**: it compares digests in constant time against its own
`INTERNAL_DISPATCH_INVALIDATE_TOKEN` secret and fails closed (`403`).

| Environment | GitHub secret                                | Target                                      |
| ----------- | -------------------------------------------- | ------------------------------------------- |
| Staging     | `STAGING_INTERNAL_DISPATCH_INVALIDATE_TOKEN` | `fouria-dispatch-staging` + staging Lerma    |
| Production  | `INTERNAL_DISPATCH_INVALIDATE_TOKEN`         | `fouria-dispatch` + production Lerma         |

Generate each with `openssl rand -hex 32`; never reuse a value across
environments. Deploy the Dispatch Worker first, then Lerma — a mismatch answers
`403` and only disables control-plane calls (cache invalidation, freeze/unfreeze,
and the pre-replacement checkpoint) until both sides hold the same value.

**`up` means the gateway is running.** A probe is `up` only when the snapshot
reports `status: "running"`. A `200` `status: "sleeping"` (an idle container whose
gateway is not running) records a down transition and marks the instance
`sleeping` — alive, but not running — and does **not** recover a client left in
`error`. Health is otherwise **observability-only**: a failed probe (a `503`
`stopped`, a transport error, or an Access challenge) records a down transition
for the uptime widget but **never** changes the instance or client status —
tenant containers are loaded lazily, so a probe miss is not an outage, and
marking the tenant `error` would take it out of service. Only a genuine `running`
probe marks the instance `running` and recovers a client left in `error` by a
provisioning failure.

### Billing and task queues

Every provisioned tenant receives `OTEL_TRACES_ENABLED` (`true` for a
production/staging instance, `false` for a client's non-billable dev instance),
the shared `BILLING_QUEUE` producer (`fouria-billing` /
`fouria-billing-staging`), and a per-tenant `TASKS` queue
(`fouria-tasks-<slug>[-dev]`) with the tenant script registered as its consumer.
The billing queue is created/adopted at provisioning; the task queue is
per-tenant because Cloudflare allows one consumer per queue. The E2E suite uses
`wrangler.jsonc` and therefore does not exercise these Lerma-provisioned
bindings.

### Cost read-back (no tenant token)

Billable rows are written by the tenant Worker through its `ANALYTICS` binding —
**emission needs no credential and no token is deployed to tenants**. Lerma
reconciles credits from Analytics Engine with its own Cloudflare credentials, so
billing is complete without any per-tenant secret.

`ANALYTICS_ENGINE_API_TOKEN` is therefore not provisioned by the provisioner. It
remains a supported **per-instance** secret (instance page) for the optional
dashboard "Usage & Billing" read-back: with it the worker queries the AE SQL API
for its own rows; without it `getCostSummary()` returns an all-zero summary with
`available: false` and the widget says usage is aggregated by the platform. There
is no `CLOUDFLARE_API_TOKEN` fallback — that alias lacks Account Analytics:Read.

### Per-client OpenCode keys

`OPENCODE_API_KEY` is the platform's OpenCode **management** credential (a
service-account key with `Models: Request` plus `This account's API keys:
Write`). At provisioning, `Lerma.Opencode` uses it to mint a restricted
(`Models: Request`) key **per client**, stores the key on the `clients` row, and
binds it as `OPENCODE_API_KEY` on that client's `dev` instances only. A ROOT
operator can generate/rotate, reveal, revoke, or paste a key manually on the
client page (`/dashboard/clients/:id`). When the management credential is unset,
provisioning skips generation and the dev agents fall back to the primary model.

The key-management endpoint is configurable (`OPENCODE_KEYS_URL`, default
`https://opencode.ai/api/v1/keys`) because OpenCode documents key permissions but
not the management HTTP surface in its public Console API reference.

### Private transports (OTLP and promotion)

Neither feature exposes an internet route on the tenant.

**OTLP** is ingested through the **sandbox bridge**: the container exports to
`http://otel.internal/v1/{traces,metrics}`, the Sandbox SDK resolves that virtual
host inside the Worker (`TenantSandbox.outboundByHost`), and the Worker still
verifies `OTEL_SHARED_SECRET`. There is no `/api/otel` route and no path-scoped
Access app.

**Promotion** uses R2, with both buckets kept separate:

- the dev instance writes the bundle + `promotion/pending.json` to its
  **own** bucket;
- the production instance receives a read-only `PROMOTION_SOURCE_BUCKET` binding
  to its sibling dev bucket (`promotion_source_bindings/2` here and in
  `TenantProvisioner`), provisioned only for `environment: "production"`;
- production applies the pending release on boot (after the restore barrier,
  before the launch) or via `POST /api/admin/releases/apply`. The apply holds the
  shared persistence lease, is skipped while parked or while a maintenance
  window/restore is in flight, and is idempotent.

There is no `PROMOTION_PUSH_TOKEN`, no `/api/admin/releases/import` route and no
machine Access bypass: the R2 binding is the authority, and a tenant can only
read its own sibling's prefix. Lerma ensures the sibling dev bucket exists before
binding it, so a production-first provisioning order still deploys.

## Cloudflare API Token Permissions

This project requires a Cloudflare API token (`LERMA_API_TOKEN`) with these minimum permissions:

| Permission                        | Reason                                                  |
| --------------------------------- | ------------------------------------------------------- |
| `Workers Scripts:Edit`            | Deploy and manage tenant worker scripts                 |
| `DNS:Edit`                        | Manage DNS records for tenant domains                   |
| `Access:Apps & Policies:Edit`     | Create and manage Cloudflare Access applications        |
| `Access:Service Tokens:Edit`      | Create service tokens for CI/CD auth                    |
| `R2:Edit`                         | Create/manage per-tenant backup buckets and read client documentation objects |
| `AI Gateway:Edit`                 | Configure AI Gateway for unified billing                |
| `Email Sending:Edit`              | Send invitation emails via Cloudflare Email Service     |
| `Analytics:Read`                  | Query Analytics Engine for cost metrics                 |
| `Account API Tokens:Edit`         | Mint account-owned R2 S3 credentials (see below)        |

## AI Gateway catalog and authentication

**Cloudflare is the source of truth for which AI Gateways exist.** The dashboard
(`/dashboard/ai-gateways`) and provisioning selection read the account's
gateways directly from the Cloudflare API. Lerma's `ai_gateways` table is only an
optional *policy override* store — `max_instances`, `enabled`, `environment`,
and display metadata — merged on top of the Cloudflare catalog. A gateway that
exists only in Cloudflare is still listed and selectable, and is always bound by
its `gateway_id` (never the local primary key).

Before a worker is deployed, provisioned, or upgraded, lerma ensures the bound
gateway exists (create-only) so a redeploy never points at a deleted gateway.
An upgrade likewise ensures the tenant's R2 backup bucket exists (create-only,
idempotent): Cloudflare rejects the worker upload (error 10085) when the
`BACKUP_BUCKET` binding points at a missing bucket. The bucket is never deleted
on an upgrade rollback — it holds tenant backups.

### Zero-loss upgrades (parked checkpoint + resume)

A container replacement starts from a fresh disk, so the replacement restores the
tenant from the checkpoint taken just before the rollout. `@upgrade_pipeline` makes
that lossless:

- `:checkpoint` calls `Cloudflare.Containers.checkpoint(instance, park: true)`,
  which uses the Worker's `checkpoint-park` control action: the tenant is drained,
  the stopped filesystem is snapshotted, and the tenant is **left parked** —
  mutating admin/debug requests and all container exec are refused until resume.
  The step is **fail-closed**: a checkpoint that cannot be taken aborts the
  upgrade before the container is replaced, because the replacement would
  otherwise discard writes made after the last archive. Only a sleeping instance
  (`skipped: not-running`) is exempt — it has nothing live to lose.
- `:dispatch_deploy` uploads the script and rolls the image.
- `:resume` (immediately after) calls `Cloudflare.Containers.resume/1`, which
  closes the park and lets the replacement container restore the checkpoint and
  boot the gateway. Both error paths also call `resume/1` best-effort, so a failed
  upgrade never leaves a tenant parked; the 15-minute window TTL is the crash
  backstop.

The direct (bundle-only) upgrade path `Instances.upgrade_instance/3` — used by
the staging validation promotion API — parks with the same contract: the
checkpoint is a hard gate before the worker bundle is re-uploaded, a non-parked
(`parked: false`) or failed response aborts the upgrade, and the park is released
after the rollout settles and on every error path. A sleeping instance
(`skipped: not-running`) is never woken, so no resume is sent.

`Cloudflare.Containers.checkpoint/1` remains available without `park: true` for
callers that only need a snapshot and do not replace the container.

The park also drains live agent sessions before it stops the gateway: it switches
them to `permissionMode: 'read-only'` and waits (bounded) for their running turn
to finish, so a turn is not interrupted mid-write. The previous modes are stored
outside the container and restored when the replacement container's gateway is up
(`resume` waits for it; the post-restore path applies them otherwise). New
sessions are unaffected — the switch is per existing session, never a default.
Telegram users who message during the park get a direct notice from the ingress
(no AI, no gateway) and the update is acked instead of queued.

AI Gateways are created **authenticated by default** (`authentication: true`) so
Cloudflare unified billing can attribute traffic to the gateway. An operator may
opt into an unauthenticated **BYOK** gateway (tenants supply their own upstream
key) only by explicitly setting `authentication: false` on the policy entry; the
Cloudflare gateway is then created/updated to match that intent.

To repair gateways that drifted to unauthenticated (e.g. ones created before
authenticated gateways were enforced), run reconciliation from an IEx/`mix run`
session:

```elixir
Lerma.AiGateways.reconcile_all_authentication()
```

Reconciliation only **enables** authentication on policy entries recorded as
`authentication: true`; explicit BYOK entries are left untouched.

### Infrastructure health checks

The instance page (`/dashboard/instances/:id`) exposes an **Infrastructure
Health** panel that probes the tenant's Cloudflare resources (AI Gateway, User
Worker, R2 backup bucket, Access app/group). The AI Gateway row checks the
gateway the instance is **actually bound to** (`instances.ai_gateway_id`, e.g. a
shared `fouria-shared-N` gateway) and only falls back to the derived dedicated id
(`fouria-<slug><env>`) for unbound/legacy instances.

The panel probes run when the page mounts. The **Re-check** button in the panel
re-runs them on demand after an operator repairs a resource (for example
reassigning a broken AI Gateway), and asks `HealthMonitor` to run a probe cycle
immediately (instead of waiting up to `HEALTH_MONITOR_INTERVAL_MS`) so a
recovered tenant's instance status reconciles in the same action.

### Staging Lerma healthcheck tunnel

The staging VM runs a remotely managed Cloudflare Tunnel sidecar for
`https://lerma-health-dev.paso4.io/health`. Its ingress matches only `^/health$`
and returns 404 for every other path; `/health` returns only `ok`. The Lerma
dashboard and its API remain on `lerma-dev.paso4.io` behind their existing Access
policies. The connector token is the GitHub Actions secret
`LERMA_STAGING_HEALTHCHECK_TUNNEL_TOKEN`, written to the root-only Docker Compose
environment file by Ansible; it is not passed into the Phoenix application.

Creating/updating the tunnel in the staging OpenTofu workspace requires a
separate `TF_VAR_healthcheck_tunnel_api_token` with only Account **Cloudflare
Tunnel:Edit** and `paso4.io` Zone **DNS:Edit**. Do not broaden the shared
`LERMA_API_TOKEN`. After applying, retrieve the tunnel run token through the
Cloudflare API and store it as the GitHub secret above; do not put the run token
in Terraform outputs or source control.

This tunnel adds a health-only route and does not replace the staging dashboard's
existing proxied A record or change the VM firewall. Production has no
healthcheck tunnel. The Cloudflare provider cannot delete a remote Tunnel config;
rollback by removing the health DNS record and stopping the sidecar, then reset
the tunnel config to the 404 catch-all or delete the tunnel through Cloudflare.

#### Observability and Tools sections

The instance page groups its signals under a dedicated **Observability** section
and its actions under a dedicated **Tools** section.

`Lerma.Instances.HealthEvents` backs two widgets: `LermaWeb.UptimeWidget`
(production only) surfaces the current up/down state and uptime over the last
24 h / 7 d instead of only logging the probe result, and
`LermaWeb.HealthEventsWidget` renders the last 24 h as a compact **time graph**
(up / down / unknown segments) for any instance. Hovering a segment animates it
and reveals that window's log (probe OK, or the HTTP status and error message) —
there is no separate event list, and the header **Inspect** action opens the
Infrastructure health panel. `LermaWeb.WorkerTelemetryWidget` sits next to them
with uptime/failure ratios derived from the pushed Cloudflare traces. The
connected widgets refresh every 15–30 seconds, so background transitions appear
without reloading the instance page.

The two uptime numbers answer different questions: **Uptime** is time-weighted
(seconds in `up` ÷ seconds in a *known* state; time before the first recorded
check is unknown and excluded, and the widget surfaces the observed window), while
**Worker telemetry** is request-weighted (successful root spans ÷ observed
invocations, head-sampled). A sleeping tenant with no traffic, for example,
lowers probe availability but produces no invocations — never "0 % uptime".

**Job activity** closes the operational side of the same section.
`Lerma.Instances.JobEvents` persists one row per instance job run in
`instance_job_events` — `backup_create` (the snapshot taken before a freeze or a
verified checkpoint/park), `restore` (the durable boot's restore-from-recovery-point
step) and `gateway_start` (the boot's launch step, until `/api/health` answers
ready). `LermaWeb.JobActivityWidget` renders the last 24 h as one lane per job;
each run is a milestone icon placed at its start time, and hovering it reveals the
status, duration, trigger, timestamp and (on failure) the error. `WorkerTelemetryWidget`
gains a **Job performance** tab over the same 24 h / 7 d / 30 d windows: run count,
success rate, average/p95/max duration, the last run and a bar graph of the
selected job's duration evolution.

Writes are instance-scoped and best-effort: a history row never fails the job it
describes. `backup_create` is recorded from the control-plane response of the
freeze/checkpoint; `restore` / `gateway_start` are folded from the boot and
restore-progress fields of the non-passive health poll lerma already runs during
provisioning (and a background poll after a manual start or restart, which also
warms the gateway immediately). A cold start triggered by tenant traffic is not
observed — lerma would have to poll (and wake) the instance to see it — and
scheduled snapshots taken by the Dispatch cron are not observed either, so
neither appears in this history. The reads never probe a tenant, so a sleeping
instance is never woken to draw the timeline or the graph.

The **Billable actions** section embeds the client's
`LermaWeb.UsageObservationWidget` — credit spend per action (stacked daily
chart), action log and stats. It is rendered for **production** instances
(billable by definition) and for any instance whose client has an active
billing plan (`Lerma.BillingPlans.client_has_plan?/1`); Paso4 instances have no
client and are never billable, so they omit it.

Billable usage is **production-only by contract**: dev/staging instances are
never billable and emit no billable-action rows (`resolveInstanceDimensions` →
`billable: false`), so credits and the stacked chart cannot contain a dev
series. Their infrastructure + AI cost is tracked separately as **development
expense** (USD, `Lerma.DevelopmentExpense`). The Usage widget therefore carries
a *Production only* badge and a two-tone environment legend (production =
billable credits, dev = non-billable development expense) so the chart is never
misread as covering both environments.

The **Tools** live in a **sticky Swiss-knife toolbar** at the top of the page,
so every operational action stays one click away while the page scrolls.
Blades: deploy, start/stop, environment, deployments, AI Gateway, gateway token,
debug routes and deprovision. Deploy and start/stop keep a visible label; the
remaining blades are icons with a tooltip and a state dot (token stored, debug
routes enabled, instance frozen). **Unfold** opens the full tool cards —
descriptions, badges and the primary buttons for each action — in a scrollable
drawer, and folding hides a revealed gateway token again. The bar condenses to
icons on mobile (the strip stays inside the viewport) and the drawer is capped
at 55vh so it never covers the page. Infrastructure health lives on the Health
events card, next to the graph it explains.

History lives in `instance_health_events` — **one row per state change**
(up → down / down → up), never one per probe, so the table stays small. Both the
async provisioning health check (`source: "provision"`) and the periodic
`HealthMonitor` (`source: "monitor"`) write transitions through
`Lerma.Instances.HealthEvents`; **non-production instances are ignored**. Uptime
over a window is reconstructed from the transition stream (each state holds
until the next event).

### Client documentation widget

The client profile (`ClientLive.Show`) embeds `LermaWeb.ClientDocsWidget`: a
source tree on the left and a live preview on the right.

Sources come from `Lerma.ClientDocs` and are read **directly from R2 through the
Cloudflare REST API** (the same `LERMA_API_TOKEN`, which needs R2 object read)
— the dashboard never wakes a tenant container and never handles R2 S3
credentials:

| Source          | Bucket / prefix                                                            |
| --------------- | -------------------------------------------------------------------------- |
| Proposal        | global leads bucket (default `paso4-leads`), `<client_slug>/`               |
| Client docs     | the instance's `BACKUP_BUCKET`, `customers/<client_slug>/docs/`             |
| Client docs dev | the dev instance's `BACKUP_BUCKET`, `customers/<client_slug>/docs/`         |

The widget is **lazy**: opening the client profile renders a *Load documentation*
placeholder and performs **no R2 request**. Clicking it lists the sources and
loads the default document; live polling starts only after that first load, so
the client page never waits for documentation.

The tree shows folders, file-type icons and thumbnails for image files.
By default it lists only the client documents operators asked for — **Word
(`.doc`/`.docx`), PDF, XLSX/XLS, CSV and PowerPoint (`.ppt`/`.pptx`)** — and
hides other artifacts (Markdown, images, SVG, JSON, diagrams, …). The tree
footer exposes **Show all files** (with the hidden count) for the full folder.

Markdown is rendered by `Lerma.ClientDocs.Markdown` (a small, XSS-safe,
dependency-free renderer: headings, lists, tables, code, links, images;
source HTML is always escaped). PDFs use the browser's native viewer, CSV/text
render in a scrollable code panel, images render inline, and Word/PowerPoint/
spreadsheet binaries offer an open/download link.

The documentation widget has a fixed height (`42rem`); the tree and preview fill
the remaining space and scroll independently (`overscroll-contain`), so long
proposals never stretch the client page. On narrow screens the two panels split
the same fixed area vertically. **Live** re-lists the active source and
re-fetches the selected document every 20 s; **Paused** stops the polling. The preview's Open action and inline assets use
`/dashboard/clients/:id/docs/raw/:source?path=…`, which is behind the dashboard
Access session. Active content (HTML/SVG) is forced to a download with a sandbox
CSP, paths are validated against traversal, and raw responses are `no-store` so
uploads appear immediately.

## R2 Backup Credentials

The fouria User Worker signs R2 presigned URLs for Sandbox SDK backups using
`R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` (plus `CLOUDFLARE_ACCOUNT_ID` and
`BACKUP_BUCKET_NAME`). Cloudflare derives these S3 credentials from a regular API
token: Access Key ID = token `id`, Secret Access Key = SHA-256 of the token
`value`. Lerma can mint one via API:

```bash
CLOUDFLARE_API_TOKEN=<parent> CLOUDFLARE_ACCOUNT_ID=<acct> \
  mix lerma.r2_token --all-buckets --write ../../apps/fouria/.dev.vars
```

`--all-buckets` grants R2 object read/write across every bucket (covers the
dynamic per-tenant `fouria-backup-*` buckets). Without it, pass the exact bucket
names instead. The task also supports `--name` and `--write <dotenv-path>`.

> The **parent** token must carry the **Account → Account API Tokens → Edit**
> permission (the account-scoped `LERMA_API_TOKEN` qualifies). The task creates
> an **account-owned** token — the right model for a platform R2 backup
> credential, since it survives individual user changes. Rotate R2 credentials
> on demand via this task.

### The credential is mandatory

Since the backup credential is required by the Sandbox SDK on every tenant
worker, lerma now treats it as **mandatory**: a provisioning run that cannot
resolve one fails and rolls back (it no longer logs a warning and deploys a
tenant that silently cannot back up).

The credential is selected by strategy (`config :lerma, :r2_token_strategy`,
env `R2_TOKEN_STRATEGY`, default `:auto`):

| Strategy                   | Behaviour                                                                                                                                                        |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `:auto` (default)          | Uses the known account token limit (`:r2_token_max_per_account`) to choose: `<= 250` → `:shared`, `> 250` or unknown → `:per_instance`.                            |
| `:per_instance`            | Mints a dedicated R2 S3 token scoped to the tenant's backup bucket at instance creation (`Lerma.Provisioning.R2Credentials`). Persisted on the instance record.    |
| `:shared`                  | Reuses the lerma-level `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` pair from the environment.                                                                     |

- Cloudflare exposes no API for the account's maximum token count
  (`GET /accounts/{account_id}/tokens` lists existing tokens only), so the limit
  defaults to **unknown**, which selects `:per_instance`. Pin a known limit with
  `R2_TOKEN_MAX_PER_ACCOUNT` (or `config :lerma, :r2_token_max_per_account`).
- Generated per-instance credentials are stored on the instance
  (`r2_access_key_id`, `r2_secret_access_key`, `r2_credential_source`) so
  redeploys/upgrades reuse the same credential; a token generated for a failed
  brand-new provisioning is deleted during rollback.
- Existing instances provisioned before this change have no stored credential;
  the next upgrade/redeploy backfills one.
- When the effective strategy is `:shared`, both `R2_ACCESS_KEY_ID` and
  `R2_SECRET_ACCESS_KEY` must be set in the lerma environment — otherwise
  provisioning fails with `:missing_shared_r2_credentials`.

## Worker deployments and Durable Object migrations

A tenant Worker's Durable Object classes (`Sandbox`, `ActivityScheduler`,
`TelegramIngress`, plus `BackupWatch` while the change-watch flag is on) are
declared through a Cloudflare **migration**, which Cloudflare accepts only once
per class — error `10074` (`Cannot apply new-sqlite-class migration to class '…'
that is already depended on by existing Durable Objects`) means the class already
has migration history.

The class set is **not** derived from the script's bindings, because the two can
diverge:

- an upgrade redeploy that binds fewer classes leaves the rest migrated but
  unbound (the upgrade path now re-declares the full DO binding set, carrying the
  existing `namespace_id` per class);
- a rollback/cleanup that deletes the script leaves its classes (and Durable
  Objects) behind. Provisioning cleanup and a failed atomic deploy therefore tear
  the classes down first (`delete_durable_object_namespaces/2`), except on an
  upgrade rollback, where the Durable Objects hold the live tenant's state.

Both upgrade entry points — the direct `Instances.upgrade_instance/3` and the
dashboard/API `TenantProvisioner.upgrade_instance/3` pipeline — pass
`preserve_do_namespaces: true`, so a failed container binding never deletes the
script that routes a live tenant. A fresh provision passes `false` and still
rolls the just-uploaded script (and its DO classes) back.

As a backstop (`Lerma.Cloudflare.DispatchNamespace`), an upload rejected with
`10074` names the offending class and is **retried without it**, so a tenant that
already regressed into this state recovers without manual Cloudflare
intervention. The retry logs `DO classes already migrated for <ns>/<script>`.

## Testing

```bash
# Phoenix tests
mix test

# ExDoc generation
mix docs

# Full repository pre-commit (run from the repository root)
cd ../..
mix precommit
```

## Project Structure

```
lerma/
├── lib/
│   ├── lerma/                  # Business logic (Ecto/PostgreSQL)
│   └── lerma_web/              # Web layer (LiveView, controllers, plugs)
├── assets/                     # Phoenix frontend (JS/CSS)
├── config/                     # Phoenix config (runtime.exs, prod.exs)
├── priv/
│   ├── repo/migrations/        # Ecto migrations
│   └── static/docs/            # ExDoc output served at /docs (generated)
├── test/                       # Phoenix tests
├── iac/                        # Infrastructure as Code
│   ├── tf/                     # OpenTofu (Hetzner, DNS, Supabase, Access)
│   └── ansible/                # Ansible (server config, deployment)
├── bin/local-tunnel.sh         # cloudflared tunnel for local CF Access auth
├── Dockerfile                  # Elixir release image (also builds /docs)
├── docker-compose.yml          # Local dev (PostgreSQL only)
├── docker-compose.prod.yml     # Production compose
├── docker-compose.pre.yml      # Staging compose
└── mix.exs                     # Elixir project + ExDoc definition
```
