F FourIA staging 2026.10.29 GitHub ↗

Lerma

On this page

0.17.0

Minor Changes

  • feat(observability): record per-instance job runs and surface their performance

    Every instance job lerma initiates or observes is now persisted as an instance_job_events row — backup_create (freeze snapshot and verified checkpoint/park), restore (the durable boot’s restore-from-recovery-point step) and gateway_start (the boot’s launch step, until /api/health answers ready). The instance Observability section gains a 24-hour Job activity milestone timeline (one lane per job, hoverable icon with status, duration, trigger, timestamp and error) and the Worker telemetry widget gains a Job performance tab: run counts, success rate, average/p95/max duration, the last run and a bar graph of the selected job’s duration evolution across the 24 h / 7 d / 30 d window.

    Recording is instance-scoped and best-effort: it never fails the job, never probes a sleeping container on its own (it folds the boot/restore fields of the non-passive health poll lerma already runs, plus a background poll after a manual start/restart), and stores only metadata. (7c58e75)

Patch Changes

  • Backups: make a retained snapshot restorable for as long as it is retained, stop archiving regenerable paths, and gate scheduled snapshots on change by default.

    • Archive TTL is retention-neutral. The Sandbox SDK refuses to restore an archive past createdAt + ttl (BackupExpiredError), so the previous 7-day value silently turned a retained recovery point into a billed-but-unrestorable object — cleanupOldBackups prunes by count, so an idle or frozen tenant could keep BACKUP_MAX_VERSIONS archives that no boot could apply, and the stable pointer was then dropped, failing the boot closed with TenantStateUnavailableError. New snapshots use SNAPSHOT_ARCHIVE_TTL_SECONDS (10 years) and the durable repair step heals existing tenants via extendArchiveTtl (reported as ttlExtended/ttlFailed from POST /api/admin/storage/reconcile).
    • SNAPSHOT_EXCLUDES drops .npm, *.log, *.tmp, .cache and workspace/*node_modules* before mksquashfs, behind BACKUP_ARCHIVE_EXCLUDES (off by default). Excluding paths makes an archive-restored tenant differ from its pre-eviction tree (and from a Tier 1 workspace snapshot), so the recovery tier stays a complete image unless an operator opts in per instance. Plugin extensions/<id>/node_modules, SQLite databases and session transcripts are never excluded, because a restore has no reinstall step of its own.
    • BACKUP_CHANGE_DETECTION defaults to on. A snapshot re-uploads the whole archive and re-streams it through the Worker for F4E1 encryption; measured on tenant-tarjetero-dev, two snapshots 37 minutes apart differed by 0.08 % (348 KB of 428 MB) while both were uploaded, encrypted and stored in full. Only an explicit false opts out, and a scan failure still reports changed, so the gate can only cost an extra snapshot — never a missed one.
    • A restore forgets the change baseline. Restored files keep their archived (older) mtimes, so the -newer scan would have called the replaced tree unchanged and skipped the first post-restore snapshot.
    • Idle eviction still stops inline. Change detection on by default means a quiet tenant takes no scheduled snapshot, so its newest archive ages past the idle window and the snapshot-age test alone would re-archive the whole tenant on every sleep cycle. onActivityExpired() now also accepts unchanged from the same change scan as proof that an existing archive covers the current state. (4e93f63)
  • fix(lerma): prepare container images before rollouts and gate promotion on live health (dae0487)

  • fix(provisioning): private OTLP and promotion transports (sandbox bridge + R2 pull)

    OTLP ingest moves to the otel.internal sandbox bridge and promotion to an R2 stage/pull between the sibling environment buckets, so neither feature exposes a public tenant route. Lerma binds PROMOTION_SOURCE_BUCKET on production instances and ensures the sibling dev bucket exists before the deploy. (4e171bb)

  • fix(billing): stop deploying the tenant AE read-back token; emission uses the ANALYTICS binding (673b1cd)

  • Default new instances to standard-1 instead of basic.

    basic is 1 GiB of memory, below the OpenClaw gateway’s ~1.1 GiB steady-state RSS (and ~2.6 GiB peak), so an instance provisioned without an explicit type OOM-looped on first use. Lerma.Instances.InstanceTypes.default/1, the Instance schema default, and Containers.deploy_script_and_container/1’s fallback now return standard-1 (4 GiB / 0.5 vCPU / 8 GB), the practical floor for an agent-running tenant. Existing instances keep their stored type; a resize is a Cloudflare full_auto rollout (one cold start + R2 restore per tenant). (2902de0)

  • Zero-loss container rollouts: park the tenant before an image rollout and resume it after the replacement.

    A container replacement always starts from a fresh disk, so the replacement restores the tenant from the checkpoint taken immediately before the rollout. Anything written after that archive was produced was lost. The upgrade now parks the tenant instead of restarting it:

    • Cloudflare.Containers.checkpoint/2 gained park: true, which uses the tenant Worker’s new checkpoint-park control action: the same verified snapshot, but the tenant is left drained and write-blocked (mutating admin and debug requests answer 409 TENANT_PARKED, and container exec is refused) until the control plane resumes it.
    • Cloudflare.Containers.resume/1 closes the park and lets the replacement container restore the checkpoint and boot the gateway. A no-op resume reports resumed: false instead of failing.
    • @upgrade_pipeline’s :checkpoint step now parks and is fail-closed: a checkpoint that cannot be taken aborts the upgrade before the container is replaced (a sleeping instance is exempt — it has nothing live to lose). A :resume step runs immediately after :dispatch_deploy, and both error paths resume best-effort so a failed upgrade cannot leave a tenant parked.

    The worker-only Instances.upgrade_instance/3 path keeps the non-parking checkpoint: it does not change the container image, so there is nothing to lose.

    Zero-loss upgrades depend on this ordering, not on the Sandbox 1.0 migration: Cloudflare’s container snapshots are image-locked (not portable across an image change) and require the durable_object scheduling policy, so they cannot carry state across an upgrade. (9f6124c)

  • fix(lerma): keep container-backed DO classes enabled across redeploys (994a087)

0.16.0

Minor Changes

  • feat: add a client documentation widget with a source tree and live preview (3054026)

  • Redesign the instance page around dedicated Observability and Tools sections.

    The instance page is now a full-width, single-page layout instead of a narrow two-column grid that squeezed every widget into a third of the screen:

    • Overview — instance details in a full-width definition grid (worker name, type, environment, deployed version, region, created, AI Gateway).
    • Observability — uptime (production) and worker telemetry side by side at roughly twice their previous width, plus a new Health events timeline (LermaWeb.HealthEventsWidget) listing the state transitions recorded by the background probes. The timeline works for every instance, not only production. The two uptime figures are now labelled for what they measure — probe availability (time-weighted) vs invocation success (request-weighted) — and the Uptime widget surfaces the window it actually observed, so a short probe history no longer reads as a 24 h outage.
    • Health events — a compact 24 h time graph (up / down / unknown) instead of an event list: hovering a segment animates it and reveals that window’s log (probe OK, or the HTTP status and error message). The header Inspect action opens the Infrastructure health panel, which moved here from the Tools grid.
    • Billable actions — a new section embedding the client’s Usage widget (credit spend per action, action log, stats) for production instances and for any instance whose client has an active billing plan. Paso4 instances are never billable and omit it. The widget is labelled Production only with a two-tone environment legend: dev/staging are non-billable by contract and their cost lives in development expense, so the credits chart has no dev series.
    • Tools — a sticky Swiss-knife toolbar at the top of the page (Deploy, Start/Stop, Environment, Deployments, AI Gateway, Gateway token, Debug routes, Deprovision). Primary actions keep a label, the other blades are icons with a tooltip and state dot; Unfold opens the detailed tool cards in a scrollable drawer, and folding clears a revealed gateway token. The bar condenses to icons on mobile. The icon-only action row in the details header and the cramped “Ops” workbench/management cards are gone.

    Visually the page drops the nested bordered cards in favour of calmer rounded-2xl surfaces, quieter section headings with one-line descriptions, and soft secondary buttons.

    LermaWeb.UptimeWidget keeps only the aggregate uptime ratios; the recent changes list moved to the new widget. No backend changes. (35f07eb)

  • feat: price semantic billing actions (telegram_query, queue_reevaluation, signal_analysis, email_draft, review_analysis) with success-only consumption, business-unit action_key idempotency, and trace-level precedence (85f8701)

Patch Changes

  • fix: lazy-load client documentation on demand (5fbcabe)
  • fix(lerma): retry dispatch uploads that re-declare migrated DO classes (#516) (d3c2f60)

0.15.2

Patch Changes

  • Decompress gzip OTLP/HTTP JSON batches before parsing so Cloudflare Observability destinations are accepted instead of retrying on a 400. (7bd792b)

Historical changes (imported from apps/fouria/AGENTS.md)

These entries predate the generated release sections above. They were maintained by hand in apps/fouria/AGENTS.md and are kept here verbatim for traceability. New entries belong in a changeset (see docs/VERSIONING.md); the sections above are generated by Changesets (scripts/changeset-changelog.mjs).

  • Container-backed tenant redeploys keep the Sandbox DO container-enabled. A tenant script that is updated after a working deploy could later fail with Sandbox.jsrpc → "Containers have not been enabled for this Durable Object class" (observed on tenant-tarjetero-dev): the DO keeps the code it was instantiated with, so the missing enablement stayed latent until the object was evicted (here, after the v2026.10.29 upgrade and the Dev instance’s idle stop). Lerma.Cloudflare.DispatchNamespace now accepts either class names or full container entries in containers, and Lerma.Cloudflare.Containers.deploy_script_and_container/1 always sends the full entry (class_name, scheduling_policy, image, instance_type, max_instances) — a class-name-only entry is enough for the first upload but not for an update. Regression tests: apps/lerma/test/lerma/cloudflare/dispatch_namespace_test.exs (metadata carries the full config) and containers_test.exs (deploy passes it through); docs apps/lerma/README.md (“Container-backed tenant deploys”).
  • Lerma deploys are idempotent around Cloudflare DO migration conflicts (#516). A tenant Worker’s Durable Object class set was derived from the script’s bindings, but Cloudflare validates new_sqlite_classes against the script’s migration history, so the two can diverge: an upgrade redeploy that binds fewer classes (or a rollback that deletes the script while leaving its classes migrated) made every subsequent deploy re-declare TelegramIngress and fail with 400 code 10074 — deterministically, so provisioning could not recover on its own. Lerma.Cloudflare.DispatchNamespace.upload_script/6 now retries the upload without any class Cloudflare reports as already migrated (10074 names it), switching an implicit full first-deploy set to an explicit new_sqlite_classes list; each retry removes at least one class, so it terminates, and the retry is logged. Lerma.Instances.instance_bindings/2 now re-declares the same DO set as provisioning (Sandbox, ACTIVITY, TELEGRAM_INGRESS, plus BACKUP_WATCH while the flag is on) so an upgrade cannot detach a migrated class from its binding. Provisioning cleanup and a failed atomic deploy tear the classes down first via delete_durable_object_namespaces/2 (never on an upgrade rollback, whose Durable Objects hold the live tenant’s state: preserve_do_namespaces). Tests: apps/lerma/test/lerma/cloudflare/dispatch_namespace_test.exs (migrated-but-unbound re-deploy, resurrected first deploy, unrelated error not retried), containers_test.exs, tenant_provisioner_pipeline_test.exs; docs apps/lerma/README.md.
  • Cloudflare Observability destination ingestion (issue #337). Account-level Observability → Destinations now push Workers Logs and Workers Traces to Lerma over OTLP/HTTP JSON, giving event-driven worker diagnostics and uptime statistics that scale with traffic instead of tenant count. (1) Config: apps/fouria/wrangler.jsonc (env.staging → lerma-worker-logs/lerma-worker-traces, env.production → lerma-worker-logs-prod/lerma-worker-traces-prod) and Lerma.Cloudflare.Observability.worker_metadata/0 reference the environment’s destination names, with per-signal head sampling (10% at rollout) and persist: false when a destination is configured (Lerma is the destination of record). The Lerma CD passes the matching names to staging and production, and the token to both; it also reconciles the account-level destination pairs (mix lerma.observability.destinations --warn-only) before each deploy so a fresh environment self-heals once the Cloudflare credential has Workers Observability Write. New WORKER_OBSERVABILITY_LOGS/TRACES_DESTINATIONS, *_SAMPLING_RATE and *_PERSIST env vars. (2) Ingest: new POST /api/ingest/otel/v1/logs and /v1/traces (bearer OTEL_INGEST_TOKEN, constant-time compare, fail closed; OTLP/HTTP JSON only — protobuf → 415 + hint; 202 {accepted, skipped} so Cloudflare never retry-loops). Lerma.Observability.Otlp is a pure allowlist parser: it keeps script/dispatch namespace/outcome/handler/status/exception class names/cpu+wall/trace+span+invocation ids/timestamps, drops log text, stacks, URLs/query strings, user agents, request/response bodies and DB query text, and keeps only invocation root spans. Rows land in observability_events with a unique dedupe_key + ON CONFLICT DO NOTHING, so replayed batches (at-least-once delivery) add no rows. Cloudflare gzips each OTLP/HTTP JSON batch while keeping Content-Type: application/json; LermaWeb.Plugs.OtlpBodyReader decompresses it before Plug.Parsers (otherwise the destination retries on a 400 forever). (3) Uptime: GET /api/analytics/v1/uptime/:script_name?window=24h|7d|30d|<n>h|<n>d computes uptime/failure ratios from pushed root spans only; no events observed (including under sampling) → null ratios + status: "unknown", never downtime. The instance page gained a Worker telemetry widget (window selector + uptime/failure/invocation/CPU/wall), and Lerma.Observability.Pruner sweeps 90-day retention daily. HealthMonitor and billing flows are untouched. Tests: test/lerma/observability/otlp_test.exs, telemetry_test.exs, test/lerma/cloudflare/observability_test.exs, test/lerma_web/controllers/api/otlp_ingest_controller_test.exs, observability_controller_test.exs, test/lerma_web/components/worker_telemetry_widget_test.exs, test/lerma_web/live/instance_live_test.exs; docs docs/CLOUD-ARCHITECTURE.md, docs/ENV-VARIABLES.md, docs/TESTING.md, apps/fouria/public/articles/PRIVACY.md, apps/fouria/docs/WORKER-DEVELOPMENT.md, SECRETS.example.yaml, docs/schemas/secrets-manifest.schema.json, apps/lerma/.env.example.
  • Health endpoints no longer require the ROOT role at the edge. A tenant ADMIN opening /_admin was 302’d to the Cloudflare Access login for GET /api/health and GET /api/health/ws (the dashboard’s health stream), because the per-tenant FourIA Health - <slug> Access app on <slug>.fouria.io/api/health allowed only the Paso4 Root Users group plus lerma’s health-monitor service token. The health app (and the platform-wide *.fouria.io/api/health / *.dev.fouria.io/api/health IaC apps) now carries no role requirement: an allow policy on everyone (any authenticated user) plus the existing non_identity service token, so the dashboard and lerma’s headless probe both work. AccessPolicy.health_policies/1 drops the root-group policy (the host app keeps it); access.tf public_health matches. Tests: test/lerma/cloudflare/access_policy_test.exs, apps/fouria/iac/tests/access_app.tftest.hcl; docs apps/lerma/README.md, apps/fouria/iac/README.md, apps/fouria/docs/WORKER-DEVELOPMENT.md, docs/ENV-VARIABLES.md. Existing tenants were reconciled in place via the Cloudflare API (aud preserved).
  • Paso4-exclusive instances + agent: lerma can now provision internal scope: "paso4" instances (nullable client_id, no billing plan, @paso4.io-only Access), and moltlazy ships an opt-in paso4-dev agent (PASO4_AGENTS, gated by MOLTLAZY_PASO4_AGENTS=true and named in MOLTLAZY_AGENTS) that picks up GitHub issues via the bundled github skill (gh CLI, installed in the image) authenticated by GITHUB_TOKEN. The Dispatch Worker resolves paso4 tenants from instances (no clients row) and rejects non-@paso4.io emails. See apps/lerma/README.md and docs/ENV-VARIABLES.md.