Lerma
On this page
0.17.0
Minor Changes
-
feat(observability): record per-instance job runs and surface their performance
Every instance job lerma initiates or observes is now persisted as an
instance_job_eventsrow —backup_create(freeze snapshot and verified checkpoint/park),restore(the durable boot’s restore-from-recovery-point step) andgateway_start(the boot’s launch step, until/api/healthanswers ready). The instance Observability section gains a 24-hour Job activity milestone timeline (one lane per job, hoverable icon with status, duration, trigger, timestamp and error) and the Worker telemetry widget gains a Job performance tab: run counts, success rate, average/p95/max duration, the last run and a bar graph of the selected job’s duration evolution across the 24 h / 7 d / 30 d window.Recording is instance-scoped and best-effort: it never fails the job, never probes a sleeping container on its own (it folds the boot/restore fields of the non-passive health poll lerma already runs, plus a background poll after a manual start/restart), and stores only metadata. (7c58e75)
Patch Changes
-
Backups: make a retained snapshot restorable for as long as it is retained, stop archiving regenerable paths, and gate scheduled snapshots on change by default.
- Archive TTL is retention-neutral. The Sandbox SDK refuses to restore an archive past
createdAt + ttl(BackupExpiredError), so the previous 7-day value silently turned a retained recovery point into a billed-but-unrestorable object —cleanupOldBackupsprunes by count, so an idle or frozen tenant could keepBACKUP_MAX_VERSIONSarchives that no boot could apply, and the stable pointer was then dropped, failing the boot closed withTenantStateUnavailableError. New snapshots useSNAPSHOT_ARCHIVE_TTL_SECONDS(10 years) and the durablerepairstep heals existing tenants viaextendArchiveTtl(reported asttlExtended/ttlFailedfromPOST /api/admin/storage/reconcile). SNAPSHOT_EXCLUDESdrops.npm,*.log,*.tmp,.cacheandworkspace/*node_modules*beforemksquashfs, behindBACKUP_ARCHIVE_EXCLUDES(off by default). Excluding paths makes an archive-restored tenant differ from its pre-eviction tree (and from a Tier 1 workspace snapshot), so the recovery tier stays a complete image unless an operator opts in per instance. Pluginextensions/<id>/node_modules, SQLite databases and session transcripts are never excluded, because a restore has no reinstall step of its own.BACKUP_CHANGE_DETECTIONdefaults to on. A snapshot re-uploads the whole archive and re-streams it through the Worker for F4E1 encryption; measured ontenant-tarjetero-dev, two snapshots 37 minutes apart differed by 0.08 % (348 KB of 428 MB) while both were uploaded, encrypted and stored in full. Only an explicitfalseopts out, and a scan failure still reportschanged, so the gate can only cost an extra snapshot — never a missed one.- A restore forgets the change baseline. Restored files keep their archived (older) mtimes, so the
-newerscan would have called the replaced tree unchanged and skipped the first post-restore snapshot. - Idle eviction still stops inline. Change detection on by default means a quiet tenant takes no scheduled snapshot, so its newest archive ages past the idle window and the snapshot-age test alone would re-archive the whole tenant on every sleep cycle.
onActivityExpired()now also acceptsunchangedfrom the same change scan as proof that an existing archive covers the current state. (4e93f63)
- Archive TTL is retention-neutral. The Sandbox SDK refuses to restore an archive past
-
fix(lerma): prepare container images before rollouts and gate promotion on live health (dae0487)
-
fix(provisioning): private OTLP and promotion transports (sandbox bridge + R2 pull)
OTLP ingest moves to the
otel.internalsandbox bridge and promotion to an R2 stage/pull between the sibling environment buckets, so neither feature exposes a public tenant route. Lerma bindsPROMOTION_SOURCE_BUCKETon production instances and ensures the sibling dev bucket exists before the deploy. (4e171bb) -
fix(billing): stop deploying the tenant AE read-back token; emission uses the ANALYTICS binding (673b1cd)
-
Default new instances to
standard-1instead ofbasic.basicis 1 GiB of memory, below the OpenClaw gateway’s ~1.1 GiB steady-state RSS (and ~2.6 GiB peak), so an instance provisioned without an explicit type OOM-looped on first use.Lerma.Instances.InstanceTypes.default/1, theInstanceschema default, andContainers.deploy_script_and_container/1’s fallback now returnstandard-1(4 GiB / 0.5 vCPU / 8 GB), the practical floor for an agent-running tenant. Existing instances keep their stored type; a resize is a Cloudflarefull_autorollout (one cold start + R2 restore per tenant). (2902de0) -
Zero-loss container rollouts: park the tenant before an image rollout and resume it after the replacement.
A container replacement always starts from a fresh disk, so the replacement restores the tenant from the checkpoint taken immediately before the rollout. Anything written after that archive was produced was lost. The upgrade now parks the tenant instead of restarting it:
Cloudflare.Containers.checkpoint/2gainedpark: true, which uses the tenant Worker’s newcheckpoint-parkcontrol action: the same verified snapshot, but the tenant is left drained and write-blocked (mutating admin and debug requests answer409 TENANT_PARKED, and container exec is refused) until the control plane resumes it.Cloudflare.Containers.resume/1closes the park and lets the replacement container restore the checkpoint and boot the gateway. A no-op resume reportsresumed: falseinstead of failing.@upgrade_pipeline’s:checkpointstep now parks and is fail-closed: a checkpoint that cannot be taken aborts the upgrade before the container is replaced (a sleeping instance is exempt — it has nothing live to lose). A:resumestep runs immediately after:dispatch_deploy, and both error paths resume best-effort so a failed upgrade cannot leave a tenant parked.
The worker-only
Instances.upgrade_instance/3path keeps the non-parking checkpoint: it does not change the container image, so there is nothing to lose.Zero-loss upgrades depend on this ordering, not on the Sandbox 1.0 migration: Cloudflare’s container snapshots are image-locked (not portable across an image change) and require the
durable_objectscheduling policy, so they cannot carry state across an upgrade. (9f6124c) -
fix(lerma): keep container-backed DO classes enabled across redeploys (994a087)
0.16.0
Minor Changes
-
feat: add a client documentation widget with a source tree and live preview (3054026)
-
Redesign the instance page around dedicated Observability and Tools sections.
The instance page is now a full-width, single-page layout instead of a narrow two-column grid that squeezed every widget into a third of the screen:
- Overview — instance details in a full-width definition grid (worker name, type, environment, deployed version, region, created, AI Gateway).
- Observability — uptime (production) and worker telemetry side by side at
roughly twice their previous width, plus a new Health events timeline
(
LermaWeb.HealthEventsWidget) listing the state transitions recorded by the background probes. The timeline works for every instance, not only production. The two uptime figures are now labelled for what they measure — probe availability (time-weighted) vs invocation success (request-weighted) — and the Uptime widget surfaces the window it actually observed, so a short probe history no longer reads as a 24 h outage. - Health events — a compact 24 h time graph (up / down / unknown) instead of an event list: hovering a segment animates it and reveals that window’s log (probe OK, or the HTTP status and error message). The header Inspect action opens the Infrastructure health panel, which moved here from the Tools grid.
- Billable actions — a new section embedding the client’s Usage widget (credit spend per action, action log, stats) for production instances and for any instance whose client has an active billing plan. Paso4 instances are never billable and omit it. The widget is labelled Production only with a two-tone environment legend: dev/staging are non-billable by contract and their cost lives in development expense, so the credits chart has no dev series.
- Tools — a sticky Swiss-knife toolbar at the top of the page (Deploy, Start/Stop, Environment, Deployments, AI Gateway, Gateway token, Debug routes, Deprovision). Primary actions keep a label, the other blades are icons with a tooltip and state dot; Unfold opens the detailed tool cards in a scrollable drawer, and folding clears a revealed gateway token. The bar condenses to icons on mobile. The icon-only action row in the details header and the cramped “Ops” workbench/management cards are gone.
Visually the page drops the nested bordered cards in favour of calmer
rounded-2xlsurfaces, quieter section headings with one-line descriptions, and soft secondary buttons.LermaWeb.UptimeWidgetkeeps only the aggregate uptime ratios; the recent changes list moved to the new widget. No backend changes. (35f07eb) -
feat: price semantic billing actions (telegram_query, queue_reevaluation, signal_analysis, email_draft, review_analysis) with success-only consumption, business-unit action_key idempotency, and trace-level precedence (85f8701)
Patch Changes
- fix: lazy-load client documentation on demand (5fbcabe)
- fix(lerma): retry dispatch uploads that re-declare migrated DO classes (#516) (d3c2f60)
0.15.2
Patch Changes
- Decompress gzip OTLP/HTTP JSON batches before parsing so Cloudflare
Observability destinations are accepted instead of retrying on a
400. (7bd792b)
Historical changes (imported from apps/fouria/AGENTS.md)
These entries predate the generated release sections above. They were maintained
by hand in apps/fouria/AGENTS.md and are kept here verbatim for traceability.
New entries belong in a changeset (see docs/VERSIONING.md); the sections above
are generated by Changesets (scripts/changeset-changelog.mjs).
- Container-backed tenant redeploys keep the
SandboxDO container-enabled. A tenant script that is updated after a working deploy could later fail withSandbox.jsrpc → "Containers have not been enabled for this Durable Object class"(observed ontenant-tarjetero-dev): the DO keeps the code it was instantiated with, so the missing enablement stayed latent until the object was evicted (here, after thev2026.10.29upgrade and the Dev instance’s idle stop).Lerma.Cloudflare.DispatchNamespacenow accepts either class names or full container entries incontainers, andLerma.Cloudflare.Containers.deploy_script_and_container/1always sends the full entry (class_name,scheduling_policy,image,instance_type,max_instances) — a class-name-only entry is enough for the first upload but not for an update. Regression tests:apps/lerma/test/lerma/cloudflare/dispatch_namespace_test.exs(metadata carries the full config) andcontainers_test.exs(deploy passes it through); docsapps/lerma/README.md(“Container-backed tenant deploys”). - Lerma deploys are idempotent around Cloudflare DO migration conflicts (#516). A tenant Worker’s Durable Object class set was derived from the script’s bindings, but Cloudflare validates
new_sqlite_classesagainst the script’s migration history, so the two can diverge: an upgrade redeploy that binds fewer classes (or a rollback that deletes the script while leaving its classes migrated) made every subsequent deploy re-declareTelegramIngressand fail with400 code 10074— deterministically, so provisioning could not recover on its own.Lerma.Cloudflare.DispatchNamespace.upload_script/6now retries the upload without any class Cloudflare reports as already migrated (10074names it), switching an implicit full first-deploy set to an explicitnew_sqlite_classeslist; each retry removes at least one class, so it terminates, and the retry is logged.Lerma.Instances.instance_bindings/2now re-declares the same DO set as provisioning (Sandbox,ACTIVITY,TELEGRAM_INGRESS, plusBACKUP_WATCHwhile the flag is on) so an upgrade cannot detach a migrated class from its binding. Provisioning cleanup and a failed atomic deploy tear the classes down first viadelete_durable_object_namespaces/2(never on an upgrade rollback, whose Durable Objects hold the live tenant’s state:preserve_do_namespaces). Tests:apps/lerma/test/lerma/cloudflare/dispatch_namespace_test.exs(migrated-but-unbound re-deploy, resurrected first deploy, unrelated error not retried),containers_test.exs,tenant_provisioner_pipeline_test.exs; docsapps/lerma/README.md. - Cloudflare Observability destination ingestion (issue #337). Account-level Observability → Destinations now push Workers Logs and Workers Traces to Lerma over OTLP/HTTP JSON, giving event-driven worker diagnostics and uptime statistics that scale with traffic instead of tenant count. (1) Config:
apps/fouria/wrangler.jsonc(env.staging→lerma-worker-logs/lerma-worker-traces,env.production→lerma-worker-logs-prod/lerma-worker-traces-prod) andLerma.Cloudflare.Observability.worker_metadata/0reference the environment’s destination names, with per-signal head sampling (10% at rollout) andpersist: falsewhen a destination is configured (Lerma is the destination of record). The Lerma CD passes the matching names to staging and production, and the token to both; it also reconciles the account-level destination pairs (mix lerma.observability.destinations --warn-only) before each deploy so a fresh environment self-heals once the Cloudflare credential has Workers Observability Write. NewWORKER_OBSERVABILITY_LOGS/TRACES_DESTINATIONS,*_SAMPLING_RATEand*_PERSISTenv vars. (2) Ingest: newPOST /api/ingest/otel/v1/logsand/v1/traces(bearerOTEL_INGEST_TOKEN, constant-time compare, fail closed; OTLP/HTTP JSON only — protobuf →415+ hint;202 {accepted, skipped}so Cloudflare never retry-loops).Lerma.Observability.Otlpis a pure allowlist parser: it keeps script/dispatch namespace/outcome/handler/status/exception class names/cpu+wall/trace+span+invocation ids/timestamps, drops log text, stacks, URLs/query strings, user agents, request/response bodies and DB query text, and keeps only invocation root spans. Rows land inobservability_eventswith a uniquededupe_key+ON CONFLICT DO NOTHING, so replayed batches (at-least-once delivery) add no rows. Cloudflare gzips each OTLP/HTTP JSON batch while keepingContent-Type: application/json;LermaWeb.Plugs.OtlpBodyReaderdecompresses it beforePlug.Parsers(otherwise the destination retries on a400forever). (3) Uptime:GET /api/analytics/v1/uptime/:script_name?window=24h|7d|30d|<n>h|<n>dcomputes uptime/failure ratios from pushed root spans only; no events observed (including under sampling) →nullratios +status: "unknown", never downtime. The instance page gained a Worker telemetry widget (window selector + uptime/failure/invocation/CPU/wall), andLerma.Observability.Prunersweeps 90-day retention daily.HealthMonitorand billing flows are untouched. Tests:test/lerma/observability/otlp_test.exs,telemetry_test.exs,test/lerma/cloudflare/observability_test.exs,test/lerma_web/controllers/api/otlp_ingest_controller_test.exs,observability_controller_test.exs,test/lerma_web/components/worker_telemetry_widget_test.exs,test/lerma_web/live/instance_live_test.exs; docsdocs/CLOUD-ARCHITECTURE.md,docs/ENV-VARIABLES.md,docs/TESTING.md,apps/fouria/public/articles/PRIVACY.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,SECRETS.example.yaml,docs/schemas/secrets-manifest.schema.json,apps/lerma/.env.example. - Health endpoints no longer require the ROOT role at the edge. A tenant ADMIN opening
/_adminwas 302’d to the Cloudflare Access login forGET /api/healthandGET /api/health/ws(the dashboard’s health stream), because the per-tenantFourIA Health - <slug>Access app on<slug>.fouria.io/api/healthallowed only thePaso4 Root Usersgroup plus lerma’s health-monitor service token. The health app (and the platform-wide*.fouria.io/api/health/*.dev.fouria.io/api/healthIaC apps) now carries no role requirement: anallowpolicy oneveryone(any authenticated user) plus the existingnon_identityservice token, so the dashboard and lerma’s headless probe both work.AccessPolicy.health_policies/1drops the root-group policy (the host app keeps it);access.tfpublic_healthmatches. Tests:test/lerma/cloudflare/access_policy_test.exs,apps/fouria/iac/tests/access_app.tftest.hcl; docsapps/lerma/README.md,apps/fouria/iac/README.md,apps/fouria/docs/WORKER-DEVELOPMENT.md,docs/ENV-VARIABLES.md. Existing tenants were reconciled in place via the Cloudflare API (aud preserved). - Paso4-exclusive instances + agent: lerma can now provision internal
scope: "paso4"instances (nullableclient_id, no billing plan,@paso4.io-only Access), and moltlazy ships an opt-inpaso4-devagent (PASO4_AGENTS, gated byMOLTLAZY_PASO4_AGENTS=trueand named inMOLTLAZY_AGENTS) that picks up GitHub issues via the bundledgithubskill (ghCLI, installed in the image) authenticated byGITHUB_TOKEN. The Dispatch Worker resolves paso4 tenants frominstances(noclientsrow) and rejects non-@paso4.ioemails. See apps/lerma/README.md anddocs/ENV-VARIABLES.md.