F FourIA staging 2026.10.29 GitHub ↗

Persistent workspace policy — Phase 0 (measurement)

On this page

Date: 2026-10-30 Status: Implemented (instrumentation only) Design: https://design.paso4.io → Backup policy page (12-board proposal) Next: Phase 1 (snapshot on evict) is gated on the numbers below.


1. Why this phase exists

The current policy treats the archive as both the routine save path and the disaster-recovery artifact. Every save is a full squashfs: staged in the container, encrypted in the Worker, uploaded to R2, pruned to three versions. BACKUP_INTERVAL_MINUTES is a floor that stretches to twice the last observed snapshot, so a slow tenant snapshots back-to-back.

The proposed rework splits persistence into two tiers:

  • Tier 1 — container snapshot (ctx.container.snapshotContainer()): the routine save path, seconds, image-pinned, 30-day TTL.
  • Tier 2 — portable archive (today’s F4E1 archive): the recovery point, portable across images, encrypted at rest.

Choosing the Tier 1 cadence, the dirty-window bound, and the archive ceiling requires data we do not have. Phase 0 collects it without changing behaviour: no snapshot, restore, eviction, or retention decision reads these events.

2. What is measured

EventEmitted fromAnswers
persistence-snapshotfinalizeSnapshot (success); ActivityScheduler (fail/skip)How big is an archive? How long does each stage take?
persistence-restorerestoreIfNeeded, restoreBackupByIdHow long does recovery actually take, and why?
backup-change-windowBackupWatch tickHow long is a tenant continuously dirty before capture?

persistence-snapshot

double1 = end-to-end (prepare start → finalize commit), double2 = sandbox.createBackup() alone, double3 = captured workspace bytes (from the archive’s meta.json). Blobs: customer, trigger (scheduled / manual / deprovision), outcome (succeeded / failed / skipped), detail (failure name or skip reason), platform, environment.

Skip rows are the denominator: interval-not-elapsed, unchanged, not-running, backup-lock-busy. Without them a quiet series looks healthy.

persistence-restore

double1 = duration, double2 = archive bytes. Blobs: customer, source (recovery-point for a cold boot, explicit for an operator restore), outcome, detail, platform, environment. An archive that is present but unusable is recorded as skipped / archive-unusable, not as a generic failure.

backup-change-window

double1 = continuous dirty time at the observation (dirtySince → now), double2 = container-side scan duration. Blobs: customer, status (unchanged / changed / active / unknown), decision (job-active, unchanged, deferred-active, deferred-floor, requested, busy, request-failed, disarmed), platform, environment.

Two decisions are taken before any container RPC, from the ActivityScheduler’s own job record (snapshotGate): job-active (a snapshot already owns the slot) and deferred-floor (inside the cadence floor). Both report status = unknown with scanMs = 0 — no scan ran — so a query must not read them as “the tenant was inspected and found clean”.

The watch is feature-flagged (BACKUP_CHANGE_WATCH), so this series only exists where the watch runs. That is intentional: the watch is the mechanism whose cadence the rework changes.

3. Queries

Archive volume per tenant per day:

SELECT blob2 AS customer,
       SUM(double3) AS bytes,
       SUM(double1) AS snapshot_ms,
       SUM(double2) AS archive_ms,
       COUNT(*)     AS snapshots
FROM fourai_cost_metrics
WHERE blob1 = 'persistence-snapshot'
  AND blob4 = 'succeeded'
  AND timestamp > NOW() - INTERVAL 1 DAY
GROUP BY customer
ORDER BY bytes DESC

Dirty-window distribution (p50/p95 by status):

SELECT blob2 AS customer, blob3 AS status,
       COUNT(*) AS ticks,
       SUM(double1) AS dirty_ms_total,
       MAX(double1) AS dirty_ms_max,
       SUM(double2) AS scan_ms_total
FROM fourai_cost_metrics
WHERE blob1 = 'backup-change-window'
  AND timestamp > NOW() - INTERVAL 7 DAY
GROUP BY customer, status
ORDER BY customer

Why snapshots did not run:

SELECT blob2 AS customer, blob4 AS outcome, blob5 AS detail, COUNT(*) AS n
FROM fourai_cost_metrics
WHERE blob1 = 'persistence-snapshot'
  AND blob4 != 'succeeded'
  AND timestamp > NOW() - INTERVAL 7 DAY
GROUP BY customer, outcome, detail
ORDER BY n DESC

Restore cost (cold boots vs operator restores):

SELECT blob2 AS customer, blob3 AS source, blob4 AS outcome,
       COUNT(*) AS n, SUM(double1) AS total_ms, MAX(double1) AS max_ms,
       SUM(double2) AS bytes
FROM fourai_cost_metrics
WHERE blob1 = 'persistence-restore'
  AND timestamp > NOW() - INTERVAL 30 DAY
GROUP BY customer, source, outcome

4. Gates for Phase 1 (snapshot on evict)

Phase 1 may start when the Phase 0 series shows, for at least three tenants over two weeks:

  1. Archive size is stable and material. A p95 archive above roughly 100 MB justifies a snapshot-based save path; a p95 below ~10 MB means the archive is already cheap and the rework should be deferred.
  2. The archive step dominates. double2 / double1 above ~0.5 means createBackup (not staging or bookkeeping) is the cost to remove.
  3. Dirty windows are bounded. The p95 dirtyWindowMs for changed ticks is the RPO the checkpoint cadence must beat. If it is already under the proposed checkpoint interval, Tier 1 buys resume latency (not freshness), and Phase 1 is justified on cold-start time rather than RPO.
  4. Skips are explained. Every non-succeeded row carries a reason. An unexplained failed cluster blocks the phase.

5. Explicitly out of scope

  • No new binding, env var, or migration.
  • No container snapshot call: Tier 1 does not exist yet, so archiveDurationMs is the createBackup time, not a snapshot time.
  • No change to retention, the schedule gate, the lease, or the restore ladder.
  • The backup-change-window event is emitted only where BACKUP_CHANGE_WATCH is enabled; enabling the watch for measurement is a separate operator choice.

6. Implementation

  • src/metrics.ts — trackPersistenceSnapshot, trackPersistenceRestore, trackBackupChangeWindow (row layouts documented above).
  • src/persistence.ts — SnapshotPhaseContext.trigger, SnapshotArchive.sizeBytes (one meta.json read at archive), and the success/failure/skip emissions.
  • src/activity/scheduler.ts — trigger labelling, skip rows (gate, sleeping, busy lock), failure rows per failed attempt.
  • src/backup-watch/watch.ts (+ durable-object.ts) — the pure tick returns an observation and the DO writes the row, so the state machine stays metric-free.
  • Corpus: test/business-rules/cost-tracking.txt and test/business-rules/fixture/expected-cost-schema.json.