# Persistent workspace policy — Phase 0 (measurement)

**Date:** 2026-10-30
**Status:** Implemented (instrumentation only)
**Design:** `https://design.paso4.io` → **Backup policy** page (12-board proposal)
**Next:** Phase 1 (snapshot on evict) is gated on the numbers below.

---

## 1. Why this phase exists

The current policy treats the archive as both the routine save path and the
disaster-recovery artifact. Every save is a full squashfs: staged in the
container, encrypted in the Worker, uploaded to R2, pruned to three versions.
`BACKUP_INTERVAL_MINUTES` is a floor that stretches to twice the last observed
snapshot, so a slow tenant snapshots back-to-back.

The proposed rework splits persistence into two tiers:

- **Tier 1 — container snapshot** (`ctx.container.snapshotContainer()`): the
  routine save path, seconds, image-pinned, 30-day TTL.
- **Tier 2 — portable archive** (today's F4E1 archive): the recovery point,
  portable across images, encrypted at rest.

Choosing the Tier 1 cadence, the dirty-window bound, and the archive ceiling
requires data we do not have. Phase 0 collects it without changing behaviour:
**no snapshot, restore, eviction, or retention decision reads these events.**

## 2. What is measured

| Event                  | Emitted from                                                  | Answers                                                 |
| ---------------------- | ------------------------------------------------------------- | ------------------------------------------------------- |
| `persistence-snapshot` | `finalizeSnapshot` (success); `ActivityScheduler` (fail/skip) | How big is an archive? How long does each stage take?   |
| `persistence-restore`  | `restoreIfNeeded`, `restoreBackupById`                        | How long does recovery actually take, and why?          |
| `backup-change-window` | `BackupWatch` tick                                            | How long is a tenant continuously dirty before capture? |

### `persistence-snapshot`

`double1` = end-to-end (`prepare` start → `finalize` commit), `double2` =
`sandbox.createBackup()` alone, `double3` = captured workspace bytes (from the
archive's `meta.json`). Blobs: customer, `trigger` (`scheduled` / `manual` /
`deprovision`), `outcome` (`succeeded` / `failed` / `skipped`), `detail`
(failure name or skip reason), platform, environment.

Skip rows are the denominator: `interval-not-elapsed`, `unchanged`,
`not-running`, `backup-lock-busy`. Without them a quiet series looks healthy.

### `persistence-restore`

`double1` = duration, `double2` = archive bytes. Blobs: customer, `source`
(`recovery-point` for a cold boot, `explicit` for an operator restore),
`outcome`, `detail`, platform, environment. An archive that is present but
unusable is recorded as `skipped` / `archive-unusable`, not as a generic failure.

### `backup-change-window`

`double1` = continuous dirty time at the observation (`dirtySince → now`),
`double2` = container-side scan duration. Blobs: customer, `status`
(`unchanged` / `changed` / `active` / `unknown`), `decision` (`job-active`,
`unchanged`, `deferred-active`, `deferred-floor`, `requested`, `busy`,
`request-failed`, `disarmed`), platform, environment.

Two decisions are taken **before** any container RPC, from the
`ActivityScheduler`'s own job record (`snapshotGate`): `job-active` (a snapshot
already owns the slot) and `deferred-floor` (inside the cadence floor). Both
report `status = unknown` with `scanMs = 0` — no scan ran — so a query must not
read them as "the tenant was inspected and found clean".

The watch is feature-flagged (`BACKUP_CHANGE_WATCH`), so this series only exists
where the watch runs. That is intentional: the watch is the mechanism whose
cadence the rework changes.

## 3. Queries

Archive volume per tenant per day:

```sql
SELECT blob2 AS customer,
       SUM(double3) AS bytes,
       SUM(double1) AS snapshot_ms,
       SUM(double2) AS archive_ms,
       COUNT(*)     AS snapshots
FROM fourai_cost_metrics
WHERE blob1 = 'persistence-snapshot'
  AND blob4 = 'succeeded'
  AND timestamp > NOW() - INTERVAL 1 DAY
GROUP BY customer
ORDER BY bytes DESC
```

Dirty-window distribution (p50/p95 by status):

```sql
SELECT blob2 AS customer, blob3 AS status,
       COUNT(*) AS ticks,
       SUM(double1) AS dirty_ms_total,
       MAX(double1) AS dirty_ms_max,
       SUM(double2) AS scan_ms_total
FROM fourai_cost_metrics
WHERE blob1 = 'backup-change-window'
  AND timestamp > NOW() - INTERVAL 7 DAY
GROUP BY customer, status
ORDER BY customer
```

Why snapshots did not run:

```sql
SELECT blob2 AS customer, blob4 AS outcome, blob5 AS detail, COUNT(*) AS n
FROM fourai_cost_metrics
WHERE blob1 = 'persistence-snapshot'
  AND blob4 != 'succeeded'
  AND timestamp > NOW() - INTERVAL 7 DAY
GROUP BY customer, outcome, detail
ORDER BY n DESC
```

Restore cost (cold boots vs operator restores):

```sql
SELECT blob2 AS customer, blob3 AS source, blob4 AS outcome,
       COUNT(*) AS n, SUM(double1) AS total_ms, MAX(double1) AS max_ms,
       SUM(double2) AS bytes
FROM fourai_cost_metrics
WHERE blob1 = 'persistence-restore'
  AND timestamp > NOW() - INTERVAL 30 DAY
GROUP BY customer, source, outcome
```

## 4. Gates for Phase 1 (snapshot on evict)

Phase 1 may start when the Phase 0 series shows, for at least three tenants over
two weeks:

1. **Archive size is stable and material.** A p95 archive above roughly 100 MB
   justifies a snapshot-based save path; a p95 below ~10 MB means the archive is
   already cheap and the rework should be deferred.
2. **The archive step dominates.** `double2 / double1` above ~0.5 means
   `createBackup` (not staging or bookkeeping) is the cost to remove.
3. **Dirty windows are bounded.** The p95 `dirtyWindowMs` for `changed` ticks is
   the RPO the checkpoint cadence must beat. If it is already under the proposed
   checkpoint interval, Tier 1 buys resume latency (not freshness), and Phase 1
   is justified on cold-start time rather than RPO.
4. **Skips are explained.** Every non-succeeded row carries a reason. An
   unexplained `failed` cluster blocks the phase.

## 5. Explicitly out of scope

- No new binding, env var, or migration.
- No container snapshot call: Tier 1 does not exist yet, so `archiveDurationMs`
  is the `createBackup` time, not a snapshot time.
- No change to retention, the schedule gate, the lease, or the restore ladder.
- The `backup-change-window` event is emitted only where `BACKUP_CHANGE_WATCH`
  is enabled; enabling the watch for measurement is a separate operator choice.

## 6. Implementation

- `src/metrics.ts` — `trackPersistenceSnapshot`, `trackPersistenceRestore`,
  `trackBackupChangeWindow` (row layouts documented above).
- `src/persistence.ts` — `SnapshotPhaseContext.trigger`, `SnapshotArchive.sizeBytes`
  (one `meta.json` read at `archive`), and the success/failure/skip emissions.
- `src/activity/scheduler.ts` — trigger labelling, skip rows (gate, sleeping,
  busy lock), failure rows per failed attempt.
- `src/backup-watch/watch.ts` (+ `durable-object.ts`) — the pure tick returns an
  observation and the DO writes the row, so the state machine stays metric-free.
- Corpus: `test/business-rules/cost-tracking.txt` and
  `test/business-rules/fixture/expected-cost-schema.json`.
