Files

kubernetes/components/kopiur/backup/restore.yaml

1 ---

2 # yaml-language-server: $schema=https://k8s-schemas.home-operations.com/kopiur.home-operations.com/restore_v1alpha1.json

3 apiVersion: kopiur.home-operations.com/v1alpha1

4 kind: Restore

5 metadata:

6 name: ${APP}

7 spec:

8 failurePolicy:

9 backoffLimit: ${KOPIUR_RESTORE_BACKOFF:=8}

10 policy:

11 onMissingSnapshot: Continue

12 source:

13 fromPolicy:

14 name: ${APP}

15 offset: ${KOPIUR_RESTORE_OFFSET:=0}

16 target:

17 populator: {}

Backoff spans the repo's read-after-write lag; offset is the plex knob

why ·ok L8

Without a failurePolicy the mover Job exhausts its few default retries in minutes and the Restore goes terminal. backoffLimit 8 gives the Job exponential backoff spanning roughly 25 minutes, so a genuinely transient read failure on this NFS-backed repository becomes a self-healing wait instead of "terminal, recreate the CR by hand". A Failed Restore never retries, so CRs that failed before this knob existed still need manual deletion.

Show the rest

Correction, 2026-09-04. This knob was added during the 2026-09-03 local-storage migration on the theory that the kopia "content not found" failures seen on first-attempt restores (17 across three batches, every recreation succeeding) were transient reads of fresh writes. The failures examined afterwards (qbittorrent, stash) were nothing of the kind: each Restore had pinned its snapshot at admission weeks earlier and retention had since pruned it, so no amount of retrying could find the content, while a recreated CR re-resolved to a live snapshot and succeeded at once. See the gotcha on `source:` below. Keep the backoff for real transients, but do not expect it to rescue a stale Restore.

KOPIUR_RESTORE_OFFSET exists for restoring an older snapshot than latest (0 = newest): set it in an app's ks postBuild when the newest snapshots are known-bad, as plex's recovery needed on 2026-09-03 when Preferences.xml was found empty, and REMOVE the override right after the restore completes, or every future PVC rebuild silently resurrects old data. Because the source is pinned at admission, changing the offset on an existing Restore does nothing; delete the CR so Flux recreates it with the new value.

2026-09-0301M1KM0JSF6EE7PJEXZNJXC0HZ

A Restore pins its source at admission; recreate it before a PVC rebuild

gotcha ·ok L12

A Restore resolves its source once, at admission, and never again. `status.resolved` pins the kopia snapshot ID (or the NoSnapshot verdict) with a `pinnedAt` timestamp, and every PVC that later claims the same Restore through `dataSourceRef` receives exactly that pinned result, however old it is.

Two shapes of this bit during the 2026-09-03/04 local-storage migration:

Show the rest

- qbittorrent and stash. Their Restores dated from the first deploy on 2026-07-28 and had pinned that day's snapshot. When the PVCs were deleted and recreated on miroir-fast-local, the populator tried to restore the pinned ID, which retention had pruned weeks earlier, and failed terminally with kopia `content <id> not found`. The Failed CR then held both PVCs in Pending for ten hours. - t3-code. Its Restore had pinned `NoSnapshot` on 2026-08-25, the first deploy, when there was nothing to restore. The rebuilt PVC claimed it on 2026-09-04 and got an empty volume although a fresh `t3-code-migrate-local` snapshot existed. That is how the t3-code data was wiped.

So before deleting any kopiur-backed PVC for a rebuild, delete the app's Restore CR first and let Flux recreate it (`flux reconcile ks <app> -n <namespace>`); the fresh CR resolves against the current snapshot list. Doing it the other way round leaves a Pending PVC claiming a Failed or stale-Completed Restore, and the only remedy is the same delete-and-recreate.

This also reframes the "content not found" first-attempt failures seen across the migration: the ones examined on 2026-09-04 were all pinned-and-pruned snapshots, not repository flakiness, which is why every recreated Restore succeeded. A Completed Restore left in place after its PVC is bound is harmless until the next rebuild.

2026-09-0401M1NFBJ05BFN5T1AZK4W7RMPR