Files

kubernetes/apps/observability/silence-operator/silences/silences.yaml

1 ---

# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/observability.giantswarm.io/silence_v1alpha2.json

3 apiVersion: observability.giantswarm.io/v1alpha2

4 kind: Silence

5 metadata:

6 name: nas-memory-high-utilization

7 spec:

8 matchers:

9 - name: alertname

10 value: NodeMemoryHighUtilization

11 - name: instance

12 value: bigmouth.internal:9100

13 ---

# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/observability.giantswarm.io/silence_v1alpha2.json

15 apiVersion: observability.giantswarm.io/v1alpha2

16 kind: Silence

17 metadata:

18 name: hpa-maxed-out

19 spec:

20 matchers:

21 - name: alertname

22 value: KubeHpaMaxedOut

23 ---

# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/observability.giantswarm.io/silence_v1alpha2.json

25 apiVersion: observability.giantswarm.io/v1alpha2

26 kind: Silence

27 metadata:

28 name: kube-memory-overcommit

29 spec:

30 matchers:

31 - name: alertname

32 value: KubeMemoryOvercommit

33 ---

# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/observability.giantswarm.io/silence_v1alpha2.json

35 apiVersion: observability.giantswarm.io/v1alpha2

36 kind: Silence

37 metadata:

38 name: bigmouth-smart-device-interface-slow

39 spec:

40 matchers:

41 - name: alertname

42 value: SmartDeviceInterfaceSlow

43 - name: instance

44 value: bigmouth.internal:9633

45 ---

# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/observability.giantswarm.io/silence_v1alpha2.json

47 apiVersion: observability.giantswarm.io/v1alpha2

48 kind: Silence

49 metadata:

50 name: flux-instance-artifact-failed

51 spec:

52 matchers:

53 - name: alertname

54 value: FluxFluxInstanceArtifactfailed

55 ---

# yaml-language-server: $schema=https://k8s-schemas.home-operations.com/observability.giantswarm.io/silence_v1alpha2.json

57 apiVersion: observability.giantswarm.io/v1alpha2

58 kind: Silence

59 metadata:

60 name: flux-oci-repository-ociartifact-pull-failed

61 spec:

62 matchers:

63 - name: alertname

64 value: FluxOCIRepositoryOciartifactpullfailed

FluxFluxInstanceArtifactfailed: temporary IPv6 noise suppression

why ·ok L53

Silence the `FluxFluxInstanceArtifactfailed` alert because it's pure notification noise, not a real failure. The alert fires when the flux-operator tries to pull `ghcr.io/controlplaneio-fluxcd/flux-operator-manifests:v0.57.0` from `pkg-containers.githubusercontent.com` over IPv6 first, fails with `dial tcp [2606:50c0:8003::154]:443: connect: network is unreachable`, and recovers automatically on the IPv4 fallback in 2-4 seconds. Alertmanager sees the fire+resolve cycle as a real blip and pushes a Pushover notification for every blip. Cluster has been working fine through hundreds of these.

Show the rest

This was moved from `alertmanagerconfig.yaml` to silence-operator on 2026-08-08 so it sits with the other suppressions and is independently auditable. The `FluxOCIRepositoryOciartifactpullfailed` silence below shares the same IPv6-pull root cause — see that annotation.

**Why:** this is a stopgap, not a permanent fix. **How to apply:** to remove the suppression, fix the underlying IPv6 unreachable condition for `pkg-containers.githubusercontent.com` (and any other AAAA-only host) by doing ONE of:

1. Set `GODEBUG=preferipv4=1` on the flux-operator container env in `kubernetes/apps/flux-system/flux-operator/app/helmrelease.yaml`. This is the cleanest fix because it's per-binary and survives node network reconfigs. 2. Add `ip6tables -I OUTPUT -d 2606:50c0:8003::/48 -j DROP` on each Talos node (machine config). Wider blast radius — affects every binary, not just flux-operator. 3. Deploy NodeLocal DNSCache with `preferIPv4: true` and route cluster DNS through it. Most invasive.

Option 1 is recommended — smallest change, scoped to flux-operator. Option 3 also fixes any other Go binary that hits AAAA-only hosts (kopiur, miroir, etc.).

**Trigger to revisit:** when migrating flux-operator off `controlplaneio-fluxcd` to a registry that resolves only on IPv4 (or when all three nodes get IPv6 connectivity from the LAN). Confirm the blip is actually gone by tailing `kubectl logs -n flux-system flux-operator` for `network is unreachable` before removing the silence. Keep this annotation alive until then so the next reader knows why a "real" alert is being silently dropped.

2026-08-0701KZEJY797HZ1JMVN4NK83XA2Q

FluxOCIRepositoryOciartifactpullfailed: same IPv6 noise

why ·ok L63

Silence `FluxOCIRepositoryOciartifactpullfailed` for the same reason as `FluxFluxInstanceArtifactfailed` above: every OCIRepository reconcile against `ghcr.io/...` first resolves an AAAA record for `pkg-containers.githubusercontent.com`, tries IPv6, fails with `network is unreachable`, and recovers on the IPv4 fallback. Alertmanager sees the fire+resolve cycle and pushes notifications per blip.

Show the rest

The alert is generated from `controller_runtime_reconcile_total{reason="OCIArtifactPullFailed",kind="OCIRepository"}` on source-controller. Flux uses a `<Kind><Reason>` naming convention, so `OCIArtifactPullFailed` on `OCIRepository` becomes `FluxOCIRepositoryOciartifactpullfailed` in Prometheus.

**Why:** sibling of `FluxFluxInstanceArtifactfailed` — both surface the same upstream IPv6-first pull behavior. Suppressing only one would leave notifications firing from the other. **How to apply:** when the IPv6 underlying reachability is fixed (see the `FluxFluxInstanceArtifactfailed` annotation for options, all of which apply here too), remove this silence in the same change. **Trigger to revisit:** confirm `OCIArtifactPullFailed` reconcile errors are gone by checking `controller_runtime_reconcile_total{reason="OCIArtifactPullFailed"}` is flat-zero for 24h before removing.

References: Flux monitoring docs at https://fluxcd.io/flux/monitoring/, fluxcd/flux2 issue #4711 for the `OCIArtifactPullFailed` reconcile reason on sub-path registries.

2026-08-0801KZGC750A131024A6C26BZ3GG