I have several statically provisioned PV/PVCs. When the PV/PVCs are first applied by either flux or via kubectl, the PVCs show up as "Bound" and everything looks good. However whenever flux next reconciles the repository, the logs say that flux detected a change and the PVC is now reported as lost:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-homer-pvc Lost nfs-homer-pv 0 11m
After digging around a bit, I figured out that spec.ClaimRef.uid is being removed from the PV. If I manually specify PV.spec.ClaimRef.uid with the value of the uid from the PersistentVolumeClaim it survives reconciliation.
This behavior does NOT occur if I apply the configuration manually with kubectl.
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-homer-pv
namespace: default
spec:
capacity:
storage: 128Mi
accessModes:
- ReadWriteMany
nfs:
path: /tank/appdata/homer
server: 192.168.1.10
persistentVolumeReclaimPolicy: Retain
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
namespace: default
name: nfs-homer-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-homer-pvc
namespace: default
spec:
storageClassName: ""
accessModes:
- ReadWriteMany
resources:
requests:
storage: 128Mi
Here is a blurb from the logs. Most of the PVs here have already had the uid removed and their associated PVCs in a "Lost" state which is why they are marked as unchanged. I was working with home, so note the "PersistentVolume/nfs-homer-pv":"configured" in the json below.
{"level":"info","ts":"2021-12-28T00:50:00.912Z","logger":"controller.kustomization","msg":"server-side apply completed","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"apps","namespace":"flux-system","output":{"Certificate/networking/mcf-ninja":"unchanged","ConfigMap/monitoring/loki-chunks-dashboard":"unchanged","ConfigMap/monitoring/loki-deletion-dashboard":"unchanged","ConfigMap/monitoring/loki-logs-dashboard":"unchanged","ConfigMap/monitoring/loki-mixin-recording-rules-dashboard":"unchanged","ConfigMap/monitoring/loki-operational-dashboard":"unchanged","ConfigMap/monitoring/loki-reads-dashboard":"unchanged","ConfigMap/monitoring/loki-reads-resources-dashboard":"unchanged","ConfigMap/monitoring/loki-retention-dashboard":"unchanged","ConfigMap/monitoring/loki-writes-dashboard":"unchanged","ConfigMap/monitoring/loki-writes-resources-dashboard":"unchanged","HelmRelease/default/heimdall":"unchanged","HelmRelease/default/homer":"configured","HelmRelease/default/paperless":"unchanged","HelmRelease/default/photoprism":"unchanged","HelmRelease/home/home-assistant":"unchanged","HelmRelease/media/plex":"unchanged","HelmRelease/monitoring/grafana":"unchanged","HelmRelease/monitoring/kube-prometheus-stack":"unchanged","HelmRelease/monitoring/loki":"unchanged","HelmRelease/monitoring/promtail":"unchanged","HelmRelease/networking/ingress-nginx":"unchanged","HelmRelease/nextcloud/nextcloud":"unchanged","PersistentVolume/nfs-heimdall-pv":"unchanged","PersistentVolume/nfs-home-assistant-config-pv":"unchanged","PersistentVolume/nfs-homer-pv":"configured","PersistentVolume/nfs-loki-pv":"unchanged","PersistentVolume/nfs-movies-pv":"unchanged","PersistentVolume/nfs-music-pv":"unchanged","PersistentVolume/nfs-nextcloud-data-pv":"unchanged","PersistentVolume/nfs-nextcloud-mariadb-pv":"unchanged","PersistentVolume/nfs-paperless-consume-pv":"unchanged","PersistentVolume/nfs-paperless-data-pv":"unchanged","PersistentVolume/nfs-paperless-export-pv":"unchanged","PersistentVolume/nfs-paperless-media-pv":"unchanged","PersistentVolume/nfs-photoprism-import-pv":"unchanged","PersistentVolume/nfs-photoprism-originals-pv":"unchanged","PersistentVolume/nfs-photoprism-storage-pv":"unchanged","PersistentVolume/nfs-plex-pv":"unchanged","PersistentVolume/nfs-prometheus-pv":"unchanged","PersistentVolume/nfs-tv-pv":"unchanged","PersistentVolumeClaim/default/nfs-heimdall-pvc":"unchanged","PersistentVolumeClaim/default/nfs-homer-pvc":"unchanged","PersistentVolumeClaim/default/nfs-paperless-consume-pvc":"unchanged","PersistentVolumeClaim/default/nfs-paperless-data-pvc":"unchanged","PersistentVolumeClaim/default/nfs-paperless-export-pvc":"unchanged","PersistentVolumeClaim/default/nfs-paperless-media-pvc":"unchanged","PersistentVolumeClaim/default/nfs-photoprism-import-pvc":"unchanged","PersistentVolumeClaim/default/nfs-photoprism-originals-pvc":"unchanged","PersistentVolumeClaim/default/nfs-photoprism-storage-pvc":"unchanged","PersistentVolumeClaim/home/nfs-home-assistant-config-pvc":"unchanged","PersistentVolumeClaim/media/nfs-movies-pvc":"unchanged","PersistentVolumeClaim/media/nfs-music-pvc":"unchanged","PersistentVolumeClaim/media/nfs-plex-pvc":"unchanged","PersistentVolumeClaim/media/nfs-tv-pvc":"unchanged","PersistentVolumeClaim/monitoring/nfs-loki-pvc":"unchanged","PersistentVolumeClaim/monitoring/prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0":"unchanged","PersistentVolumeClaim/nextcloud/nfs-nextcloud-data-pvc":"unchanged","PersistentVolumeClaim/nextcloud/nfs-nextcloud-mariadb-pvc":"unchanged","Plan/system-upgrade/k3s-agent":"unchanged","Plan/system-upgrade/k3s-server":"unchanged"}}
Describe the bug
I have several statically provisioned PV/PVCs. When the PV/PVCs are first applied by either flux or via kubectl, the PVCs show up as "Bound" and everything looks good. However whenever flux next reconciles the repository, the logs say that flux detected a change and the PVC is now reported as lost:
After digging around a bit, I figured out that spec.ClaimRef.uid is being removed from the PV. If I manually specify PV.spec.ClaimRef.uid with the value of the uid from the PersistentVolumeClaim it survives reconciliation.
This behavior does NOT occur if I apply the configuration manually with kubectl.
Here is my configuration
Steps to reproduce
flux reconcileor just wait for it to do it on it's own.Expected behavior
The dynamically added uid should not be removed from
PV.spec.claimRef.uidwhen the repository is reconciled.Screenshots and recordings
No response
OS / Distro
Ubuntu 20.04
Flux version
flux: v0.24.1
Flux check
► checking prerequisites
✔ Kubernetes 1.22.5+k3s1 >=1.19.0-0
► checking controllers
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.19.0
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.14.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.18.2
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.19.2
✔ all checks passed
Git provider
GitHub
Container Registry provider
n/a
Additional context
Here is a blurb from the logs. Most of the PVs here have already had the uid removed and their associated PVCs in a "Lost" state which is why they are marked as unchanged. I was working with home, so note the
"PersistentVolume/nfs-homer-pv":"configured"in the json below.Code of Conduct