In certain scenarios underlying volume used by Kubeflow's MinIO instance might become full. This article talks about steps of growing the size of underlying volume.
Determine if PVC need to be expanded
1) Find the minio PVC
kubectl -n kubeflow get pvc | grep minio
kubectl -n kubeflow describe pvc minio-pv-claim
2) Confirm the logs for failure and check the disk usage by exec-ing into the pod to assure that disk is indeed full.
kubectl -n kubeflow logs <pod>
kubectl -n kubeflow exec -ti <pod> /bin/sh
df -h
In our case above cmd was showing that we only have 0.9G available on data volume
above cmd output should shows that PVC is mounted by none
Expand the PVC
1) If you are convinced that MinIO volume is indeed full and need to expand the size of the volume, we will follow the offline volume expand process.
- Take the snapshot of desired volume
- Scaled down the pod which is using the volume so that it will get unmounted.
- Add a new patch for desired PVC and add data-source as snapshot of original volume
- Use rok-deploy to apply above customize changes and create new bigger volume
- Scale up the pod to remount the larger volume.
2) Bring down the minio pod and wait for pvc to be unmounted
kubectl -n kubeflow scale deploy minio --replicas=0
kubectl -n kubeflow get pods |grep minio
kubectl -n kubeflow describe pvc minio-pv-claim
3) Following doc explains how to take the snapshot of pvc
https://docs.arrikto.com/user/rok/snapshots.html#snapshot-pvc-backed-by-rok
- Create new snapshot using above spec for ns kubeflow and mino pvc
vi minio-snapshot.yaml
---------------------------
apiVersion: snapshot.storage.k8s.io/v1alpha1 kind: VolumeSnapshot metadata: name: minio-pv-snapshot
namespace: kubeflow spec: snapshotClassName: rok source: name: minio-pv-claim
kind: PersistentVolumeClaim - Create the snapshot
kubectl apply -f minio-snapshot.yaml
- Wait for snapshot to be ready
kubectl get -f minio-snapshot.yaml -o yaml
kubectl get -f pvc.yaml -o yaml
kubectl -n kubeflow get volumesnapshot
4) Create a new patch for minio pvc to create a new volume from the snapshot.
- Find the current spec for minio pvc
kustomize build kubeflow/manifests/apps/pipeline/upstream/overlays/deploy |less
- Copy the spec for minio-pvc-claim form above and create new patch file with it
mkdir kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches
vi kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/minio-restore-from-snapshot.yamlapiVersion: v1 kind: PersistentVolumeClaim metadata: name: minio-pv-claim
namespace: kubeflow spec: storageClassName: rok dataSource: name: minio-snapshot kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 50Gi
- Add the new patch to kustomize
vi kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/kustomization.yaml
add following- path: patches/minio-restore-from-snapshot.yaml
your file will look as below:
verify that kustomize build is creating the patch correctlyapiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization
namespace: kubeflow
resources: - ../ekf
patches: - path: patches/minio-restore-from-snapshot.yamlkustomize build kubeflow/manifests/apps/pipeline/upstream/overlays/deploy |less
commit the new changesgit status
git add
git commit -m "Grow minio-pv-claim PVC to 50Gi"
5) Delete the old pvc
kubectl -n kubeflow delete pvc minio-pv-claim
6) apply new kustomize patch to create new pvc from the snapshot
rok-deploy --apply kubeflow/manifests/apps/pipeline/upstream/overlays/deploy
wait for pvc to come up
kubectl -n kubeflow describe pvc minio-pv-claim
9) scale minio pod back up.
kubectl -n kubeflow scale deploy minio --replicas=1
10) make sure minio pod is back up and running and volume size is as expected.
kubectl -n kubeflow logs <pod>
kubectl -n kubeflow exec -ti <pod> /bin/sh
> df -h
At the end you should have your cluster back up and running with new desired size of minio volume.
Comments
0 comments
Please sign in to leave a comment.