Expand the existing volume for Kubeflow MinIO instance – Arrikto

In certain scenarios underlying volume used by Kubeflow's MinIO instance might become full. This article talks about steps of growing the size of underlying volume.

Determine if PVC need to be expanded

1) Find the minio PVC

kubectl -n kubeflow get pvc | grep minio

kubectl -n kubeflow describe pvc minio-pv-claim

2) Confirm the logs for failure and check the disk usage by exec-ing into the pod to assure that disk is indeed full.

kubectl -n kubeflow logs <pod>

kubectl -n kubeflow exec -ti <pod> /bin/sh

df -h

In our case above cmd was showing that we only have 0.9G available on data volume

Expand the PVC

1) If you are convinced that MinIO volume is indeed full and need to expand the size of the volume, we will follow the offline volume expand process.

Take the snapshot of desired volume
Scaled down the pod which is using the volume so that it will get unmounted.
Add a new patch for desired PVC and add data-source as snapshot of original volume
Use rok-deploy to apply above customize changes and create new bigger volume
Scale up the pod to remount the larger volume.

2) Bring down the minio pod and wait for pvc to be unmounted

kubectl -n kubeflow scale deploy minio --replicas=0
kubectl -n kubeflow get pods |grep minio
kubectl -n kubeflow describe pvc minio-pv-claim

above cmd output should shows that PVC is mounted by none

3) Following doc explains how to take the snapshot of pvc

https://docs.arrikto.com/user/rok/snapshots.html#snapshot-pvc-backed-by-rok

Create new snapshot using above spec for ns kubeflow and mino pvc

vi minio-snapshot.yaml
---------------------------
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
  name: minio-pv-snapshot
  namespace: kubeflow
spec:
  snapshotClassName: rok
  source:
    name:  minio-pv-claim
    kind: PersistentVolumeClaim

Create the snapshot
```
kubectl apply -f minio-snapshot.yaml
```

Wait for snapshot to be ready

kubectl get -f minio-snapshot.yaml -o yaml
kubectl get -f pvc.yaml -o yaml
kubectl -n kubeflow get volumesnapshot

4) Create a new patch for minio pvc to create a new volume from the snapshot.

Find the current spec for minio pvc

kustomize build kubeflow/manifests/apps/pipeline/upstream/overlays/deploy |less

Copy the spec for minio-pvc-claim form above and create new patch file with it

mkdir kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches
vi kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/minio-restore-from-snapshot.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-pv-claim
  namespace: kubeflow
spec:
  storageClassName: rok
  dataSource:
    name: minio-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Add the new patch to kustomize

vi kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/kustomization.yaml

add following

- path: patches/minio-restore-from-snapshot.yaml

your file will look as below:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: kubeflow

resources:
- ../ekf

patches:
- path: patches/minio-restore-from-snapshot.yaml

verify that kustomize build is creating the patch correctly

kustomize build kubeflow/manifests/apps/pipeline/upstream/overlays/deploy |less

commit the new changes

git status
git add
git commit -m "Grow minio-pv-claim PVC to 50Gi"

5) Delete the old pvc

kubectl -n kubeflow delete pvc minio-pv-claim

6) apply new kustomize patch to create new pvc from the snapshot

rok-deploy --apply kubeflow/manifests/apps/pipeline/upstream/overlays/deploy

wait for pvc to come up

kubectl -n kubeflow describe pvc minio-pv-claim

9) scale minio pod back up.

kubectl -n kubeflow scale deploy minio --replicas=1

10) make sure minio pod is back up and running and volume size is as expected.

kubectl -n kubeflow logs <pod>

kubectl -n kubeflow exec -ti <pod> /bin/sh

> df -h

At the end you should have your cluster back up and running with new desired size of minio volume.

Determine if PVC need to be expanded

Expand the PVC

Related articles