Adding Custom Metadata to a Notebook or Pod

Issues This KB Resolves

Customers have asked us how to easily annotate notebooks and pods so that they can use the metadata for project tracking or other purposes. This KB walks the user through how to easily mutate pods with custom metadata.

Summary

The basic workflow is that a user will create a PodDefault resource in the kubeflow-skel namespace. That kubeflow-skel namespace uses the EKF skel-overlay to replicate the PodDefault resource across user namespaces. Once the PodDefault resource is present, the ability to add the configuration to a pod will be a configuration option during notebook creation.

How does this work?

A PodDefault resource is a custom resource used by our admission-webhook-deployment controller located in the Kubeflow namespace. This is a great place to look if you are having issues with your pod mutating.  Admission controllers come in 2 flavours, validating admission webhook and mutating admission webhook.  The doc states, "Mutating admission webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults."This is exactly what the PodDefault resource does best! Please visit the API doc for more details on what the PodDefault controller can do. The Profile Controller takes care of deploying some Kubernetes resources under each user namespace by default. Those resources allow users to access Rok and Kubeflow Pipelines.  The Profile Controller takes care of creating user namespaces and keeping them updated. The Skel Controller provides an automated way to apply a skeleton of Kubernetes resources to every user namespace.

On each reconciliation loop it

• Applies to the user namespace the resources that are under the kubeflow-skel namespace.
• Prunes resources under the user namespace, deleting those that are no longer under the kubeflow-skel namespace.

We can create a PodDefault resource that adds our custom annotation to any pod based on labels. The webhook will mutate the pod and add the required fields if the label is present.

The Process

1. Write a PodDefault manifest specifying the annotation you would like to apply and save it in a file named something identifiable such as "skynet-annotation-pd.yaml" in your admin workstation or notebook terminal.

apiVersion: "kubeflow.org/v1alpha1"
kind: PodDefault
metadata:
  name: add-skynet-annotation
spec:
  selector:
    matchLabels:
      skynet-project: "true"
    desc: "Add skynet project annotation"
    annotations:
      project: "skynet"

2. Manually apply the resource to your namespace to confirm it works with kubectl apply -f skynet-annotation-pd.yaml  -n <your namespace> from an admin workstation or a notebook terminal.

To validate that this works, create a new notebook and select the check box next to the configuration that matches the description we set above.  You can skip this if you are confident in your PodDefault definition.

create the notebook and wait for the notebook to be ready then connect.


You can then run a kubectl describe pod command on the notebook pod to confirm your annotation is there and the PodDefault was formatted properly.


You should note that you will see an annotation referencing that the pod was mutated and the PodDefault in question. In our example, we see poddefault.admission.kubeflow.org/poddefault-add-skynet-annotation .

You can also see that the project: skynet annotation was added as well as the skynet-project=true label.

3. Once the PodDefault definition is ready, you can apply it directly to the kubeflow-skel namespace by adding the appropriate namespace metadata to the PodDefault definition.

apiVersion: "kubeflow.org/v1alpha1"
kind: PodDefault
metadata: 
   name: add-skynet-annotation
   namespace:  kubeflow-skel
spec: 
   selector: 
     matchLabels: 
       skynet-project: "true"
 desc: "Add skynet project annotation"
 annotations:
  project: "skynet"

In theory, you could run kubectl apply -f skynet-annotation-pd.yaml 

This would directly apply that manifest into your cluster. You should always be checking manifests into git.  At the time of writing this article we currently leverage rok-tools as our GitOps repo. This manifest can be saved if you use something else to manage your GitOps.

For leveraging rok-tools:

1. Exec into your rok-tools pod by running:

   kubectl exec -it rok-tools-0 -- bash 
   or
   docker exec -ti rok-tools /bin/bash
   

2. Change directories to your ops/deployments directory by running:

   cd ~/ops/deployments/kubeflow/manifests
   

3. Change directories to your ops/deployments directory by running:

   cd ~/ops/deployments/kubeflow/manifests
   

4. Create your PodDefault manifest: 

   vim common/skel-resources/overlays/deploy/skynet-annotation-pd.yaml

5. Copy the below example and adapt it to your needs

   apiVersion: "kubeflow.org/v1alpha1"
   kind: PodDefault
   metadata: 
      name: add-skynet-annotation
      namespace:  kubeflow-skel
   spec: 
      selector: 
        matchLabels: 
          skynet-project: "true"
    desc: "Add skynet project annotation"
    annotations:
     project: "skynet"

    

6. Save and exit vim by pressing Esc button on your keyboard and typing

   :wq 

7. and press Enter
   
   
8. Modify the common/skel-resources/overlays/deploy/kustomization.yaml using your text editor of choice to include the files with your resources 

   vim common/skel-resources/overlays/deploy/kustomization.yaml

9. Edit the file to match your specifications.
   

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   
   namespace: kubeflow-skel
   
   resources:
   - ../../base
   - skynet-annotation-pd.yaml

10. Make sure that the configmap patch is enabled. Edit
    apps/profiles/upstream/overlays/deploy/kustomization.yaml
    and add patches/configmap.yaml under patches, if it doesn’t exist already:

     
    
    
    
11. Commit your changes with:
    

    git commit -am "Add extra skel resources"

    
    
12. Apply the customization by running:

    rok-deploy --apply common/skel-resources/overlays/deploy
    

     

13. Verify if the change was applied
    

    kubectl get poddefaults --all-namespaces | grep skynet
    

14. Example:

    
    # kubectl get poddefaults --all-namespaces | grep skynet
    default add-skynet-annotation 52m
    kubeflow-test-user add-skynet-annotation 46m
    kubeflow-skel add-skynet-annotation 46m
    kubeflow-user add-skynet-annotation 46m