Acessing External System with User Credentials (Kaggle Example) – Arrikto

Issues This KB Resolves

Pulling data from an external system without workload identity mapping
Accessing Kaggle datasets from a Kubeflow as a Service

Summary

Often during a pipeline step or from a notebook, we need to access an external API. This NORMALLY can be done with workload identity on GCP. If we are unable to use our service account tokens directly, we can query an API using stored credentials. This is NOT totally secure. You shouldn't be storing any credentials on a cluster, but if you need to store credentials as a secret and use the token for a non super secure project, this is the best way.
How does this work?

This process goes through the vanilla pipeline and Kale ways of adding a secret to a pipeline step. The process creates a secret, mounts that secret to a pod with the "kaggle-secret: "true"" label, uses the secret mounted on the volume to pull the specified data set from the Kaggle API, and saves it locally to the volume at a specified path.

The Process

1. Create the secret for our credentials so we can mount them to our pods

kubectl create secret generic kaggle-secret --from-literal=KAGGLE_USERNAME=<username> --from-literal=KAGGLE_KEY=<api_token>

2. Create a pod-default resource to mount the secret to any pod with a specific label ( in our case kaggle-secret=true)

apiVersion: kubeflow.org/v1alpha1
desc: kaggle-access
kind: PodDefault
metadata:
  name: kaggle-access
spec:
  selector:
    matchLabels:
      gitaccess: "true"
  env:
  - name: KAGGLE_USERNAME
    valueFrom:
      secretKeyRef:
        name: kaggle-secret
        key: KAGGLE_USERNAME
  - name: KAGGLE_KEY
    valueFrom:
      secretKeyRef:
        name: kaggle-secret
        key: KAGGLE_KEY

3. Leverage our "download_kaggle_dataset" function without passing our credentials directly since they should be mounted at the desired path.

def download_kaggle_dataset(data_set:str, path:str):
     import os
     import glob
     import kaggle
     os.chdir(os.environ.get('HOME'))
     os.system("mkdir " + path)
     os.chdir(path)
     os.system("kaggle datasets download -d " + data_set + " --unzip")
     extension = 'csv'
     csv = glob.glob('*.{}'.format(extension))
     csv=csv[0]
     print(csv)
     os.system("mv " + csv + " data.csv") # renames our data to data.csv for easier calling

4. Last but not least, we need to make sure our pipeline step has the right label. Easy enough.

get_data = download_kaggle_dataset_op("l3llff/-dark-souls-3-weapon","/mnt/data").add_pvolumes({"/mnt": vop.volume}).add_pod_label("kaggle-secret", "true")

If you are using Kale, you can add a label via DeployConfig Parameter or if the notebook itself has the checkbox checked to use the appropriate PodDefaults, it should pass the labels to the step.

if you are running KubeFlow 1.5 or later (at the time of this writing KFaaS leverages 1.4.3)
Basically Navigate to the configuration type you want to add and expand the menu of that group. For each section you want to add a configuration for, click the + Add button and fill in the form.

If you are using earlier to 1.4.3 and once you deploy the PodDefault resource, you can create a notebook with access to the kaggle APIs.
from that notebook, you can run the Kaggle function manually, and then skip the step. The data will be present and used across your pipeline. The data will be on the initial notebook you are leveraging.

def download_kaggle_dataset(data_set:str, path:str):
     import os
     import glob
     import kaggle
     os.chdir(os.environ.get('HOME'))
     os.system("mkdir " + path)
     os.chdir(path)
     os.system("kaggle datasets download -d " + data_set + " --unzip")
     extension = 'csv'
     csv = glob.glob('*.{}'.format(extension))
     csv=csv[0]
     print(csv)
     os.system("mv " + csv + " data.csv") # renames our data to data.csv for easier calling

download_kaggle_dataset("l3llff/-dark-souls-3-weapon","./")

Troubleshooting Step

Kaggle 401 error

If you are getting a 401 error when downloading a Kaggle dataset, make sure you saved the appropriate credentials within your "kaggle-secret" secret in step 1. You may have to recreate your token on the kaggle site.

Checking Secrets

If you change the values of a Kubernetes secret or configmap, you will need to recreate the pod. To check if the secrets are mounted as the appropriate environment variables run the env | grep KAGGLE command from your terminal within your Jupyter notebook. You should see KAGGLE_USERNAME and KAGGLE_KEY with the desired values. Confirm these are the values you previously generated.
If you generate new values, you will need to start and stop your notebook from the Kubeflow UI. This will scale the pod down and then back up. Hit the square "stop" button to stop the notebook. Wait for the loading circle to stop, and then hit the triangle "play" button to restart the notebook. Wait and connect once available. You should have the new environment values present.

If the values AREN'T present, you must make sure you installed the appropraite PodDefault resource, as well as confirm the pod was indeed mutated. The PodDefault resource must have the EXACT spacing as shown here so that the mutation happens appropriatly.
To check for the mutation run kubectl get pods command and find the pod with your notebook name.

Then run kubectl get pods <your pod name> -o yaml | grep poddefault

If you do not see the above annotions, you will need to confirm you did select the appropriate configuration check box during notebook creation. You can also delete and recreate the notebook with the appropriate configuration check boxes checked to recreate the pod and mutate the resource.

Notebook Won't Start After PodDefault Deployment

By default, if a secret is not available that the pod is requesting, the pod will not create properly. Find the pending pod by running kubectl get pods and finding the pod that is stuck. The pod should be names <your notebookname>-0. Then you can describe the pod by running kubectl describe pod <pod name> and see what the issue is with the pod. The most common error is the "kaggle-secret" secret doesn't exist. Please make sure the kaggle-secret secret does exist on the cluster by running kubectl get secrets and finding the kaggle-secret secret.

Kaggle Competition Rules Error (403 Forbidden)

If you have not accepted the competition's rules, you will not be able to download the dataset. You must navigate to the appopriate Kaggle competition page and accept the competition rules.

Related articles