Health Check Commands: Rok Monitoring Stack – Arrikto

Please refer to the following commands to perform a health check or investigation into Rok Monitoring Stack.

Prometheus Operator

1. Check the status of the Prometheus Operator Deployment:

$ kubectl get deployment -n monitoring prometheus-operator -o wide

2. Inspect the events on the Prometheus Operator Deployment:

$ kubectl describe deployment -n monitoring prometheus-operator

3. Check the status of the Prometheus Operator Pod:

$ kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus-operator -o wide

Prometheus

1. Check the status of the Prometheus StatefulSet:

$ kubectl get statefulset -n monitoring prometheus-k8s -o wide

2. Inspect the events on the Prometheus StatefulSet:

$ kubectl describe statefulset -n monitoring prometheus-k8s

3. Check the status of the Prometheus Pod:

$ kubectl get pods -n monitoring -l app=prometheus -o wide

Grafana

1. Check the status of the Grafana Deployment:

$ kubectl get deployment -n monitoring grafana -o wide

2. Inspect the events on the Grafana Deployment:

$ kubectl describe deployment -n monitoring grafana

3. Check the status of the Grafana Pod:

$ kubectl get pods -n monitoring -l app=grafana -o wide

Node Exporter

1. Check the status of the Node Exporter DaemonSet:

$ kubectl get daemonset -n monitoring node-exporter -o wide

2. Inspect the events on the Node Exporter DaemonSet:

$ kubectl describe daemonset -n monitoring node-exporter

3. Check the status of the Node Exporter Pod:

$ kubectl get pods -n monitoring -l app.kubernetes.io/name=node-exporter -o wide

Kube State Metrics

1. Check the status of the Kube State Metrics Deployment:

$ kubectl get daemonset -n monitoring -o wide

2. Inspect the events on the Kube State Metrics Deployment:

$ kubectl describe deployment -n monitoring kube-state-metrics

3. Check the status of the Kube State Metrics Pod:

$ kubectl get pods -n monitoring -l app.kubernetes.io/name=kube-state-metrics -o wide

Target Discovery

1. Verify that the following ServiceMonitors exist:

$ kubectl get servicemonitors -A
NAMESPACE NAME                      AGE
monitoring   coredns                2h
monitoring   grafana                2h
monitoring   kube-apiserver         2h
monitoring   kube-controller-manager   2h
monitoring   kube-scheduler         2h
monitoring   kube-state-metrics     2h
monitoring   kubelet                2h
monitoring   node-exporter          2h
monitoring   prometheus             2h
monitoring   prometheus-operator    2h
rok       rok-etcd-metrics       2h
rok       rok-metrics            2h
rok       rok-redis-metrics      2h

2. Verify that Prometheus has discovered Rok as a target and is collecting metrics from it:

$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO - 
localhost:9090/metrics | grep 'discovered.*rok-metrics'

3. Verify that Prometheus has discovered Rok etcd as a target and is collecting metrics from it:

$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO - 
localhost:9090/metrics | grep 'discovered.*rok-etcd-metrics'

4. Verify that Prometheus has discovered Rok Redis as a target and is collecting metrics from it:

$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO - 
localhost:9090/metrics | grep 'discovered.*rok-redis-metrics'

5. Verify that Prometheus has discovered Node Exporter as a target and is collecting metrics from it:

$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO - 
localhost:9090/metrics | grep 'discovered.*node-exporter’

6. Verify that Prometheus has discovered Kube State Metrics as a target and is collecting metrics from it:

$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO - 
localhost:9090/metrics | grep 'discovered.*kube-state-metrics’

Useful links