Please refer to the following commands to perform a health check or investigation into Rok Monitoring Stack.
Prometheus Operator
1. Check the status of the Prometheus Operator Deployment:
$ kubectl get deployment -n monitoring prometheus-operator -o wide
2. Inspect the events on the Prometheus Operator Deployment:
$ kubectl describe deployment -n monitoring prometheus-operator
3. Check the status of the Prometheus Operator Pod:
$ kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus-operator -o wide
Prometheus
1. Check the status of the Prometheus StatefulSet:
$ kubectl get statefulset -n monitoring prometheus-k8s -o wide
2. Inspect the events on the Prometheus StatefulSet:
$ kubectl describe statefulset -n monitoring prometheus-k8s
3. Check the status of the Prometheus Pod:
$ kubectl get pods -n monitoring -l app=prometheus -o wide
Grafana
1. Check the status of the Grafana Deployment:
$ kubectl get deployment -n monitoring grafana -o wide
2. Inspect the events on the Grafana Deployment:
$ kubectl describe deployment -n monitoring grafana
3. Check the status of the Grafana Pod:
$ kubectl get pods -n monitoring -l app=grafana -o wide
Node Exporter
1. Check the status of the Node Exporter DaemonSet:
$ kubectl get daemonset -n monitoring node-exporter -o wide
2. Inspect the events on the Node Exporter DaemonSet:
$ kubectl describe daemonset -n monitoring node-exporter
3. Check the status of the Node Exporter Pod:
$ kubectl get pods -n monitoring -l app.kubernetes.io/name=node-exporter -o wide
Kube State Metrics
1. Check the status of the Kube State Metrics Deployment:
$ kubectl get daemonset -n monitoring -o wide
2. Inspect the events on the Kube State Metrics Deployment:
$ kubectl describe deployment -n monitoring kube-state-metrics
3. Check the status of the Kube State Metrics Pod:
$ kubectl get pods -n monitoring -l app.kubernetes.io/name=kube-state-metrics -o wide
Target Discovery
1. Verify that the following ServiceMonitors exist:
$ kubectl get servicemonitors -A
NAMESPACE NAME AGE
monitoring coredns 2h
monitoring grafana 2h
monitoring kube-apiserver 2h
monitoring kube-controller-manager 2h
monitoring kube-scheduler 2h
monitoring kube-state-metrics 2h
monitoring kubelet 2h
monitoring node-exporter 2h
monitoring prometheus 2h
monitoring prometheus-operator 2h
rok rok-etcd-metrics 2h
rok rok-metrics 2h
rok rok-redis-metrics 2h
2. Verify that Prometheus has discovered Rok as a target and is collecting metrics from it:
$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO -
localhost:9090/metrics | grep 'discovered.*rok-metrics'
3. Verify that Prometheus has discovered Rok etcd as a target and is collecting metrics from it:
$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO -
localhost:9090/metrics | grep 'discovered.*rok-etcd-metrics'
4. Verify that Prometheus has discovered Rok Redis as a target and is collecting metrics from it:
$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO -
localhost:9090/metrics | grep 'discovered.*rok-redis-metrics'
5. Verify that Prometheus has discovered Node Exporter as a target and is collecting metrics from it:
$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO -
localhost:9090/metrics | grep 'discovered.*node-exporter’
6. Verify that Prometheus has discovered Kube State Metrics as a target and is collecting metrics from it:
$ kubectl exec -ti -n monitoring sts/prometheus-k8s -c prometheus -- wget -qO -
localhost:9090/metrics | grep 'discovered.*kube-state-metrics’
Comments
0 comments
Please sign in to leave a comment.