Here is a quick one I ran into today after installing Prometheus/Grafana again to my AKS.
I was looking into the graphs for my Pods but it basically didn’t show any data, all other graphs like Nodes or Deployments were showing everything fine.
All I saw was “No data points” …
So I was trying to figure out why it was missing data and had a look into Prometheus metrics:
kubectl --namespace monitoring port-forward $(kubectl get pod --namespace monitoring -l prometheus=kube-prometheus -l app=prometheus -o template --template "{{(index .items 0).metadata.name}}") 9090:9090
What I saw was all kubelet targets not working with an 401 Unauthorized error, so I had a quick google and found that it is a known issue.
https://github.com/coreos/prometheus-operator/issues/926
From here its pretty easy to fix:
Get the kube-prometheus-exporter-kubelets yaml:
kubectl -n monitoring get servicemonitor kube-prometheus-exporter-kubelets -o yamlReplace all https with http and apply the yaml again:
kubectl replace -f youryamlfile.yamlIt should show this then:
“servicemonitor.monitoring.coreos.com “kube-prometheus-exporter-kubelets” replaced”
After you have done this, give it a few minutes and check your Grafana again, it should show all the data points now.