Prometheus/Grafana in AKS – No data points for pods

Here is a quick one I ran into today after installing Prometheus/Grafana again to my AKS.

I was looking into the graphs for my Pods but it basically didn’t show any data, all other graphs like Nodes or Deployments were showing everything fine.

All I saw was “No data points” …

So I was trying to figure out why it was missing data and had a look into Prometheus metrics:

kubectl --namespace monitoring port-forward $(kubectl get pod --namespace monitoring -l prometheus=kube-prometheus -l app=prometheus -o template --template "{{(index .items 0).metadata.name}}") 9090:9090

What I saw was all kubelet targets not working with an 401 Unauthorized error, so I had a quick google and found that it is a known issue.

https://github.com/coreos/prometheus-operator/issues/926

From here its pretty easy to fix:

Get the kube-prometheus-exporter-kubelets yaml:

kubectl -n monitoring get servicemonitor kube-prometheus-exporter-kubelets -o yaml

Replace all https with http and apply the yaml again:

kubectl replace -f youryamlfile.yaml

It should show this then:

“servicemonitor.monitoring.coreos.com “kube-prometheus-exporter-kubelets” replaced”

After you have done this, give it a few minutes and check your Grafana again, it should show all the data points now.