Prometheus/Grafana in AKS – No data points for pods

Here is a quick one I ran into today after installing Prometheus/Grafana again to my AKS.

I was looking into the graphs for my Pods but it basically didn’t show any data, all other graphs like Nodes or Deployments were showing everything fine.

All I saw was “No data points” …

So I was trying to figure out why it was missing data and had a look into Prometheus metrics:

kubectl --namespace monitoring port-forward $(kubectl get pod --namespace monitoring -l prometheus=kube-prometheus -l app=prometheus -o template --template "{{(index .items 0).metadata.name}}") 9090:9090

What I saw was all kubelet targets not working with an 401 Unauthorized error, so I had a quick google and found that it is a known issue.

https://github.com/coreos/prometheus-operator/issues/926

From here its pretty easy to fix:

Get the kube-prometheus-exporter-kubelets yaml:

kubectl -n monitoring get servicemonitor kube-prometheus-exporter-kubelets -o yaml

Replace all https with http and apply the yaml again:

kubectl replace -f youryamlfile.yaml

It should show this then:

“servicemonitor.monitoring.coreos.com “kube-prometheus-exporter-kubelets” replaced”

After you have done this, give it a few minutes and check your Grafana again, it should show all the data points now.

Designing Distributed Systems

I recently saw a Microsoft post on Twitter advertising this free e-book, so I gave it a go and it’s quite an interesting easy read. It’s only 160 pages, so good stuff for a few evenings. If you are interested in distributed systems..microservices..container and so on, you should read it.

You can download it here:

Designing Distributed Systems

This book gives you an introduction into some of the basic patterns you normally use in distributed systems. If you are like me, you probably will have used at least a few of them but might have not heard the correct “term” of it. That’s fine, I’m really bad it, it’s basically the reason I’m bad at those tech knowledge questions in interviews, where you have to explain pattern xy but you have no idea, until they explain it to you quick and you think, well I have used that for ages. That’s why I like technically discussions about actual work you did more than stupid questions.

Coming back to the book and its content. The book contains four parts:

Continue reading “Designing Distributed Systems”