Tag: Azure Kubernetes Service

Azure Kubernetes Service – Autoscaler and multiple node pools

I’m currently working a lot with Azure Kubernetes Service – AKS again and since I’m trying to determine the best setup regarding nodes and their settings I started testing the “Cluster Autoscaler” a bit more and the rather new “Multiple Node Pools” as-well.

The cluster autoscaler is around for a bit now, so its basically working fine but it’s still in preview. Multiple node pools are pretty new and also in preview.

If you are not familiar with those two features:

https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler

https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools

So let me share a few issues I ran into and how I solved them…or not.

Continue reading “Azure Kubernetes Service – Autoscaler and multiple node pools”

Prometheus/Grafana in AKS – No data points for pods

Here is a quick one I ran into today after installing Prometheus/Grafana again to my AKS.

I was looking into the graphs for my Pods but it basically didn’t show any data, all other graphs like Nodes or Deployments were showing everything fine.

All I saw was “No data points” …

So I was trying to figure out why it was missing data and had a look into Prometheus metrics:

kubectl --namespace monitoring port-forward $(kubectl get pod --namespace monitoring -l prometheus=kube-prometheus -l app=prometheus -o template --template "{{(index .items 0).metadata.name}}") 9090:9090

What I saw was all kubelet targets not working with an 401 Unauthorized error, so I had a quick google and found that it is a known issue.

https://github.com/coreos/prometheus-operator/issues/926

From here its pretty easy to fix:

Get the kube-prometheus-exporter-kubelets yaml:

kubectl -n monitoring get servicemonitor kube-prometheus-exporter-kubelets -o yaml

Replace all https with http and apply the yaml again:

kubectl replace -f youryamlfile.yaml

It should show this then:

“servicemonitor.monitoring.coreos.com “kube-prometheus-exporter-kubelets” replaced”

After you have done this, give it a few minutes and check your Grafana again, it should show all the data points now.