Prometheus/Grafana in AKS – No data points for pods

Here is a quick one I ran into today after installing Prometheus/Grafana again to my AKS.

I was looking into the graphs for my Pods but it basically didn’t show any data, all other graphs like Nodes or Deployments were showing everything fine.

All I saw was “No data points” …

So I was trying to figure out why it was missing data and had a look into Prometheus metrics:

kubectl --namespace monitoring port-forward $(kubectl get pod --namespace monitoring -l prometheus=kube-prometheus -l app=prometheus -o template --template "{{(index .items 0).metadata.name}}") 9090:9090

What I saw was all kubelet targets not working with an 401 Unauthorized error, so I had a quick google and found that it is a known issue.

https://github.com/coreos/prometheus-operator/issues/926

From here its pretty easy to fix:

Get the kube-prometheus-exporter-kubelets yaml:

kubectl -n monitoring get servicemonitor kube-prometheus-exporter-kubelets -o yaml

Replace all https with http and apply the yaml again:

kubectl replace -f youryamlfile.yaml

It should show this then:

“servicemonitor.monitoring.coreos.com “kube-prometheus-exporter-kubelets” replaced”

After you have done this, give it a few minutes and check your Grafana again, it should show all the data points now.

“ReferencePath” does not define a value for metadata “CopyLocal”

After upgrading Visual Studio 2017 to 15.8.5 I ran into an error while building an existing WPF project:

The item "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\lib\Microsoft.Win32.Primitives.dll" in item list "ReferencePath" does not define a value for metadata "CopyLocal". In order to use this metadata, either qualify it by specifying %(ReferencePath.CopyLocal), or ensure that all items in this list define a value for this metadata.

Google didn’t really help that time and since it was happening on building while working fine before I just tried replacing the Microsoft.Common.CurrentVersion.targets file in C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin and C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\amd64 with an version before the upgrade. I basically had to copy it from a different installation.

So this is an annoying solution and with the latest upgrade, it doesn’t seem fixed. It also works fine with existing WPF projects. You could remove the project and create a new one and just copy all old code over, but that is annoying too especially if you have a ton of references.

So I decided to compare the target files to see what actually changed and to be fair, you only have to change one single thing in the file:

Line 3961: Change …Include=”@(ReferencePath)” to …Include=”@(ReferencePath.CopyLocal)”

That’s all you have to do, like the error said itself, by specifying %(ReferencePath.CopyLocal).

Happy Days.