I’m currently working a lot with Azure Kubernetes Service – AKS again and since I’m trying to determine the best setup regarding nodes and their settings I started testing the “Cluster Autoscaler” a bit more and the rather new “Multiple Node Pools” as-well.
The cluster autoscaler is around for a bit now, so its basically working fine but it’s still in preview. Multiple node pools are pretty new and also in preview.
To start with I will give you a little context about the ELK setup I was working on and how it was being used to better understand the issue I will describe I ran into.
I’m currently using the ELK stack for logging purposes, so I dump all log messages from my code onto a RabbitMQ and from there it will get consumed by multiple Logstash server dumping the messages into Elasticsearch. Those Logstash server are setup to push to different indexes in Elasticsearch, mainly to differentiate the log messages.
So imagine there is an error log queue and an audit log queue in RabbitMQ, you now have a Logstash server to consume from both queues and depending on the message/queue it should push to either the error index or the audit index.
So each Logstash server had two config files, one for every queue to consume from. Every config file had the RabbitMQ input plugin, a custom transformation for some fields and the Elasticsearch output plugin.