What?
Imaging you are running RabbitMQ in production, everything is fine…until you introduce a new product or feature which increases the amount of queues and connections you have. Suddenly your system is experiencing a drop in performance at peak time and you have no idea whats going. You log into RabbitMQ management website and under “Overview” the File Descriptors are showing a very low number of available descriptors, it is possible you started with the default number. If you are lucky you started with 4096 if not you might have started with 1024, which is incredibly low unless you are only running on a few queues and connections but in my case I was running with over 10k queues and over 500 connections. So is that number affecting the performance of your RabbitMQ? It certainly does, that number shows you how many file handles and network sockets RabbitMQ has available and you can imagine what happens if its running out of those, like mine was on Saturday afternoon. But the good thing is, you can increase them!
Increase them!
First lets check the limits.
ulimit -a
ulimit -n
Under “open files” you might see 4096, lets change that.
First we are increasing the maximum number of files in sysctl.
nano /etc/sysctl.conf
Add and Save:
fs.file-max = 100000
Now we load sysctl again.
sysctl -p
Next step is actually increasing the limits, but first we check the Hard/Soft limits.
ulimit -Hn
ulimit -Sn
Your soft limit can not go higher than your hard limit. So if you hard limit is 4096 and you want to go up to 65535 for example, then you have to increase the hard limit. Setting the limits only in the limits.conf would not work.
nano /etc/security/limits.conf
Add and Save:
* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535This will also increase the socket descriptors that you see on the overview.
It will increase the limits for all users, if you only want to do the RabbitMQ user, add:
rabbitmq soft nofile 65535
rabbitmq hard nofile 65535
If you check the limits again after saving, it will still show you the old values, that’s because it can’t change the limits for running services.
Now we have to enable pam, so that users can have higher values.
nano /etc/pam.d/login
Add and Save:
session required pam_limits.so
You can now check the limits for the RabbitMQ user with:
su -rabbitmq -s /bin/sh -c 'ulimit -n'
It should show you 65535 already but RabbitMQ will still show the old value on the overview or with:
rabbitmqctl status |grep -A 4 limit
Now check the limits.conf of RabbitMQ, mine had an entry with 4096 which made it load that value all the time.
nano etc/systemd/system/rabbitmq-server.service.d/limits.conf
Add and Save:
[Service]
LimitNOFILE=65535
If you dont have systemd, just do:
ulimit -n 65535
Now you are all set. Just reboot!
After reboot you can check the limits again.
ulimit -Hn
ulimit -Sn
They should show 65535.
Same with:
rabbitmqctl status |grep -A 4 limit
If you go to the RabbitMQ management website now, the file descriptor and socket descriptor should show a lot of free descriptors. If not you might have to increase them again.
Useful links for how to set up your production RabbitMQ are:
https://www.rabbitmq.com/production-checklist.html
https://www.rabbitmq.com/install-rpm.html#linux-max-open-files-limit-options-other-linux