Fix "too many open files" for SystemD managed service¶
- Table of contents
- Fix "too many open files" for SystemD managed service
Synopsis¶
For processes that are spawned normally (like your shell), the maximum number of open file handles is configured through /etc/security/limits.conf and files in /etc/security/limits.d/.
SystemD being what it is, completely ignores these settings and uses its own configuration, falling back to (what it considers) reasonable defaults if no explicit limits are set.
This means that to increase limits for services that are started by SystemD, you'll need to edit the .service file of the service.
Terminology¶
There are two kinds of limits: soft limits and hard limits:- hard limit: the upper limit for a given resource. This cannot be increased (unless you're root)
- soft limit: the current limit for a given resource. Can be increased up to the hard limit
Attribution¶
The guide at https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ helped me to figure out how to do this.
Checking current limits¶
System-wide limit¶
The upper limit (for file handles) imposed by the kernel can not be exceeded.
Check it with:
# cat /proc/sys/fs/file-max 9223372036854775807
That's plenty :)
The limits of a specific process¶
I used ps auxww | grep <processname> to get the PID of the process I was interested in.
With the PID, we can check the limits for that process:
# cat /proc/2315036/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 128309 128309 processes
Max open files 1024 524288 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 128309 128309 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
So we see that the soft limit is 1024, which is a bit on the low side. Let's increase it to 4096.
Increasing the limit for open files¶
List of limits that can be changed¶
SystemD allows to specify limits in a .service file.
These are the limits that can be specified:- LimitCPU
- LimitFSIZE
- LimitDATA
- LimitSTACK
- LimitCORE
- LimitRSS
- LimitNOFILE
- LimitAS
- LimitNPROC
- LimitMEMLOCK
- LimitLOCKS
- LimitSIGPENDING
- LimitMSGQUEUE
- LimitNICE
- LimitRTPRIO
- LimitRTTIME
We are interested in LimitNOFILE.
See the systemd.exec(5) manpage for more information.
Change the limit¶
Instead of directly editing the .service file, we'll use a so-called systemd drop-in. This keeps the original .service file intact (which is greatly appreciated by your package manager), but allows for selective overriding of settings from that original file.
NOTE: For some reason SystemD uses nano as its default editor. If you want to use vim, you can do that by executing export EDITOR=vim before executing the command below.
Execute systemctl edit <service-name>.service (in my case the service is zabbix-agent2) to open an editor with the override file and add the lines in yellow:
### Editing /etc/systemd/system/zabbix-agent2.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file
[Service]
LimitNOFILE=4096
### Lines below this comment will be discarded
### /lib/systemd/system/zabbix-agent2.service
# [Unit]
# Description=Zabbix Agent 2
# After=syslog.target
# After=network.target
#
# [Service]
# Environment="CONFFILE=/etc/zabbix/zabbix_agent2.conf"
# EnvironmentFile=-/etc/default/zabbix-agent2
# Type=simple
# Restart=on-failure
# PIDFile=/run/zabbix/zabbix_agent2.pid
# KillMode=control-group
# ExecStart=/usr/sbin/zabbix_agent2 -c $CONFFILE
# ExecStop=/bin/kill -SIGTERM $MAINPID
# RestartSec=10s
# User=zabbix
# Group=zabbix
#
# [Install]
# WantedBy=multi-user.target
You can leave the commented out stuff as it is, it gives you a bit of context with the contents of the original .service file.
NOTE: Don't forget to add the [Service] header! If you don't, you will get no error, but it won't work. Ask me how I know…
Restart the service¶
After saving and quitting the editor, there is no need to explicitly execute systemctl daemon-reload. Because the editing was done through a systemd mechanism, that is taken care of automatically.
Restart the service: systemctl restart zabbix-agent2.service.
Get the PID of the restarted process and check its limits:
# cat /proc/2333192/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 128309 128309 processes
Max open files 4096 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 128309 128309 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Apparently both the soft and the hard limit has been set to the value configured in the service override, but that's not the end of the world.
Both soft and hard limits can be specified with LimitNOFILE=4096:35536.
Updated by offbyone 5 months ago · 3 revisions