Following from a previous article where I demonstrated how to synchronise data to a remote host using lsyncd, this one with show how to deal with the below problem I faced.
The problem I faced was, although lsyncd has started successfully, the synchronisation had not begun. In the previous article, I created a directory and some files to test the sync – and this worked. However, when I tried to copy some data for real – 500G odd – the sync did not happen.
After checking the log file:
[email protected]:~# less /var/log/lsyncd/lsyncd.log
…I found the following error:
... Wed Apr 7 14:27:40 2021 Normal: Finished (list): 0 Wed Apr 7 16:48:55 2021 Normal: --- TERM signal, fading --- Wed Apr 7 16:49:13 2021 Normal: recursive startup rsync: /srv/repositories-remote/ -> 192.168.124.16:/srv/repositories/ Wed Apr 7 16:49:14 2021 Normal: Startup of "/srv/repositories-remote/" finished: 0 Wed Apr 7 16:50:38 2021 Normal: --- TERM signal, fading --- Wed Apr 7 16:53:50 2021 Error: Terminating since out of inotify watches. Consider increasing /proc/sys/fs/inotify/max_user_watches Wed Apr 7 17:07:53 2021 Normal: --- TERM signal, fading --- ...
Fortunately this problem is easy to solve – we just need to increase a kernel-based limitation. I believe one
inotify watch is required per file/directory and they use about 1kB each for a 64 bit system. The variable we need to alter is
You can check the current value with:
[email protected]:~# cat /proc/sys/fs/inotify/max_user_watches 8192
To adjust this value – first stop the
lsyncd daemon process:
[email protected]:~# systemctl stop lsyncd.service
Issue the following command to change the current running value in the kernel:
[email protected]:~# sysctl fs.inotify.max_user_watches=400000
And then restart the
[email protected]:~# systemctl start lsyncd.service
Then check the log again and make sure the previous error is no longer showing up. In which case the synchronisation should begin.
The change this permanently, run something like this:
[email protected]:~# sysctl -w fs.inotify.max_user_watches=400000 | tee -a /etc/sysctl.conf && sysctl -p
The above command sets the current kernel value as well as update the configuration file. We also force
sysctl to reload the configuration for good measure.