Lsyncd fails with Out of inotify Watches Error

Following from a previous article where I demonstrated how to synchronise data to a remote host using lsyncd, this one with show how to deal with the below problem I faced.

The problem I faced was, although lsyncd has started successfully, the synchronisation had not begun. In the previous article, I created a directory and some files to test the sync – and this worked. However, when I tried to copy some data for real – 500G odd – the sync did not happen.

After checking the log file:

root@repo:~# less /var/log/lsyncd/lsyncd.log

…I found the following error:

...
Wed Apr  7 14:27:40 2021 Normal: Finished (list): 0
Wed Apr  7 16:48:55 2021 Normal: --- TERM signal, fading ---
Wed Apr  7 16:49:13 2021 Normal: recursive startup rsync: /srv/repositories-remote/ -> 192.168.124.16:/srv/repositories/
Wed Apr  7 16:49:14 2021 Normal: Startup of "/srv/repositories-remote/" finished: 0
Wed Apr  7 16:50:38 2021 Normal: --- TERM signal, fading ---
Wed Apr  7 16:53:50 2021 Error: Terminating since out of inotify watches.
Consider increasing /proc/sys/fs/inotify/max_user_watches
Wed Apr  7 17:07:53 2021 Normal: --- TERM signal, fading ---
...

Fortunately this problem is easy to solve – we just need to increase a kernel-based limitation. I believe one inotify watch is required per file/directory and they use about 1kB each for a 64 bit system. The variable we need to alter is max_user_watches.

You can check the current value with:

root@repo:~# cat /proc/sys/fs/inotify/max_user_watches 
8192

To adjust this value – first stop the lsyncd daemon process:

root@repo:~# systemctl stop lsyncd.service

Issue the following command to change the current running value in the kernel:

root@repo:~# sysctl fs.inotify.max_user_watches=400000

And then restart the lsyncd process:

root@repo:~# systemctl start lsyncd.service

Then check the log again and make sure the previous error is no longer showing up. In which case the synchronisation should begin.

The change this permanently, run something like this:

root@repo:~# sysctl -w fs.inotify.max_user_watches=400000 | tee -a /etc/sysctl.conf && sysctl -p

The above command sets the current kernel value as well as update the configuration file. We also force sysctl to reload the configuration for good measure.