Sync Data to a Remote Host using lsyncd

Based on rsync, lsyncd is a demonised implementation that monitors for file modifications, and triggers a synchronisation if detected.

Here I am using Ubuntu 18.04 at both ends but the process should pretty much be the same regardless with platform you’re using.

Here we are going to monitor for changes to files on a server called repo, and push when changes are detected to a server called mirror. The master server, repo, will need lsyncd installed and be able to SSH to mirror without a password.

On the master server, install lsyncd. On Ubuntu, this would be:

[email protected]:~# apt update -y && apt install lsyncd

You can check the version with:

[email protected]:~# lsyncd --version
Version: 2.1.6

To setup passwordless authentication – you’ll need a public/private key-pair. If you don’t already have one, you can create one using the ssh-keygen command. Accept all the default answers.

[email protected]:~# ssh-keygen

You can now copy the public key (~/.ssh/id_rsa.pub) to the remote host using the ssh-copy-id command:

[email protected]:~# ssh-copy-id [email protected] -p22

Check you can now SSH into the remote host without a password.

[email protected]:~# ssh [email protected]

Then you’ll need to create the following directory and configuration file.

[email protected]:~# mkdir -v /etc/lsyncd
mkdir: created directory '/etc/lsyncd'
[email protected]:~# vim /etc/lsyncd/lsyncd.conf.lua

Here I’m going to create a demo directory of data I want to monitor for changes.

[email protected]:~# mkdir -v test-lsyncd-source
mkdir: created directory 'test-lsyncd-source'

And on the remote host, I created the following directory that data will be synchronised to.

[email protected]:~# mkdir -v /srv/repositories/mirror
mkdir: created directory '/srv/repositories/mirror'

Create something like below:

settings {
  logfile = "/var/log/lsyncd/lsyncd.log",
  statusFile = "/var/log/lsyncd/lsyncd.status",
  statusInterval = 20,
  nodaemon   = false
}

sync {
  default.rsyncssh,
  source = "/root/test-lsyncd-source/",
  host = "192.168.124.16",
  targetdir = "/srv/repositories/mirror/",
  rsync = {
    owner = true,
    group = true,
    perms = true
  }
}

Before restarting the lsyncd service, you will first need to create the log directory it would seem.

[email protected]:~/test-lsyncd-source# mkdir -v /var/log/lsyncd
mkdir: created directory '/var/log/lsyncd'

Then restart the lsyncd service.

[email protected]:~/test-lsyncd-source# systemctl restart lsyncd

Start creating some files and check it syncs as expected.

[email protected]:~/test-lsyncd-source# ls -il
total 4
4387318 lrwxrwxrwx 1 root  root    10 Apr  7 13:52 data2 -> dir1/dir2/
4387319 lrwxrwxrwx 1 root  root    10 Apr  7 13:52 data3 -> dir1/dir3/
4387308 drwxr-xr-x 4 root  root  4096 Apr  7 13:48 dir1
4387306 -rw-r--r-- 1 andy andy    0 Apr  6 16:13 file2.txt
4387237 -rw-r--r-- 1 root  root     0 Apr  6 15:40 test.txt

You can check the status by tailing the following file.

[email protected]:~/test-lsyncd-source# fail -f /var/log/lsyncd/lsyncd.status
Lsyncd status report at Wed Apr  7 13:52:29 2021

Sync1 source=/root/test-lsyncd-source/
There are 0 delays
Excluding:
  nothing.


Inotify watching 4 directories
  1: /root/test-lsyncd-source/
  5: /root/test-lsyncd-source/dir1/dir2/
  6: /root/test-lsyncd-source/dir1/dir3/
  4: /root/test-lsyncd-source/dir1/

And by tailing the log file.

[email protected]:~/test-lsyncd-source# tail -f /var/log/lsyncd/lsyncd.log 
/dir1/dir2/data2.txt
Wed Apr  7 13:49:52 2021 Normal: Finished (list): 0
Wed Apr  7 13:50:14 2021 Normal: Rsyncing list
/dir1/dir3/data3.txt
/dir1/dir3/data4.txt
Wed Apr  7 13:50:15 2021 Normal: Finished (list): 0
Wed Apr  7 13:52:24 2021 Normal: Rsyncing list
/data2
/data3
Wed Apr  7 13:52:25 2021 Normal: Finished (list): 0

You may also want to check at this point that lsyncd is set to start at boot-up.

[email protected]:~# systemctl enable lsyncd.service

And that’s about it!