Skip to content

Commit

Permalink
Changes mirroring_lsyncd.md (#2403)
Browse files Browse the repository at this point in the history 
* replace passive voice with active
* simplify many sentences
* change some wording
* replace contractions with words
* consistently apply fenced code blocks and contexts
  • Loading branch information
@sspencerwire
sspencerwire authored Oct 1, 2024
1 parent a3ae1a5 commit 199d5d9
Showing 1 changed file with 115 additions and 89 deletions.
204 changes: 115 additions & 89 deletions docs/guides/backup/mirroring_lsyncd.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,64 +8,67 @@ tags:
- synchronization
- mirroring
---
# Mirroring Solution - `lsyncd`

## Prerequisites

This is everything you'll need to understand and follow along with this guide:

- A computer running Rocky Linux
- A comfort level with modifying configuration files from the command-line
- Knowledge of how to use a command line editor (we use vi here, but you could use your favorite editor)
- You will need root access, and ideally be signed in as the root user in your terminal
- Knowledge of how to use a command line editor (using `vi` here, but use your favorite editor)
- You will need root access or `sudo` privileges (using `sudo -s` from the start is a good idea)
- Public and Private SSH key pairs
- The EPEL repositories from Fedora
- The EPEL (Extra Packages for Enterprise Linux) repositories from Fedora
- You will need to be familiar with *inotify*, an event monitor interface
- Optional: familiarity with *tail*

## Introduction

If you're looking for a way to synchronize files and folders between computers automatically, `lsyncd` is a pretty great option. The only downside for beginners? You have to configure everything at the command line, and text files.
If you are looking for a way to synchronize files and folders between computers automatically, `lsyncd` is a great option. But, you must configure everything from the command line.

Even so, it is a program worth learning for any sysadmin.
It is a program worth learning for any system administrator.

The best description of `lsyncd`, comes from its own man page. Slightly paraphrased, `lsyncd` is a light-weight live mirror solution that is not hard to install. It does not require new filesystems or block devices, and does not hamper local filesystem performance. In short, it mirrors files.
The best description of `lsyncd`, comes from its own man page. Slightly paraphrased, `lsyncd` is a light-weight live mirror solution that is not hard to install. It does not require new file systems or block devices, and does not hamper local file system performance. In short, it mirrors files.

`lsyncd` watches a local directory tree's event monitor interface (inotify). It aggregates and combines events for a few seconds, and then spawns one (or more) process(es) to synchronize the changes. By default this is `rsync`.
`lsyncd` watches a local directory tree's event monitor interface (inotify). It aggregates and combines events for a few seconds, and spawns one (or more) process(es) to synchronize the changes. By default this is `rsync`.

For the purposes of this guide, you will call the system with the original files the "source", and the one that we are synchronizing to will be the "target". It is actually possible to completely mirror a server using `lsyncd` by very carefully specifying directories and files that you want to synchronize. It's pretty sweet!
For the purposes of this guide, you will call the system with the original files the "source", and the one that you are synchronizing to will be the "target". It is actually possible to completely mirror a server using `lsyncd` by carefully specifying directories and files that you want to synchronize.

For remote syncing, you will also want to set up [Rocky Linux SSH Public Private Key Pairs](../security/ssh_public_private_keys.md). The examples here use SSH (port 22).

## Installing `lsyncd`

There are actually two ways to install `lsyncd`. We will include them both here. The RPM tends to lag behind the source packages by a little, but only by a little. The version installed by the RPM method at the time of this writing is 2.2.2-9, whereas the source code version is now 2.2.3. That said, we want to give you both options and let you choose.

## Installing `lsyncd` - RPM Method
Installing `lsyncd` happens in one of two ways. Included are descriptions of each method. The RPM tends to lag behind the source packages by a little, but only by a little. The version installed by the RPM method at the time of this writing is 2.2.3-5, whereas the source code version is now 2.3.1. Choose the method you feel the most comfortable with.

Installing the RPM version is not hard. The only thing you will need to install first is the EPEL software repository from Fedora. This can be done with a single command:
## Installing `lsyncd` - RPM method

`dnf install -y epel-release`
Installing the RPM version is not hard. The only thing you will need to install first is the EPEL software repository from Fedora. Do this with:

Then, we just need to install `lsyncd` and any missing dependencies will be installed along with it:
```bash
dnf install -y epel-release
```

`dnf install lsyncd`
Now install `lsyncd` along with any missing dependencies:

Set up the service to start on boot, but don't start it just yet:
```bash
dnf install lsyncd
```

`systemctl enable lsyncd`
Set up the service to start on boot, but do not start it just yet:

That's it!
```bash
systemctl enable lsyncd
```

## Installing `lsyncd` - Source Method
## Installing `lsyncd` - source method

Installing from the source is not as bad as it sounds. Just follow this guide and you will be up and running in no time!
Installing from the source is not difficult.

### Install Dependencies
### Install dependencies

We will need some dependencies: a few that are required by `lsyncd` itself, and a few that are required to build packages from source. Use this command on your Rocky Linux computer to ensure you have the dependencies you need. If you are going to be building from source, it is a good idea to have all of the development tools installed:
You will need some dependencies for `lsyncd` and for building packages from source. Use this command on your Rocky Linux computer to ensure you have the dependencies you need. If you are going to be building from source, it is a good idea to have all of the development tools installed:

`dnf groupinstall 'Development Tools'`
```bash
dnf groupinstall 'Development Tools'
```

!!! warning "For Rocky Linux 9.0"

Expand All @@ -75,57 +78,69 @@ We will need some dependencies: a few that are required by `lsyncd` itself, and
dnf config-manager --enable crb
```

Doing this in 9 before the next steps, will allow you to finish the build without backtracking.
Doing this in 9 before the next steps will allow you to finish the build without backtracking.

And here are the dependencies we need for `lsyncd` itself, and its build process:
Here are the dependencies needed for `lsyncd`:

`dnf install lua lua-libs lua-devel cmake unzip wget rsync`
```bash
dnf install lua lua-libs lua-devel cmake unzip wget rsync
```

### Download `lsyncd` And Build It
### Download `lsyncd` and build

Next we need the source code:
Next you need the source code:

`wget https://github.com/axkibe/lsyncd/archive/master.zip`
```bash
wget https://github.com/axkibe/lsyncd/archive/master.zip
```

Now uncompress the master.zip file:
Decompress the `master.zip` file:

`unzip master.zip`

This will create a directory called "lsyncd-master". We need to change to this directory and create a directory called build:
This will create a directory called "lsyncd-master". You need to change to this directory and create a directory called build:

`cd lsyncd-master`
```bash
cd lsyncd-master
```

And then:
Then:

`mkdir build`
```bash
mkdir build
```

Now change directories again so that you are in the build directory:
Change directories to access the build directory:

`cd build`
```bash
cd build
```

Now run these commands:
Run these commands:

```bash
cmake ..
make
make install
```

When done, you will have the `lsyncd` binary installed and ready for use in */usr/local/bin*
When completed, you will have the `lsyncd` binary installed and ready for use in */usr/local/bin*

### `lsyncd` Systemd Service
### `lsyncd` systemd service

With the RPM install method, the systemd service will be installed for you, but if you choose to install from source, you will need to create the systemd service. While you can start the binary without the systemd service, we want to ensure that it *does* start on boot. If not, a server reboot would stop your synchronization effort. If you forgot to start it again, which is highly likely, that would be a problem for any systems administrator!
With the RPM install method, the systemd service will install for you, but if you install from source, you will need to create the systemd service. While you can start the binary without the systemd service, you want to ensure that it *does* start on boot. If not, a server reboot will stop your synchronization effort. If you forgot to manually start it again, it would be a problem!

Creating the systemd service is not terribly difficult, though, and will save you time in the long run.
Creating the systemd service is not terribly difficult, and will save you time in the long run.

#### Create The `lsyncd` Service File
#### Create the `lsyncd` service file

This file can be created anywhere, even in the root directory of your server. When created, you can move it to the right location.
Create this file anywhere, even in the root directory of your server. When created, you can move it to the right location.

`vi /root/lsyncd.service`
```bash
vi /root/lsyncd.service`
```

The contents of this file should be:
The contents of this file will be:

```bash
[Unit]
Expand All @@ -144,21 +159,25 @@ PIDFile=/run/lsyncd.pid
WantedBy=multi-user.target
```

Now let's install the file you just made to the correct location:
Install the file you just made to the correct location:

`install -Dm0644 /root/lsyncd.service /usr/lib/systemd/system/lsyncd.service`
```bash
install -Dm0644 /root/lsyncd.service /usr/lib/systemd/system/lsyncd.service
```

Finally, reload the `systemctl` daemon so that systemd will "see" the new service file:

`systemctl daemon-reload`
```bash
systemctl daemon-reload
```

## `lsyncd` Configuration
## `lsyncd` configuration

Whichever method you choose for installing `lsyncd`, you will need a configuration file: */etc/lsyncd.conf*. The next section will tell you how to build a configuration file, and test it.
With either `lsyncd` install method, you will need a configuration file: */etc/lsyncd.conf*. The next section will tell you how to build a configuration file, and test it.

## Sample Configuration For Testing
### Sample configuration for testing

Here's an example of a simplistic configuration file that synchronizes */home* to another computer. Our target computer is going to be a local IP address: *192.168.1.40*
Here is an example of a simplistic configuration file that synchronizes */home* to another computer. Our target computer is going to be a local IP address: *192.168.1.40*

```bash
settings {
Expand Down Expand Up @@ -187,26 +206,28 @@ sync {

Breaking down this file a bit:

- The `logfile` and `statusFile` will be automatically created when the service starts.
- The `statusInterval` is the number of seconds to wait before writing to the statusFile.
- `maxProcesses` is the number of processes `lsyncd` is allowed to spawn. Honestly, unless you are running this on a super busy computer, 1 process is enough.
- In the sync section `default.rsyncssh` says to use rsync over SSH
- The `source=` is the directory path we are syncing from.
- The `host=` is our target computer that we are syncing to.
- The `logfile` and `statusFile` will be automatically created when the service starts
- The `statusInterval` is the number of seconds to wait before writing to the status file
- `maxProcesses` is the number of `lsyncd` processes allowed to spawn. Unless you are running this on a busy computer, 1 process is enough.
- In the sync section `default.rsyncssh` says to use `rsync` over SSH
- The `source=` is the directory path you are syncing from
- The `host=` is our target computer that you are syncing to
- The `excludeFrom=` tells `lsyncd` where the exclusions file is. It must exist, but can be empty.
- The `targetdir=` is the target directory we are sending files to. In most cases this will be equal to the source, but not always.
- Then we have the `rsync =` section, and these are the options that we are running rsync with.
- Finally we have the `ssh =` section, and this specifies the SSH port that is listening on the target computer.
- The `targetdir=` is the target directory you are sending files to. In most cases this will be equal to the source, but not always.
- The `rsync =` section, and the options that you are running `rsync` with
- The `ssh =` section, specifying the SSH port that is listening on the target computer

If you are adding more than one directory to sync, then you need to repeat the entire "sync" section including all the opening and closing brackets for each directory.
If you are adding more than one directory to sync, you need to repeat the entire "sync" section including all the opening and closing brackets for each directory.

## The lsyncd.exclude File
## The lsyncd.exclude file

As noted earlier, the `excludeFrom` file must exist, so let's create that now:
As noted earlier, the `excludeFrom` file must exist. Create that now:

`touch /etc/lsyncd.exclude`
```bash
touch /etc/lsyncd.exclude
```

If you were syncing the `/etc` folder on our computer, there would be many files and directories that you should exclude. Each excluded file or directory is listed in the file, one per line, like this:
As an example, if you were syncing the `/etc` folder on our computer there would be many files and directories that you want to exclude in the `lsyncd` process. Each excluded file or directory is in the file, one per line:

```bash
/etc/hostname
Expand All @@ -215,42 +236,47 @@ If you were syncing the `/etc` folder on our computer, there would be many files
/etc/fstab
```

## Test And Turn Up
## Test And turn Up

Now that everything else is set up, you can test it all. For starters, lets ensure our systemd lsyncd.service will start:
With everything set up, you can test it all. Ensure our systemd `lsyncd.service` will start:

`systemctl start lsyncd`
```bash
systemctl start lsyncd
```

If no errors appear after executing this command, check the status of the service, just to ensure:

`systemctl status lsyncd`
```bash
systemctl status lsyncd
```

If it shows the service running, use tail to see the ends of the two log files, and ensure everything shows up OK:

`tail /var/log/lsyncd.log`

And then:

`tail /var/log/lsyncd-status.log`

Assuming that this all looks correct, navigate to the `/home/[user]` directory, where `[user]` is a user on the computer and create a new file there with *touch*.
```bash
tail /var/log/lsyncd.log
tail /var/log/lsyncd-status.log
```

`touch /home/[user]/testfile`
Assuming that this all looks correct, navigate to the `/home/[user]` directory, where `[user]` is a user on the computer and create a file there with *touch*.

Now go to the target computer and see if the file shows up. If so, everything is working as it should. Set the lsyncd.service to start on boot with:
```bash
touch /home/[user]/testfile
```

`systemctl enable lsyncd`
Go to the target computer and see if the file shows up. If so, everything is working. Set the `lsyncd.service` to start on boot with:

And you are ready to go.
```bash
systemctl enable lsyncd
```

## Remember To Be Careful
## Remember to be careful

Anytime you are synchronizing a set of files or directories to another computer, think carefully about the effect it will have on the target computer. If you go back to **The lsyncd.exclude File** in our example above, can you imagine what might happen if */etc/fstab* is synchronized?
Anytime you are synchronizing a set of files or directories to another computer, think carefully about the effect it will have on the target computer. If you go back to **The lsyncd.exclude File** in our example above, can you imagine what might happen if you failed to exclude */etc/fstab*?

For newbies, *fstab* is the file that is used to configure storage drives on any Linux computer. The disks and labels are almost certainly different. The next time the target computer was rebooted it would likely fail to boot entirely.
`fstab` is the file used to configure storage drives on any Linux computer. The disks and labels are almost certainly different on different machines. The next reboot of the target computer would likely fail entirely.

## Conclusions And References
## Conclusions and references

`lsyncd` is a powerful tool for directory synchronization between computers. As you've seen, it is not hard to install, and it is not complex to maintain going forward. You can not ask for more than that.
`lsyncd` is a powerful tool for directory synchronization between computers. As you have seen, it is not hard to install, and it is not complex to maintain going forward.

You can find out more about `lsyncd` by going to [The Official Site](https://github.com/axkibe/lsyncd)

0 comments on commit 199d5d9

Please sign in to comment.