Contents
Rclone
is a command line tool used to mount and synchronise files and directories between a Linux workstation and various cloud-based storage provides like Amazon S3, Dropbox, Google Cloud storage, and many more. In this article we will be using rclone to sync with an AWS S3 bucket.
Installation
You can install rclone by using any of the methods shown here on their website. Here I use the script installation method:
[root@centos7 ~]# curl https://rclone.org/install.sh | sudo bashHopefully the output should indicate success.
rclone v1.47.0 has successfully installed. Now run "rclone config" for setup. Check https://rclone.org/docs/ for more details.You can also confirm by running the below command to check the current version number:
[root@centos7 ~]# rclone version rclone v1.47.0 - os/arch: linux/amd64 - go version: go1.12.4The goal here is to mount and synchronise an AWS S3 bucket to a directory on a CentOS 7 server. You will need to know your AWS security key pair. They should look similar to the below.
Key ID: AKIAPJIFUCTJWDAYF2LZ Secret Key: AcbQW4syws9bL5mVCWqAwys1TpC3BIlZeEQXi+7jArmed with this, configure
rclone
as shown below.[root@centos7 ~]# rclone config 2019/05/28 11:03:08 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n name> aws-s3-bucketHere we type
n
to create a new remote connection and give it the nameaws-s3-bucket
. Next select the type of storage to configure. In my case, I select number4
and then1
for AWS S3:Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / A stackable unification remote, which can appear to merge the contents of several remotes \ "union" 2 / Alias for a existing remote \ "alias" 3 / Amazon Drive \ "amazon cloud drive" 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc) \ "s3" 5 / Backblaze B2 \ "b2" 6 / Box \ "box" 7 / Cache a remote \ "cache" 8 / Dropbox \ "dropbox" 9 / Encrypt/Decrypt a remote \ "crypt" 10 / FTP Connection \ "ftp" 11 / Google Cloud Storage (this is not Google Drive) \ "google cloud storage" 12 / Google Drive \ "drive" 13 / Hubic \ "hubic" 14 / JottaCloud \ "jottacloud" 15 / Koofr \ "koofr" 16 / Local Disk \ "local" 17 / Mega \ "mega" 18 / Microsoft Azure Blob Storage \ "azureblob" 19 / Microsoft OneDrive \ "onedrive" 20 / OpenDrive \ "opendrive" 21 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH) \ "swift" 22 / Pcloud \ "pcloud" 23 / QingCloud Object Storage \ "qingstor" 24 / SSH/SFTP Connection \ "sftp" 25 / Webdav \ "webdav" 26 / Yandex Disk \ "yandex" 27 / http Connection \ "http" Storage> 4 ** See help for s3 backend at: https://rclone.org/s3/ ** Choose your S3 provider. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Amazon Web Services (AWS) S3 \ "AWS" 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun \ "Alibaba" 3 / Ceph Object Storage \ "Ceph" 4 / Digital Ocean Spaces \ "DigitalOcean" 5 / Dreamhost DreamObjects \ "Dreamhost" 6 / IBM COS S3 \ "IBMCOS" 7 / Minio Object Storage \ "Minio" 8 / Netease Object Storage (NOS) \ "Netease" 9 / Wasabi Object Storage \ "Wasabi" 10 / Any other S3 compatible provider \ "Other" provider> 1Here we choose to enter our credentials manually by selecting number
1 from the options.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank. Enter a boolean value (true or false). Press Enter for the default ("false"). Choose a number from below, or type in your own value 1 / Enter AWS credentials in the next step \ "false" 2 / Get AWS credentials from the environment (env vars or IAM) \ "true" env_auth> 1 AWS Access Key ID. Leave blank for anonymous access or runtime credentials. Enter a string value. Press Enter for the default (""). access_key_id> AKIAPJIFUCTJWDAYF2LZ AWS Secret Access Key (password) Leave blank for anonymous access or runtime credentials. Enter a string value. Press Enter for the default (""). secret_access_key> AcbQW4syws9bL5mVCWqAwys1TpC3BIlZeEQXi+7jNext select the region to connect to. I select
London here.
Region to connect to. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value / The default endpoint - a good choice if you are unsure. 1 | US Region, Northern Virginia or Pacific Northwest. | Leave location constraint empty. \ "us-east-1" / US East (Ohio) Region 2 | Needs location constraint us-east-2. \ "us-east-2" / US West (Oregon) Region 3 | Needs location constraint us-west-2. \ "us-west-2" / US West (Northern California) Region 4 | Needs location constraint us-west-1. \ "us-west-1" / Canada (Central) Region 5 | Needs location constraint ca-central-1. \ "ca-central-1" / EU (Ireland) Region 6 | Needs location constraint EU or eu-west-1. \ "eu-west-1" / EU (London) Region 7 | Needs location constraint eu-west-2. \ "eu-west-2" / EU (Stockholm) Region 8 | Needs location constraint eu-north-1. \ "eu-north-1" / EU (Frankfurt) Region 9 | Needs location constraint eu-central-1. \ "eu-central-1" / Asia Pacific (Singapore) Region 10 | Needs location constraint ap-southeast-1. \ "ap-southeast-1" / Asia Pacific (Sydney) Region 11 | Needs location constraint ap-southeast-2. \ "ap-southeast-2" / Asia Pacific (Tokyo) Region 12 | Needs location constraint ap-northeast-1. \ "ap-northeast-1" / Asia Pacific (Seoul) 13 | Needs location constraint ap-northeast-2. \ "ap-northeast-2" / Asia Pacific (Mumbai) 14 | Needs location constraint ap-south-1. \ "ap-south-1" / South America (Sao Paulo) Region 15 | Needs location constraint sa-east-1. \ "sa-east-1" region> 7Leave the endpoint for S3 API blank.
Endpoint for S3 API. Leave blank if using AWS to use the default endpoint for the region. Enter a string value. Press Enter for the default (""). endpoint>Set the location constraint to match the region.
Location constraint - must be set to match the Region. Used when creating buckets only. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Empty for US Region, Northern Virginia or Pacific Northwest. \ "" 2 / US East (Ohio) Region. \ "us-east-2" 3 / US West (Oregon) Region. \ "us-west-2" 4 / US West (Northern California) Region. \ "us-west-1" 5 / Canada (Central) Region. \ "ca-central-1" 6 / EU (Ireland) Region. \ "eu-west-1" 7 / EU (London) Region. \ "eu-west-2" 8 / EU (Stockholm) Region. \ "eu-north-1" 9 / EU Region. \ "EU" 10 / Asia Pacific (Singapore) Region. \ "ap-southeast-1" 11 / Asia Pacific (Sydney) Region. \ "ap-southeast-2" 12 / Asia Pacific (Tokyo) Region. \ "ap-northeast-1" 13 / Asia Pacific (Seoul) \ "ap-northeast-2" 14 / Asia Pacific (Mumbai) \ "ap-south-1" 15 / South America (Sao Paulo) Region. \ "sa-east-1" location_constraint> 7Select number
1, owner gets full control from the ACL list of options.
Canned ACL used when creating buckets and storing or copying objects. This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too. For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl Note that this ACL is applied when server side copying objects as S3 doesn't copy the ACL from the source but rather writes a fresh one. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Owner gets FULL_CONTROL. No one else has access rights (default). \ "private" 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. \ "public-read" / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 3 | Granting this on a bucket is generally not recommended. \ "public-read-write" 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. \ "authenticated-read" / Object owner gets FULL_CONTROL. Bucket owner gets READ access. 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. \ "bucket-owner-read" / Both the object owner and the bucket owner get FULL_CONTROL over the object. 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. \ "bucket-owner-full-control" acl> 1Select
none for the server-side encryption.
The server-side encryption algorithm used when storing this object in S3. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / None \ "" 2 / AES256 \ "AES256" 3 / aws:kms \ "aws:kms" server_side_encryption> 1And select
none for the KMS ID question:
If using KMS ID you must provide the ARN of Key. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / None \ "" 2 / arn:aws:kms:* \ "arn:aws:kms:us-east-1:*" sse_kms_key_id> 1Lastly select number
1, the default option for the storage class:
The storage class to use when storing new objects in S3. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Default \ "" 2 / Standard storage class \ "STANDARD" 3 / Reduced redundancy storage class \ "REDUCED_REDUNDANCY" 4 / Standard Infrequent Access storage class \ "STANDARD_IA" 5 / One Zone Infrequent Access storage class \ "ONEZONE_IA" 6 / Glacier storage class \ "GLACIER" 7 / Glacier Deep Archive storage class \ "DEEP_ARCHIVE" storage_class> 1If you don’t need to make any changes, select no below.
Edit advanced config? (y/n) y) Yes n) No y/n> nYou should then see your configuration printed to screen.
Remote config -------------------- [aws-s3-bucket] type = s3 provider = AWS env_auth = false access_key_id = AKIAPJIFUCTJWDAYF2LZ secret_access_key = AcbQW4syws9bL5mVCWqAwys1TpC3BIlZeEQXi+7j region = eu-west-2 location_constraint = eu-west-2 acl = private -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> yConfirm this looks okay and then quit.
Current remotes: Name Type ==== ==== aws-s3-bucket s3 e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> qUsage
If you created a connection called
aws-s3-bucket
, you can list the available buckets with the below command:View Buckets
[root@centos7 ~]# rclone lsd aws-s3-bucket: -1 2018-05-19 03:35:33 -1 gitlab-application-backup -1 2019-04-01 10:26:24 -1 opnit-uploadsList all Objects
You can then use the
ls
command to list all objects within a bucket:[root@centos7 ~]# rclone ls aws-s3-bucket:gitlab-application-backup 133120 1526701826_2018_05_19_10.7.3_gitlab_backup.tar 133120 1526702266_2018_05_19_10.7.3_gitlab_backup.tar 133120 1526788065_2018_05_20_10.7.3_gitlab_backup.tar 133120 1526837713_2018_05_20_10.7.3_gitlab_backup.tar 133120 1526874335_2018_05_21_10.7.3_gitlab_backup.tar 133120 1526960757_2018_05_22_10.7.3_gitlab_backup.tar 133120 1527047142_2018_05_23_10.7.3_gitlab_backup.tarCreate a new Bucket
To create a new bucket called
pikedom-pics
on theaws-s3-bucket
remote connection, run the below command:[root@centos7 ~]# rclone mkdir aws-s3-bucket:pikedom-picsCopy Files
To copy files from a local directory to a remote bucket.
[root@centos7 ~]# du -sh Pictures/ 50M Pictures/ [root@centos7 ~]# rclone copy Pictures/ aws-s3-bucket:pikedom-picsList all objects again to confirm that worked:
[root@centos7 ~]# rclone ls aws-s3-bucket:pikedom-pics 828928 2018-11-30-105431_5760x1080_scrot.png 718638 2018-12-03-110057_5760x1080_scrot.png 589146 2018-12-04-101917_5760x1080_scrot.png 772318 2018-12-05-141304_5760x1080_scrot.png 104242 2018-12-06-092823_939x1030_scrot.png 134906 2018-12-06-092830_939x1030_scrot.png 117729 2018-12-06-092836_939x1030_scrot.png 132996 2018-12-06-092845_939x1030_scrot.png 114258 2018-12-06-092851_939x1030_scrot.png ...Synchronise Files
Now lets add a new file to the Pictures directory and synchronise the directory to the AWS S3 bucket.
[root@centos7 ~]# touch Pictures/newfile.txtThe above just create a new empty file called
newfile.txt
in thePictures
directory. If you want to see what files would be copied over without actually copying anything, issue the command with--dry-run
appended:[root@centos7 ~]# rclone sync Pictures/ aws-s3-bucket:pikedom-pics --dry-run 2019/05/28 13:52:20 NOTICE: newfile.txt: Not copying as --dry-runHere we can see there is just one file that needs to be copied over. Remove the
--dry-run
option to actually copy the files.[root@centos7 ~]# rclone sync Pictures/ aws-s3-bucket:pikedom-picsLocal Mounts
To mount an S3 bucket to a local directory, issue the below command. For some reason the above command does not work for me without appending the ampersand (&).
[root@centos7 ~]# mkdir -v local-mount mkdir: created directory ‘local-mount’ [root@centos7 ~]# rclone mount aws-s3-bucket:gitlab-application-backup local-mount/ & [2] 6084 [root@centos7 ~]# ls -la local-mount/ total 3640 -rw-r--r--. 1 root root 133120 May 19 2018 1526701826_2018_05_19_10.7.3_gitlab_backup.tar -rw-r--r--. 1 root root 133120 May 19 2018 1526702266_2018_05_19_10.7.3_gitlab_backup.tar -rw-r--r--. 1 root root 133120 May 20 2018 1526788065_2018_05_20_10.7.3_gitlab_backup.tar -rw-r--r--. 1 root root 133120 May 20 2018 1526837713_2018_05_20_10.7.3_gitlab_backup.tar -rw-r--r--. 1 root root 133120 May 21 2018 1526874335_2018_05_21_10.7.3_gitlab_backup.tar -rw-r--r--. 1 root root 133120 May 22 2018 1526960757_2018_05_22_10.7.3_gitlab_backup.tar -rw-r--r--. 1 root root 133120 May 23 2018 1527047142_2018_05_23_10.7.3_gitlab_backup.tar -rw-r--r--. 1 root root 133120 May 24 2018 1527133543_2018_05_24_10.7.3_gitlab_backup.tar -rw-r--r--. 1 root root 133120 May 25 2018 1527219943_2018_05_25_10.7.3_gitlab_backup.tar ...A normal
umount
should unmount the bucket:[root@centos7 ~]# umount -v local-mount/ umount: /root/local-mount (aws-s3-bucket:gitlab-application-backup) unmounted [2]+ Done rclone mount aws-s3-bucket:gitlab-application-backup local-mount/ [root@centos7 ~]#If the above fails, try the following:
[root@centos7 ~]# fusermount -u mount/ [root@centos7 ~]# ls -la mount/ total 8 drwxr-xr-x. 2 root root 4096 May 28 14:41 . dr-xr-x---. 9 root root 4096 May 28 14:41 .. [1]+ Done rclone mount aws-s3-bucket:gitlab-application-backup mount/Permanently Mount
Making a mount persistent over a reboot is not quite as straightforward as was taken from jamescoyle.net.
First create a new systemd service file.
[root@centos7 ~]# vim /etc/systemd/system/rclone.servicePopulate with the following:
# /etc/systemd/system/rclone.service [Unit] Description=Google Drive (rclone) AssertPathIsDirectory=/root/local-mount After=plexdrive.service [Service] Type=simple ExecStart=/usr/bin/rclone mount \ --config=/root/.config/rclone/rclone.conf \ --allow-other \ --cache-tmp-upload-path=/tmp/rclone/upload \ --cache-chunk-path=/tmp/rclone/chunks \ --cache-workers=8 \ --cache-writes \ --cache-dir=/tmp/rclone/vfs \ --cache-db-path=/tmp/rclone/db \ --no-modtime \ --drive-use-trash \ --stats=0 \ --checkers=16 \ --bwlimit=40M \ --dir-cache-time=60m \ --cache-info-age=60m aws-s3-bucket:gitlab-application-backup /root/local-mount ExecStop=/bin/fusermount -u /root/local-mount Restart=always RestartSec=10 [Install] WantedBy=default.targetReplace all instances of
/root/local-mount
with the location you want to mount to. Also replaceaws-s3-bucket:gitlab-application-backup
with the remote connection name you created earlier and the bucket name you wish to mount.If you now start the process, you should see the directory is now populated. If not, amend the script and reload with:
[root@centos7 ~]# systemctl daemon-reloadIf that worked once started, you will need to
enable
the service for it to start on reboot.[root@centos7 ~]# systemctl enable rclone.service
Be the first to comment