The “rsync” is a powerful command under the Linux environment.
The rsync stands for Remote Sync. Normally rsync is used to transfer file from one server(source) to another server(destination).
rsync has a lot of switches or option for managing the command much usefully. Here I am explaining the switches of rsync command with examples.
Rsync will actually look and see what in the file has changed and upload only the part of the file that has changed. Unlike ftp and other transfer solutions rsync doesn’t simply re-upload the entire file.
Common Syntax for Rsync:
# rsync [options] Source Destinations.
Switches of rsync:
1, -v, –verbose
I’ve created a file ‘rsync’ under ‘/root/Rsync/’ for testing purpose.
root@server [~]# rsync -v /root/Rsync/rsync . rsync sent 65 bytes received 31 bytes 192.00 bytes/sec total size is 0 speedup is 0.00
2, -r, –recursive
Recurse into directories.
3, -l, –links
Copy symlinks as symlinks.
4, -p, –perms
5, -t, –times
Preserve modification times.
6, -g, –group
7, -o, –owner
preserve owner (super-user only)
Same as –devices –specials.
–devices : preserve device files (super-user only).
–specials : preserve special files.
9, -H, –hard-links
Preserve hard links
10, -A, –acls
Preserve ACLs (implies –perms)
11, -X, –xattrs
Preserve extended attributes
12, -a, –archive
This is very important rsync switch, because it can be done the functions of some other switches combinations.
Archive mode; equals -rlptgoD (no -H,-A,-X)
13, -q, –quiet
Suppress non-error messages.
14, To specify the file size for sync:
Don’t transfer any file larger than SIZE
Don’t transfer any file smaller than SIZE
Delete extraneous files from destination dirs.
16, W, –whole-file
Copy files whole (without delta-xfer algorithm)
17, -u, –update
Skip files that are newer on the receiver. Means, Do Not Overwrite the Modified Files at the Destination.
View the rsync Progress during Transfer.
19, Include and Exclude Pattern.
Patterns are expressed in single quote.
[root@ser crybit]# rsync -avz --include 'c*' --exclude '*' /root/crybit/ 188.8.131.52:/root/Rsinc101 [email protected]'s password: stdin: is not a tty sending incremental file list cry1.doc cry2.doc cry3.doc cry4.doc cry5.doc sent 267 bytes received 107 bytes 68.00 bytes/sec total size is 0 speedup is 0.00
In the above example files starting with ‘c’ are included for Rsync and all other files are excluded on Rsync operation.
20 -e : Rsync over Shell(SSH)
# rsync -avz -e ssh Source Destination
Some useful switches combinations and Useful “Rsync” examples:
If the destination server has a different SSH port, you need to know this command to connect and Rsync files from that server. More details
1, How to Sync two directories in the same machine ?
I have created two directories(also some files inside each dir) under /root location to check the rsync usages.
root@server [~]# rsync -zvr /root/Rsync1/ /root/Rsync2/ sending incremental file list rsync1.txt rsync10.txt rsync2.txt rsync3.txt ...... ...... sent 502 bytes received 202 bytes 1408.00 bytes/sec total size is 0 speedup is 0.00
-z is to enable compression
-r indicates recursive
2, Preserve the following option while Sync files using -a switch.
r, l, p, t, g, o, D : Switch details are mentioned in the above section. Recursive mode, symbolic links, permissions, timestamp, owner and group
See the example below:
root@server [~]# ll /root/Rsync1/rsync10.txt -rw-r--r-- 1 root root 0 Feb 17 07:40 /root/Rsync1/rsync10.txt root@server [~]# ll /root/Rsync2/rsync10.txt -rw-r--r-- 1 root root 0 Feb 17 07:40 /root/Rsync2/rsync10.txt
3, How to Rsync files from remote server to local ?
It is quit similar to SCP, the syntax for doing the same is pasted below:
# rsync -azv Remote-IP:/path/for/Sync /Path/to/Sync
Sync the directory from remote to local
[root@ser crybit]# rsync -azv MY.SERVER.IP:/root/Rsync1 /root/crybit/ [email protected]'s password: stdin: is not a tty receiving incremental file list Rsync1/ Rsync1/rsync1.txt Rsync1/rsync10.txt Rsync1/rsync2.txt Rsync1/rsync3.txt ..... ..... sent 205 bytes received 527 bytes 77.05 bytes/sec total size is 0 speedup is 0.00
Sync all files from remote to local
[root@ser crybit]# rsync -azv 184.108.40.206:/root/Rsync1/* /root/crybit/ [email protected]'s password: stdin: is not a tty receiving incremental file list rsync1.txt rsync10.txt rsync2.txt rsync3.txt ... ... sent 201 bytes received 499 bytes 93.33 bytes/sec total size is 0 speedup is 0.00
4, How to sync files from local to remote server ?
# rsync -azv /local-path/for/Sync/ Remote-IP:/path/to/Sync
[root@support crybit]# rsync -avz /root/crybit/ 220.127.116.11:/root/Rsinc101 [email protected]'s password: stdin: is not a tty sending incremental file list created directory /root/Rsinc101 ./ rsync1.txt rsync10.txt rsync2.txt rsync3.txt ... ... sent 1028 bytes received 399 bytes 190.27 bytes/sec total size is 0 speedup is 0.00
Note 1 : In example 3 and 4 you can use user@IP:/path format, if you have only the user privilege on server Note 2 : The rsync will automatically create the destination directory/location if it isn't there.
That’s it !! 🙂
Requirement: Need to write a script to create Mongo DB collections dump and move the dump to an AWS s3 bucket.
Prerequisites: SSH access to Mongo DB server, IAM user with AWS s3 full [or write] access, aws-cli on server, knowledge in Mongo commands for dump creation.
As we need to move the dump to an S3 bucket, first we need to configure IAM user. Then only we can move the dump to S3 bucket. To configure IAM, you need to install aws-cli tool on the machine.