rsync command examples

Rsync(Remote Sync) is one of the most efficient and secure data copying tool in Linux and Unix world. Rather than just copying, it can be called as synchronization tool. Whenever rsync is used to copy the files, it does it like a copy for the first time, but during its successive iterations, it will synchronize the files by just copying the delta change between them by using the remote-update protocol. And this synchronization can be either between the files on the local system or on the local system and remote system. Mainly between the local and the remote, as rsync generally termed for “Remote Synchronization”.

NOTE: The rsync command cannot be used to copy the files between the two remote systems. Either you can copy data from local machine to remote machine or remote machine to a local one.
In this post, we will see some examples of rsync along with some rarely known options that work wonders in the Linux data administration.

Linux/Unix command syntax for ‘rsync’

rsync [options] [SRC] [DEST]

Rsync can be used either in the push mode or a pull mode when the remote system is in scope. The basic syntax would remain same:

For pushing the files to a remote system using Rsync

rsync [options] [SRC] user@host:[DEST]

Pulling the files from the remote system using rsync

rsync [options] user@host:[SRC] [DEST]

Let’s jump onto some basic use cases for rsync :

Example 1:  We can use rsync as a replacement to cp command to copy files between the directories on the same system. Suppose if we want to copy all the files in /etc whose name contain serv use below command.

root@sujitkumar:~# pwd
 /root
 root@sujitkumar:~# ls
 root@sujitkumar:~# rsync /etc/*serv* ~
 skipping directory insserv
 skipping directory insserv.conf.d
 root@sujitkumar:~# ls
 insserv.conf  services
 root@sujitkumar:~#

Example 2: Usually we want to copy and synchronize the directories across the systems.

rsync /path/*file* user@host:/home/

Example:

root@sujitkumar:~# rsync -rv /etc/default sujit@10.10.1.125:/home/sujit/
 sujit@10.10.1.125's password:
 sending incremental file list
 default/
 default/acpid
 default/bootlogd
 default/console-setup
 default/crda
 default/cron
 default/dbus
 default/devpts
 default/ntfs-3g
 default/ntpdate
 default/puppetmaster
 sent 19620 bytes  received 529 bytes  3663.45 bytes/sec
 total size is 17982  speedup is 0.89
 root@sujitkumar:~#

The ‘r’ option copies the ‘foo’ directory in a recursive manner into ‘/home/user’ directory.
I will use the rsync between the local and the remote system, as that is required in the majority of cases. And since we are copying the directories, we need to do it recursively. Verbose with any command informs about the command execution, hence we can use ‘-v’ option for detailed output flow.

Note: The permissions, ownership, and groups are not retained for the rsynced files and directories on the remote system. Files owned by root will be owned by ‘sujit’ with default permissions for the discussed example.

Example 3: Archive files once copied.

root@sujitkumar:~# rsync -av /etc/default/ sujit@10.10.1.125:/home/sujit/

The ‘a’ option is for archive, this will synchronize the contents of ‘default’ directory recursively, preserving symbolic links, special and device files, modification times, group, owner, and permissions.
Note:1 Here I have one interesting point to make, trailing slash ‘/’ after ‘default’, synchronizes only the contents of default directory, without it the directory ‘default’ along with its contents would be synchronized on the remote system.
Note:2 The owner and groups are retained in the remote user is a super-user, and the user and group exist there.

Example 4: Sometimes you want to do a dry-run before you actually run the rsync, so that you know what data will be transferred during the execution, so we use ‘n’ and ‘v’ options.

Example:

root@sujitkumar:~# rsync -nv /etc/default sujit@10.10.1.125:/home/sujit/
 sujit@10.10.1.125's password:
 skipping directory default
 sent 8 bytes  received 12 bytes  4.44 bytes/sec
 total size is 0  speedup is 0.00 (DRY RUN)
 root@sujitkumar:~#

‘n’  option is used to do a dry run, in order to understand what the dry run does, we use the ‘v’ – verbose option that provides the information of what the command executes.

Example 5: For tracking the progress during the rsync data transfer, we use ‘p’ option and can be used as below:

root@sujitkumar:~# rsync -avP /etc/default sujit@10.10.1.125:/home/sujit/
sujit@10.10.1.125's password:
sending incremental file list
default/
default/acpid
346 100%    0.00kB/s    0:00:00 (xfer#1, to-check=26/28)
default/apache2
637 100%  622.07kB/s    0:00:00 (xfer#2, to-check=25/28)
default/apport
149 100%  145.51kB/s    0:00:00 (xfer#3, to-check=24/28)
default/bootlogd
47 100%   22.95kB/s    0:00:00 (xfer#4, to-check=23/28)
default/console-setup
2025 100%  988.77kB/s    0:00:00 (xfer#5, to-check=22/28)
default/crda
549 100%  268.07kB/s    0:00:00 (xfer#6, to-check=21/28)
default/cron
183 100%   89.36kB/s    0:00:00 (xfer#7, to-check=20/28)
...............
default/ssh
133 100%   25.98kB/s    0:00:00 (xfer#23, to-check=4/28)
default/tomcat7
1960 100%  382.81kB/s    0:00:00 (xfer#24, to-check=3/28)
default/ufw
2072 100%  404.69kB/s    0:00:00 (xfer#25, to-check=2/28)
default/useradd
1118 100%  218.36kB/s    0:00:00 (xfer#26, to-check=1/28)
default/whoopsie
31 100%    5.05kB/s    0:00:00 (xfer#27, to-check=0/28)
sent 19622 bytes  received 529 bytes  3663.82 bytes/sec
total size is 17982  speedup is 0.89
root@sujitkumar:~#

Example 6: Sometimes along with archive, compression is expected for faster data transfers. For compression based transfers we use ‘-z’ option along with archive option for faster operation along with the preservation of the directory/files attributes on the source.

root@sujitkumar:~# rsync -azvP /etc/default sujit@10.10.1.125:/home/sujit/
sujit@10.10.1.125's password:
sending incremental file list
default/
default/acpid
346 100%    0.00kB/s    0:00:00 (xfer#1, to-check=26/28)
default/apache2
637 100%  622.07kB/s    0:00:00 (xfer#2, to-check=25/28)
default/apport
149 100%   72.75kB/s    0:00:00 (xfer#3, to-check=24/28)
default/bootlogd
47 100%   22.95kB/s    0:00:00 (xfer#4, to-check=23/28)
default/console-setup
2025 100%  988.77kB/s    0:00:00 (xfer#5, to-check=22/28)
default/crda
549 100%  178.71kB/s    0:00:00 (xfer#6, to-check=21/28)
default/cron
183 100%   59.57kB/s    0:00:00 (xfer#7, to-check=20/28)
default/dbus
297 100%   96.68kB/s    0:00:00 (xfer#8, to-check=19/28)
default/devpts
92 100%   29.95kB/s    0:00:00 (xfer#9, to-check=18/28)
default/grub
1225 100%  398.76kB/s    0:00:00 (xfer#10, to-check=17/28)
default/halt
86 100%   27.99kB/s    0:00:00 (xfer#11, to-check=16/28)
default/irqbalance
126 100%   41.02kB/s    0:00:00 (xfer#12, to-check=15/28)
default/keyboard
559 100%  181.97kB/s    0:00:00 (xfer#13, to-check=14/28)
default/locale
19 100%    4.64kB/s    0:00:00 (xfer#14, to-check=13/28)
default/nss
1756 100%  428.71kB/s    0:00:00 (xfer#15, to-check=12/28)
default/ntfs-3g
default/ufw
2072 100%  404.69kB/s    0:00:00 (xfer#25, to-check=2/28)
default/useradd
1118 100%  218.36kB/s    0:00:00 (xfer#26, to-check=1/28)
default/whoopsie
31 100%    6.05kB/s    0:00:00 (xfer#27, to-check=0/28)
sent 11349 bytes  received 529 bytes  2159.64 bytes/sec
total size is 17982  speedup is 1.51
root@sujitkumar:~#

Note: You can just verify the difference in the data bytes and the speedup in example 5 and example 6, which is because of the compression (-z) based data transfer.

Example 7: One of the major use cases of rsync is its support to include and exclude the specific files (also supports the regex) for the data transfer.
Consider that we are interested in transferring only the files with a pattern that starts with ‘puppet’, we can achieve this as below :

root@sujitkumar:~# rsync -avzP  --include='puppet*' --exclude='*' /etc/default/ sujit@10.10.1.125:/home/sujit
sujit@10.10.1.125's password:
sending an incremental file list
puppetmaster
418 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1/3)
puppetqd
875 100%  854.49kB/s    0:00:00 (xfer#2, to-check=0/3)
sent 857 bytes  received 50 bytes  201.56 bytes/sec
total size is 1293  speedup is 1.43
root@sujitkumar:~#

We can also manage all the files to be included or excluded for the data transfer in a file and mention it using ‘–exclude-from’ and ‘–include-from’ options.

Example 8: In case we want only the new files to be synchronized between the directories but the old files already transferred, need not be touched on the destination, then we can use the rsync with ‘u’ option:

root@sujitkumar:~# rsync -avzPu  /etc/default/ sujit@10.10.1.125:/home/sujit

Here rsync would transfer only the new files from the source to the destination without replacing the modified files of the destination server, which is usually an expected case on the live application servers.

Example 9: Rsync for deletion of the files on source or destination.

‘-delete’ option is used with rsync to delete the files/directory that do not exist on the ‘src’ but already exist on the destination so that src and destination are synchronized.

root@sujitkumar:~# rsync -avzP --delete  /etc/default/ sujit@10.10.1.125:/home/sujit
sujit@10.10.1.125's password:
sending an incremental file list
deleting file4
deleting file3
./
puppetmaster
418 100%    0.00kB/s    0:00:00 (xfer#1, to-check=9/28)
sent 729 bytes  received 40 bytes  53.03 bytes/sec
total size is 17982  speedup is 23.38

Example 10: Other use case is that we might also come up with a requirement of deleting the source directory after its synchronization with the target. This can be handled by –remove-source-files option:

root@sujitkumar:~# rsync  --remove-source-files -avzP --delete  /etc/default/

Example 11: Though rsync transfers the only the incremental data, if bandwidth is not the matter of concern for the data transfer, the ‘-W’ option can be used to transfer the complete data from source to destination. This would generally speeds-up the rsync process as there is no “checksum’ comparison required for the same.

root@sujitkumar:~# rsync -avzP  /etc/default/ sujit@10.10.1.125:/home/sujit

Example 12: All the examples above are highlighted with the push operation and using the remote ssh shell, rsync can be used in the  pull operation from the destination server even(shown below):

sujit@sujitremote:~# rsync -avzP  root@sujitkumar:/etc/default/ /home/sujit

Last but not the least, if the only src is mentioned without the destination in the rsync command, then it lists all the files/directories similar to the one we obtain from “ls -l” command.
Trying rsync in&out with its options can make wonders in your data transfer speed, accuracy, and efficiency.

Post Views: 15,588
The following two tabs change content below.

author_sujit

Latest posts by author_sujit (see all)