Skip to content

File transfer using rsync

Learning outcomes

  • Practice using the documentation of your favorite HPC cluster
  • Can transfer files using rsync
For teachers

Teaching goals are:

  • Learners have practiced using the documentation of their favorite HPC cluster
  • Learners have transferred files using rsync

Prior questions:

  • How can one do a file transfer from the terminal?
  • Can you name some programs that can do file transfer from the terminal?
  • Who has heard of rsync?
  • Who has already used rsync?

Notes:

  • Bianca users will have the most complex procedure by far

Feedback:

  • When would you recommend a colleague to use rsync, and when not?

What is rsync?

A tool to transfer files or folders, with/without compressing.

rsync my_local_file.txt sven@pelle.uppmax.uu.se:/home/sven/
rsync --recursive my_local_folder sven@pelle.uppmax.uu.se:/home/sven/
rsync --compress my_local_file.txt sven@pelle.uppmax.uu.se:/home/sven/

Why use rsync?

Fairly sure you'll want to upload or download files to your favorite HPC cluster. Here we do so.

We use a free and open-source command-line tool to do this, called rsync. It works under Linux, Mac and Windows.

  • rsync is the most powerful transfer tool between local files and remote files. For example, it provides an option to compress files for you
  • rsync can deal with interrupted downloads, making it suitable for big files
  • rsync will only transfer the difference between two files. If you want to upload a same file again, it will see so and do nothing. This is perfect for backups!

Exercises

Need a video?

HPC Cluster YouTube video
Alvis YouTube video
Berzelius Cannot: is not a NAISS cluster
Bianca YouTube video
COSMOS YouTube video
Dardel YouTube video
Kebnekaise YouTube video
LUMI YouTube video
Pelle YouTube video
Tetralith YouTube video
Vera Cannot: is not a NAISS cluster
How difficult will this be?

This depends on:

  • operating system
  • your HPC cluster

Per operating system, it differs how easy it is to install rsync:

Operating system Difficulty
Linux Easy
Mac Easy
Windows Medium

HPC Cluster Difficulty Reason
Alvis Medium No specialized documentation
Berzelius Easy Has specialized documentation
Bianca Hard Has specialized documentation, complex procedure
COSMOS Medium No specialized documentation
Dardel Medium No specialized documentation
Kebnekaise Hard Unknown error messages
LUMI Medium CSC refuses to document this, but UPPMAX has specialized documentation
Pelle Easy Has specialized documentation
Tetralith Easy Has specialized documentation
Vera Easy Has specialized documentation

Exercise 1: transfer a file

Use the documentation of your favorite HPC cluster.

Where is the documentation of my favorite HPC cluster?

HPC Cluster Documentation
Alvis Documentation
Berzelius Documentation
Bianca Documentation
COSMOS Documentation
Dardel Documentation
Kebnekaise Documentation
LUMI Documentation
Pelle Documentation
Rackham Documentation
Tetralith Documentation
Vera Documentation

For maximally 5 minutes, search for how to transfer files to/from your HPC cluster using rsync. Which URL is it described?

Take a look at the answer if you cannot find it: sometimes there is no documentation. If the center maintaining you HPC cluster has not documented how to use rsync, follow the Pelle documentation.

Where is that documentation?

HPC Cluster Documentation
Alvis Documentation that is closest. For now, use the UPPMAX documentation
Berzelius Documentation
Bianca Documentation
COSMOS None. Documentation that is closest. Use the scp documentation and replace scp by rsync
Dardel Documentation that is closest
Kebnekaise Documentation
LUMI CSC refuses to document this, but UPPMAX has specialized documentation
Pelle Documentation
Sigma Documentation that is closest. For now, use the UPPMAX documentation
Tetralith Documentation
Vera Documentation that is closest. For now, use the UPPMAX documentation

Copy a file from your local computer to your HPC cluster's home folder.

Answer

To copy a local file, below is the general form, with a table that explains the parts in more details.

rsync [my_local_file] [my_user_name]@[my_hpc_cluster_url]:[remote_destination]
Notation Description Example
[my_local_file] The path to the local file my_local_file.txt
[my_user_name] Your user name for that HPC cluster sven
[my_hpc_cluster_url] Your user name for that HPC cluster pelle.uppmax.uu.se
[remote_destination] The folder on the HPC cluster you want to upload to /home/sven

For example:

rsync my_local_file.txt sven@rackham.uppmax.uu.se:/home/sven/

Optionally, verify that you have successfully uploaded that file.

Why optionally?

Because if the file was transferred without an error message, you can be reasonably sure it is uploaded successfully.

Answer

There are many ways to check:

  • In remote desktop environment, you can see file with the file explorer
  • In a terminal, navigate to that folder and use ls to list all the files in that folder
  • In a terminal, navigate to that folder and use cat [my_file_name] to display that file (e.g. cat my_file.txt)

Copy a file from your HPC cluster's home folder to your local computer. Verify that this worked.

Answer

To copy a file from an HPC cluster to your local computer, below is the general form, with a table that explains the parts in more details.

rsync [my_user_name]@[my_hpc_cluster_url]:[remote_path] [local_folder]
Notation Description Example
[my_user_name] Your user name for that HPC cluster sven
[my_hpc_cluster_url] Your user name for that HPC cluster pelle.uppmax.uu.se
[remote_path] Path to the file on the HPC cluster you want to download /home/sven/my_file.txt
[local_folder] The folder to save the local file in . (i.e. a period)

As a local folder, . (i.e. a period) is read as 'here'/'this folder'.

For example:

rsync sven@rackham.uppmax.uu.se:/home/sven/my_file.txt .

(optional) Exercise 2: transfer a compressed file

One of the biggest bottlenecks for file transfer speed is file size.

In this optional exercise, we'll compress the file before transferring, and decompress it at its destination. rsync can do this, by adding the --compress flag, e.g.

rsync --compress my_local_file.txt sven@pelle.uppmax.uu.se:/home/sven/

  • Download an example file from examplefile.com, that has the size you are interested in, for example this 100 MB text file

  • Let rsync transfer the file with and without using the --compress flag. Does the compress and decompress make the operation faster as a whole?

Answer

This depends.

The slower the network/internet connection is, the likelier it is that --compress will make the transfer faster.