Skip to content

File transfer using scp

Learning outcomes

  • Practice using the documentation of your favorite HPC cluster
  • Can transfer files using scp
For teachers

Teaching goals are:

  • Learners have practiced using the documentation of their favorite HPC cluster
  • Learners have transferred files using scp

Overview of terminal transfer tools

Info

  • Terminal transfer tools are handy when you are already working in the terminal and can sometimes be superior to Graphical tools.

    • No switching applications
    • Your hands stay on the keyboard, you don't need to grab the mouse
    • Tab completion can make finding files faster
    • If you are working on servers or over ssh you might not be able to use a graphical user interface (GUI).
  • scp has the similar arguments as for the linux copy function cp.
  • sftp is more versatile with more file management capabilities. Transfer 201 workshop

  • rsync is perfect for syncing and have many capabilities. Transfer 201 workshop

  • All are considered secure!

SCP is an abbreviation for Secure Copy Protocol

Prior questions
  • Q: Who has heard of scp?
  • Q: Who has already used scp?
  • Q: What is SCP?
    • A: a protocol to transfer files
  • Q: What is an SCP client?
    • A: the program to transfer files via SSH with

Pros

  • Simple
  • One-line command
  • Use cases
    • copy just a file
    • copy just a specific directory (with sub-directories).

Risk of overwriting files

  • There is no warning if a file is about to be overwritten.
    • There is no scp -i ... as for rm -i that asks if you really want to remove the file.
  • rsync may be a better tool if you want to sync existing content.

Attention

  • Some Windows users (not using MobaXterm) may need to use pscp instead of scp.
  • The syntax is however the same in general.

Procedure

  • Run the scp commands on YOUR computer, since you probably do not have a server address to your computer!

  • In the terminal (from local, not server session)

    scp <from> <to>

  • Where <from> is the file(s) you want to copy, and <to> is the destination.

Tip

  • Keep one terminal window open with the ssh session where you can check the remote files.
  • and one window with the local session where you run the scp commands!
Set paths

Example: Copy the local file file.txt in the folder transfer_course (located in the present directory) to the transfer_course folder on your cluster:

scp transfer_course/file.txt <username>@<cluster adress>:~/transfer_course/
Example for Arrhenius

This is how you copy a file from your local computer directly to your HOME folder (~/):

scp my_file.txt nisse@arrhenius1.hpc.arrhenius.naiss.se:~/

To copy a file from Tetralith to your local computer (and present folder), do the command above in reverse order:

scp nisse@login.hpc.arrhenius.naiss.se:~/remote_file.txt .
  • If asked, give your center's password, and possibly, 2nd-factor 6-digit code.
  • You can get rid of this prompt if you have setup SSH keys
Example for Dardel

This is how you copy a file from your local computer directly to your HOME folder (~/):

scp my_file.txt nisse@dardel.pdc.kth.se:~/

To copy a file from Dardel to your local computer (and present folder), do the command above in reverse order:

scp nisse@dardel.pdc.kth.se:~/remote_file.txt .
  • You may need to give your ssh-key, then start the command line with:
scp -i ~/.ssh/id-ed25519-pdc ...

Exercises

Exercise 0: Use the documentation of your HPC cluster
  • Search for how to transfer files to/from your HPC cluster using scp. At which URL is it described?

    • Tip: not all HPC centers have documented this, so you should give up searching after a while.

Where is that documentation?
HPC Cluster Documentation
Alvis Documentation.
Arrhenius Documentation.
Berzelius Documentation
Bianca Available for download via the transit server, see documentation
COSMOS Documented on UPPMAX page.
Dardel Documentation
Kebnekaise Documentation
LUMI Documentation
Pelle Documentation
Tetralith Documentation
Vera Documentation.

Tip

  • Keep one terminal window open with the ssh session where you can check the remote files.
  • and one window with the local session where you run the scp commands!
Exercise 1: Upload a file from your computer, using scp

Tips

  • Useful terminal commands (both locally and remotely)

    • pwd - which folder am I in?
    • cd [path] - change folder (go up in hierarchy with cd ..)
    • ls - list content of folder
    • mkdir - make a new folder
    • touch - create empty file

  • (If you want to create a file in local terminal: $ touch local_file)

  • (You can check the file structure in an ssh session)
  • Send it to an existing folder (e.g. transfer) on Tetralith
    • use mkdir <folder name> if it is not there
  • Check on server that it is there
Answer (Arrhenius example)

Locally

  • (If you want to create a file in local terminal: $ touch local_file)

  • Send it to an existing folder (e.g. transfer) on Dardel:

  • $ scp local_file <username>@login.hpc.arrhenius.naiss.se:~/transfer/

or if you get "permission denied":

  • $ scp -i ~/.ssh/id-ed25519-pdc local_file <username>@login.hpc.arrhenius.naiss.se:~/transfer/

Check on Arrhenius that it is there

  • $ ls ~/transfer
Answer (Dardel example)

Locally

  • (If you want to create a file in local terminal: $ touch local_file)

  • Send it to an existing folder (e.g. transfer) on Dardel:

  • $ scp local_file <username>@dardel.pdc.kth.se:~/transfer/

or if you get "permission denied":

  • $ scp -i ~/.ssh/id-ed25519-pdc local_file <username>@dardel.pdc.kth.se:~/transfer/

Check on Arrhenius that it is there

  • $ ls ~/transfer

???- tip "Answer (Tetralith example)"

Locally

- (If you want to create a file in *local* terminal: ``$ touch local_file``)
- Send it to an existing folder (e.g. ``transfer``) on Tetralith:

- ``$ scp local_file <username>@tetralith.nsc.liu.se:~/transfer/``

Check on Arrhenius that it is there

- ``$ ls ~/transfer``

- [Video for Tetralith](https://youtu.be/rvL-s5vi13I)
Exercise 2: Download a file from the server to your computer, using scp

Tips

  • (If you want to create a file in remote ssh terminal: $ touch remote_file)
  • Send it to an existing local folder
  • Check locally that it is there
Answer (Arrhenius example)

On Server

  • (If you want to create a remote file first, in an SSH session, do: $ touch remote_file)
  • Get it in the present local folder: $ scp <username>@login.hpc.arrhenius.naiss.se:~/transfer/remote_file .

Check locally that it is there

Answer (Dardel example)

On Server

  • (If you want to create a remote file first, in an SSH session, do: $ touch remote_file)

  • Get it the present local folder:

  • $ scp <username>@dardel.pdc.kth.se:~/transfer/remote_file .

or if you get "permission denied":

  • $ scp -i ~/.ssh/id-ed25519-pdc <username>@dardel.pdc.kth.se:~/transfer/remote_file .

Check locally that it is there

Answer (Tetralith example)

On Server

  • (If you want to create a remote file first, in an SSH session, do: $ touch remote_file)
  • Get it the present local folder: $ scp <username>@tetralith.nsc.liu.se:~/transfer/remote_file .

Check locally that it is there

(Optional) Exercise 3: Learn more

But what is wget and curl?

  • These tools are used to download files from websites or ftp servers
wget
  • wget saves downloaded contents to local files, like
  • wget ftp://ftp.sunet.se/mirror/archive/ftp.sunet.se/pub/pictures/space/*
  • wget https://upload.wikimedia.org/wikipedia/commons/3/37/Grace_Hopper_and_UNIVAC.jpg -O grace_hopper.jpg
  • typical use cases:
    • download data from a service
    • download a program or compressed source code
  • supports HTTP, HTTPS, and FTP
  • user-friendly for basic tasks
  • good for mirroring websites
  • downloading entire directories recursively for offline viewing or backups.
  • most popular from Unix-based systems, like Linux
curl
  • curl outputs the content to the terminal by default.
  • add -O to download as file.
  • supports a wide range of protocols: HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP...
  • often preferred for scripting and automation due to its versatility
  • interacting with APIs, handling complex web requests
  • often available by default on Windows and MacOS.

Extra

Cheat sheet for extra scp

  • scp followed by none or any of the following option flags and the files and servers involved

  • scp provides a number of options that control every aspect of its behavior. The most widely used options are:

  • -P - Specifies the remote host ssh port.

  • -p - Preserves file modification and access times.
  • -q - Use this option if you want to suppress the progress meter and non-error messages.
  • -C - This option forces scp to compress the data as it is sent to the destination machine.
  • -r - This option tells scp to copy directories recursively.

When not to use (use rsync or sftp)

  • When needing several one-line commands
    • requires credentials every time
  • When looking to do more than a basic file transfer, SCP falls short.
  • Transfers that are interrupted you have to restart the entire transfer.
  • A file with the same name in the same directory is transferred, will be overwritten.
More about SCP in 'Transfer 201' workshop
About compressing and archiving in 'Transfer 201' workshop