Storage¶

The basics¶

The storage on Arrhenius is intended for short- and medium-term storage associated with one or more ongoing compute projects.

It is not an archive system for long-term storage of data not in active use.

There are three basic types of disk storage available to Arrhenius HPC users:

Your home directory

(e.g /home/USERNAME). All users have one. This is a small area (limited to 30 GiB and 1 million files) where you can store applications settings and small personal files not related to any project. Also there are restriction on running containers in this directory. Your home directory is by default only accessible by you, not other project members.

Project storage directories

(e.g /nobackup/proj/disk/DIRECTORYNAME). All projects have access to project storage (at least 250 GiB, but projects can apply for more space at any time if needed).

The local disk

in each compute node. This is where you should store temporary files that are only needed during a compute job and that does not need to be shared between compute nodes.

There are limits to how much data you can store in each location. On /home and /nobackup/proj, a quota system limits how much you can use.

The command storagequota will show how much space is available in each location (except the node local disk) and how much is used.

On the local disk in each node, in /scratch/local, you can store data in proportion to how many CPU cores your jobs has allocated, e.g if you allocate 128 of the 256 CPU cores, you get access to half available disk space on the local disk (i.e 50% of 1.7 TiB per node for the "thin" nodes). You can see how much space is available and currently used using the command df -h /scratch/local from within the job.

To see how much "cold" data is stored in project directories and where that data is located you can use the command storagereport, e.g storagereport /nobackup/proj/disk/mydirectory.

Do not store large amounts of data in other writable locations on the login nodes (e.g on /tmp, /var/tmp, /dev/shm), since the space there is very limited and shared by all users.

Project directories¶

Each project that has been allocated computing time on Arrhenius HPC normally has a directory under /nobackup/proj/disk and/or /nobackup/proj/flash where project members can store their data associated with that project.

The name of the directory is decided by the project Principal Investigator ("PI") when applying for the project in SUPR. Most projects choose to use either the project name as the directory name (e.g /nobackup/proj/disk/naiss2026-1-234) or a name that describes the project or the group using it (e.g /nobackup/proj/disk/metaphysics101).

If you don't know the location of your project directory, you can:

• Ask the project PI
• Look for the "Project storage directories available to you" message when you log in to the cluster using SSH.
• Run the storagequota command, it shows project directories available to you, and how much you can store in them. storagequota -a will show how much disk space is used by individual project members. Note: the per-user data is not updated in real time.

If you can find the project directory for a project you believe you are a member of, but cannot access it, try logging out and back in again. Group memberships in Linux are only initialized when you log in. If you are using ThinLinc, make sure you log out rather than just disconnect your session.

The amount of data and the number of files a project can store in the project directory is limited. Both limits can be raised, all that is needed in most cases is an electronic application in SUPR, or sometimes just an email to NAISS Support.

When you run storagequota, you will see how much data is used ("Used"), the long-term limit is ("Quota") and the absolute limit ("Limit") are. The long-term limit in a project directory can be exceeded for up to 30 days ("Grace" time).

Due to the significant impact it will have on your running jobs (they will almost certainly fail of usage hits the hard Limit or the grace timer expires), you should plan your disk usage to stay below the soft limit ("Quota") at all times. The additional space available up to the hard Limit is intended for e.g a job that created unexpectedly large output files.

If you cannot keep your disk usage below Quota we recommend that you (the project PI should make the request) ask for a higher storage allocation.

Your personal area inside a project storage directory¶

Unless the project PI has decided otherwise, each member of the project is automatically given a personal directory in the "users" sub-directory of the project storage directory (e.g /nobackup/proj/disk/metaphysics101/someusername). This directory is created when the user logs in for the first time after becoming a member of the project.

This directory is by default not accessible by any other user (not even other project members). Exception: the project PI and proxy are allowed to access any data in the project directory (but might require assistance from NAISS to do so).

It is intended as a personal work area where you put temporary files, job work directories etc.

If you want to open up your personal area to others you can do so using the chmod command. If you do, please be aware that this also (unless you change your "umask" and change permissions on existing files) will make it possible for other project members to delete or change your files (intentionally or by mistake).

Non-personal areas in a project storage directory¶

If you want to store data or applications that should be accessible by all project members, we recommend that you create directories inside the "shared" sub-directory instead of inside your personal area.

All directories and files you create there will by default be read- and writable by all project members.

If the project is going to have lots of shared data, we recommend that the PI decide on a suitable directory structure, e.g

/nobackup/proj/disk/metaphysics101/personal/...                - personal areas
/nobackup/proj/disk/metaphysics101/shared/datasets/YYYY/MM/DD  - shared datasets
/nobackup/proj/disk/metaphysics101/shared/scripts              - useful scripts
/nobackup/proj/disk/metaphysics101/apps/someapp-X.Y            - applications
[...]

What types of data to store in project storage directories¶

You should use the project storage directory for all data associated with the project, except for temporary files that are only used from a single compute node during a job (such files should be stored on the local disk in the compute node. This includes:

• Input files
• Output files
• Job scripts
• Any applications installed by project members

If you want extra protection for small-volume, high-value data such as source code or scripts, you can store it on /home (or keep an extra copy there or outside NAISS).

Important changes for Tetralith users¶

• Arrhenius unfortunately has no "snapshots" of data in project directories. This means that if you delete a file, it is gone forever!

• Arrhenius unfortunately has no "snapshots" of data in home directories. We do however plan to make tape backups of the home directories. If you accidentally delete a file in your home directory, contact NAISS Support to see if it is possible to restore the file from a backup.

Is my data safe¶

We consider the storage system itself to be reliable. It is based on the proven technology (Lustre). Data on the system is protected against multiple disk failures, etc.

However, due to the lack of snapshots or other backups of the project storage directories, we strongly recommend that you keep copies of any important data outside Arrhenius.

Project storage data will be deleted 90 days after the project ends. It is your responsibility to copy or move your data before this happens.

When projects end¶

When the project that "owns" a project storage directory ends, the project no longer has the right to continue using the Arrhenius storage system to store their data. At that point, the process of removing the data starts.

You should plan to remove all your data from the project directory within 30 days of the project end date. After this point accessing the data will become more difficult and the data will eventually be deleted

Phase 1: "30 days of normal use + email warnings"¶

When a project ends, the project directory remains writable for 30 days.

This period allows running jobs to finish (jobs are allowed to start up until the time the project ends) and lets you do limited post-processing (e.g package files into zip or tar archives) and transfer data away from NAISS.

During this period, NAISS will send out automated emails weekly to the project PI and project members.

Please note that we cannot guarantee that you will receive those emails. Common problems are:

• The email address you have registered in SUPR is no longer valid, or you don't read it very often.

• The automated emails are classified as spam and end up in your spam folder.

Phase 2: "another 30 days, the data is delete-only + email warnings"¶

From 30 to 60 days after the project ends, the data remains on disk, but you cannot write new data, only read and delete old data.

By making the data "delete-only", any jobs that use the project directory for output will fail, and that user will notice something is about to happen.

During this period, NAISS will continue to send out the automated email reminders.

Phase 3: "another 30 days, data unavailable but can still be restored"¶

From 60 to 90 days after the project ends, the data remains on disk, but is hidden from view.

The data is hidden to make sure any users or applications that use the data (e.g as a read-only data set) notices that the data is gone and acts before the data is actually deleted.

No automated email reminders are sent out during this period. Instead NAISS will make a reasonable effort to reach the project PI to verify that he/she is aware that the remaining data will be deleted and that the project has saved any important data.

Phase 4: deletion¶

Once NAISS gets an OK from the PI to delete the remaining data, the project directory will be deleted even if less than 90 days have passed since the project ended.

If NAISS cannot reach the PI we reserve the right to delete the data anyway, but we will not do so until at least 90 days after the project ended.

Once NAISS deletes the data, there is no way to get it back. There are no backup copies on tape, etc.

If you need more storage space or be able to store a larger number of files¶

Depending on how large the increase is, you may need to submit a new project Proposal in SUPR, but sometimes an email can suffice. Please contact NAISS Support to find out what applies to you.

Who can access my data¶

If you don't change any system settings (e.g "umask") or file permissions, only you (with some exceptions, see below) can access files in your home directory (e.g /home/x_abcde) or files in your personal area (e.g /nobackup/proj/disk/projname/personal/x_abcde) of project directories. Data in other parts of the project directory (e.g /nobackup/proj/disk/projname/data/somedata) can be read, modified and deleted by all project members.

Exceptions to the above:

• Some NAISS staff can access all your data, but are not allowed to do so unless specifically requested by you, e.g to help you solve a technical problem.

• The PI and proxy for the PI are allowed to access any data in the project directory, but NOT your home directory.

Unix file permissions¶

You can manage who has access to your files and directories using traditional Unix file permissions, using the chmod, chgrp and umask commands.

Ownership of data and legal issues¶

NAISS' position is that computing time and project storage are allocated to the project by NAISS, and that NAISS has authorized the PI to decide who gets to be a member in the project, and how much computing and storage resources each member may use.

In order to manage the project storage, the project PI (or someone designated by the PI, e.g the SUPR "Proxy for the PI") can therefore get assistance from NAISS with the following:

• See how much storage space is used by each project member.
• Read the contents of any directories/files within the project storage directory (e.g to determine what type of data is stored if it should be kept).
• Delete any data within the project storage directory (e.g to free up space used by unwanted files owned by inactive users).
• Change ownership and file permissions of any data within the project directory (e.g to take ownership of still-relevant files from inactive users).

We strongly encourage PI:s to talk to project members before deleting or changing the ownership of their data.

Please note that NAISS considers this to be unrelated to the intellectual property rights of the actual file contents. This just concerns the rights to store and organize bits and bytes on disk.

When a user is removed from a project, files in the project directory owned by that user are not automatically removed. It is up to the project PI to decide if the files should be kept on disk or not.

A user's home directory is considered to be personal and not connected to any project. NAISS will not allow PI:s to delete or view contents from the home directory without permission from the user. Home directories of inactive users will eventually be deleted.

NAISS' interpretation¹ of Swedish law is that data on Arrhenius is not public ("allmänna handlingar").

This also means that NAISS is not required to archive data stored Arrhenius.