Quick Start¶

Brief description¶

Arrhenius is the new powerful supercomputer that NAISS is building together with EuroHPC. You can find hardware documentation at https://www.naiss.se/resource/arrhenius/

Requesting a login account on Arrhenius HPC¶

In order to get access to arrhenius HPC you need to apply for a user account via SUPR

1. Go to the Accounts page. On this page you will be able to see your existing accounts and their status.
2. Press Request Account on Arrhenius HPC @ NAISS. If you already have a NAISS username, that will be reused. Otherwise, you have 2 options to select a username:
   1. Use the dropdown list to select one of the suggested usernames (from earlier systems or automatically constructed)
   2. Enter another choice in the Requested Username text box. This will cause a manual check that will add some time to the process.
3. Press Request Account to proceed.

Accepting the new NAISS General Terms of Use¶

For Arrhenius, the new NAISS General Terms of Use replace the earlier NAISS User Agreement. If you have not already accepted them, you will be redirected to the page where you do that.

When you have accepted the terms, you should get redirected to the page for setting password and two-factor authenticaiton for your Arrhenius HPC login account

Setting password and registering two-factor authentication¶

You should automatically get to this page for your Arrhenius HPC account when you have requested an account and accepted the NAISS General Terms of Use. Otherwise, you get here via the Set Password and Two-Factor Authentication link in the Accounts table. If your Account Request was waiting for project membership and/or a manual username check, you get here from the link in the email sent when ready.

Follow the instructions on the page to install a TOTP app if needed, scan the QR code with the TOTP app and then enter your desired Arrhenius HPC password and the first verification code from the TOTP app to complete the process.

Checking login account status¶

The status of your account creation can be seen on the Accounts page.

How to log in¶

To log in to NAISS resources, you first need a NAISS account. Once your account is created, you will be granted access to the resources. For example, in order to login to Arrhenius...

ssh [username]@login.hpc.arrhenius.naiss.se
password: ******
2FA: *****

In case this does not work, please use arrhenius1.hpc.arrhenius.naiss.se or arrhenius3.hpc.arrhenius.naiss.se.

Storage¶

Important changes¶

For Tetralith users¶

• Arrhenius unfortunately has no "snapshots" of data in project directories. This means that if you delete a file, it is gone forever!

• Arrhenius unfortunately has no "snapshots" of data in home directories. We do however plan to make tape backups of the home directories. If you accidentally delete a file in your home directory, contact NAISS Support to see if it is possible to restore the file from a backup.

The basics¶

The storage on Arrhenius is intended for short- and medium-term storage associated with one or more ongoing compute projects.

It is not an archive system for long-term storage of data not in active use.

There are three basic types of disk storage available to Arrhenius HPC users:

• Your home directory (e.g /home/USERNAME). All users have one. This is a small area (limited to 30 GiB and 1 million files) where you can store applications settings and small personal files not related to any project. Your home directory is by default only accessible by you, not other project members.
• Project storage directories (e.g /nobackup/proj/disk/DIRECTORYNAME). All projects have access to project storage (at least 250 GiB, but projects can apply for more space at any time if needed).
• The local disk in each compute node. This is where you should store temporary files that are only needed during a compute job and that does not need to be shared between compute nodes.

There are limits to how much data you can store in each location. On /home and /nobackup/proj, a quota system limits how much you can use.

The command storagequota will show how much space is available in each location (except the node local disk) and how much is used.

On the local disk in each node (/scratch/local, a.k.a $SNIC_TMP) you can store data in proportion to how many CPU cores your jobs has allocated, e.g if you allocate 128 of the 256 CPU cores, you get access to half available disk space on the local disk (i.e 50% of 1.7 TiB per node for the "thin" nodes). You can see how much space is available and currently used using the command df -h /scratch/local from within the job.

We will at a later time provide a tool similar to the Tetralith storagereport command that can show more detailed information about "cold data".

Do not store large amounts of data in other writable locations on the login nodes (e.g on /tmp, /var/tmp, /dev/shm), since the space there is very limited and shared by all users.

Project directories¶

Each project that has been allocated computing time on Arrhenius HPC normally has a directory under /nobackup/proj/disk and/or /nobackup/proj/flash where project members can store their data associated with that project.

The name of the directory is decided by the project Principal Investigator ("PI") when applying for the project in SUPR. Most projects choose to use either the project name as the directory name (e.g /nobackup/proj/disk/naiss2026-1-234) or a name that describes the project or the group using it (e.g /nobackup/proj/disk/metaphysics101).

If you don't know the location of your project directory, you can:

• Ask the project PI
• Look for the "Project storage directories available to you" message when you log in to the cluster using SSH.
• Run the storagequota command, it shows project directories available to you, and how much you can store in them. There is currently no option to see how much data each project member is using.

If you can find the project directory for a project you believe you are a member of, but cannot access it, try logging out and back in again. Group memberships in Linux are only initialized when you log in. If you are using ThinLinc, make sure you log out rather than just disconnect your session.

The amount of data and the number of files a project can store in the project directory is limited. Both limits can be raised, all that is needed in most cases is an electronic application in SUPR, or sometimes just an email to NAISS Support.

When you run storagequota, you will see how much data is used ("Used"), the long-term limit is ("Quota") and the absolute limit ("Limit") are. The long-term limit in a project directory can be exceeded for up to 30 days ("Grace" time).

Due to the significant impact it will have on your running jobs (they will almost certainly fail of usage hits the hard Limit or the grace timer expires), you should plan your disk usage to stay below the soft limit ("Quota") at all times. The additional space available up to the hard Limit is intended for e.g a job that created unexpectedly large output files.

If you cannot keep your disk usage below Quota we recommend that you (the project PI should make the request) ask for a higher storage allocation.

Your personal area inside a project storage directory¶

Unless the project PI has decided otherwise, each member of the project is automatically given a personal directory in the "users" sub-directory of the project storage directory (e.g /nobackup/proj/disk/metaphysics101/someusername). This directory is created when the user logs in for the first time after becoming a member of the project.

This directory is by default not accessible by any other user (not even other project members). Exception: the project PI and proxy are allowed to access any data in the project directory (but might require assistance from NAISS to do so).

It is intended as a personal work area where you put temporary files, job work directories etc.

If you want to open up your personal area to others you can do so using the chmod command. If you do, please be aware that this also (unless you change your "umask" and change permissions on existing files) will make it possible for other project members to delete or change your files (intentionally or by mistake).

Non-personal areas in a project storage directory¶

If you want to store data or applications that should be accessible by all project members, we recommend that you create directories inside the "shared" sub-directory instead of inside your personal area.

All directories and files you create there will by default be read- and writable by all project members.

If the project is going to have lots of shared data, we recommend that the PI decide on a suitable directory structure, e.g

/nobackup/proj/disk/metaphysics101/personal/...                - personal areas
/nobackup/proj/disk/metaphysics101/shared/datasets/YYYY/MM/DD  - shared datasets
/nobackup/proj/disk/metaphysics101/shared/scripts              - useful scripts
/nobackup/proj/disk/metaphysics101/apps/someapp-X.Y            - applications
[...]

What types of data to store in project storage directories¶

You should use the project storage directory for all data associated with the project, except for temporary files that are only used from a single compute node during a job (such files should be stored on the local disk in the compute node. This includes:

• Input files
• Output files
• Job scripts
• Any applications installed by project members

If you want extra protection for small-volume, high-value data such as source code or scripts, you can store it on /home (or keep an extra copy there or outside NAISS).

Is my data safe¶

We consider the storage system itself to be reliable. It is based on the proven technology (Lustre). Data on the system is protected against multiple disk failures, etc.

However, due to the lack of snapshots or other backups of the project storage directories, we strongly recommend that you keep copies of any important data outside Arrhenius.

Project storage data will be deleted 90 days after the project ends. It is your responsibility to copy or move your data before this happens.

When projects end¶

When the project that "owns" a project storage directory ends, the project no longer has the right to continue using the Arrhenius storage system to store their data. At that point, the process of removing the data starts.

You should plan to remove all your data from the project directory within 30 days of the project end date. After this point accessing the data will become more difficult and the data will eventually be deleted

Phase 1: "30 days of normal use + email warnings"¶

When a project ends, the project directory remains writable for 30 days.

This period allows running jobs to finish (jobs are allowed to start up until the time the project ends) and lets you do limited post-processing (e.g package files into zip or tar archives) and transfer data away from NAISS.

During this period, NAISS will send out automated emails weekly to the project PI and project members.

Please note that we cannot guarantee that you will receive those emails. Common problems are:

• The email address you have registered in SUPR is no longer valid, or you don't read it very often.

• The automated emails are classified as spam and end up in your spam folder.

Phase 2: "another 30 days, the data is delete-only + email warnings"¶

From 30 to 60 days after the project ends, the data remains on disk, but you cannot write new data, only read and delete old data.

By making the data "delete-only", any jobs that use the project directory for output will fail, and that user will notice something is about to happen.

During this period, NAISS will continue to send out the automated email reminders.

Phase 3: "another 30 days, data unavailable but can still be restored"¶

From 60 to 90 days after the project ends, the data remains on disk, but is hidden from view.

The data is hidden to make sure any users or applications that use the data (e.g as a read-only data set) notices that the data is gone and acts before the data is actually deleted.

No automated email reminders are sent out during this period. Instead NAISS will make a reasonable effort to reach the project PI to verify that he/she is aware that the remaining data will be deleted and that the project has saved any important data.

Phase 4: deletion¶

Once NAISS gets an OK from the PI to delete the remaining data, the project directory will be deleted even if less than 90 days have passed since the project ended.

If NAISS cannot reach the PI we reserve the right to delete the data anyway, but we will not do so until at least 90 days after the project ended.

Once NAISS deletes the data, there is no way to get it back. There are no backup copies on tape, etc.

If you need more storage space or be able to store a larger number of files¶

Depending on how large the increase is, you may need to submit a new project Proposal in SUPR, but sometimes an email can suffice. Please contact NAISS Support to find out what applies to you.

Who can access my data¶

If you don't change any system settings (e.g "umask") or file permissions, only you (with some exceptions, see below) can access files in your home directory (e.g /home/x_abcde) or files in your personal area (e.g /nobackup/proj/disk/projname/personal/x_abcde) of project directories. Data in other parts of the project directory (e.g /nobackup/proj/disk/projname/data/somedata) can be read, modified and deleted by all project members.

Exceptions to the above:

• Some NAISS staff can access all your data, but are not allowed to do so unless specifically requested by you, e.g to help you solve a technical problem.

• The PI and proxy for the PI are allowed to access any data in the project directory, but NOT your home directory.

Unix file permissions¶

You can manage who has access to your files and directories using traditional Unix file permissions, using the chmod, chgrp and umask commands.

Ownership of data and legal issues¶

NAISS' position is that computing time and project storage are allocated to the project by NAISS, and that NAISS has authorized the PI to decide who gets to be a member in the project, and how much computing and storage resources each member may use.

In order to manage the project storage, the project PI (or someone designated by the PI, e.g the SUPR "Proxy for the PI") can therefore get assistance from NAISS with the following:

• See how much storage space is used by each project member.
• Read the contents of any directories/files within the project storage directory (e.g to determine what type of data is stored if it should be kept).
• Delete any data within the project storage directory (e.g to free up space used by unwanted files owned by inactive users).
• Change ownership and file permissions of any data within the project directory (e.g to take ownership of still-relevant files from inactive users).

We strongly encourage PI:s to talk to project members before deleting or changing the ownership of their data.

Please note that NAISS considers this to be unrelated to the intellectual property rights of the actual file contents. This just concerns the rights to store and organize bits and bytes on disk.

When a user is removed from a project, files in the project directory owned by that user are not automatically removed. It is up to the project PI to decide if the files should be kept on disk or not.

A user's home directory is considered to be personal and not connected to any project. NAISS will not allow PI:s to delete or view contents from the home directory without permission from the user. Home directories of inactive users will eventually be deleted.

NAISS' interpretation¹ of Swedish law is that data on Arrhenius is not public ("allmänna handlingar").

This also means that NAISS is not required to archive data stored Arrhenius.

Transfer of files¶

In order to copy data to and from Arrhenius, you can use any SSH or SFTP-based tool (e.g rsync, scp, sftp, FileZilla, ...). Later more file transfer mechanisms might be implemented.

The Arrhenius hostname to use login.hpc.arrhenius.naiss.se.

If you have copied data to or from e.g Tetralith, this should already be familiar to you. If not, there are many rsync guides available online for beginners and pros alike, e.g 1, 2, 3, 4, 5 or 6.

When transferring large amounts of data, we recommend that you use a tool (e.g rsync) that can resume an interrupted transfer.

When moving data within Arrhenius, e.g between two project storage directories, you can use any normal Linux commands, but you should be aware that e.g mv is not instantaneous since the project quota counters needs to be updated which makes the process a little slower than you might expect.

If you want to make rsync do something other than the "preserve mostly everything" that the -a option does, read the man page (man rsync on the cluster).

Basic example 1: push data from Tetralith to Arrhenius

[x_examp@tetralith1 ~]$ rsync -av /proj/somedirectory/somesubdirectory/ myarrheniususer@login.hpc.arrhenius.naiss.se:/nobackup/proj/disk/somedirectory/somewhere/
(myarrheniususer@login.hpc.arrhenius.naiss.se) Password: <ENTER YOUR ARRHENIUS PASSWORD HERE>
(myarrheniususer@login.hpc.arrhenius.naiss.se) Verification code: <ENTER YOUR ARRHENIUS 2FA CODE HERE>
sending incremental file list
created directory /nobackup/proj/disk/somedirectory/somewhere
[...]
[x_examp@tetralith1 ~]$

Basic example 2: pull data from Tetralith to Arrhenius, preserve hard links and handle sparse files efficiently:

[myarrheniususer@arrhenius11 ~]$ rsync -av -H --sparse x_examp@tetralith:/proj/somedirectory/somesubdirectory/ /nobackup/proj/disk/somedirectory/somewhere/
(x_examp@tetralith.nsc.liu.se) Password: <ENTER YOUR TETRALITH PASSWORD HERE>
(x_examp@tetralith.nsc.liu.se) Verification code: <ENTER YOUR TETRALITH 2FA CODE HERE>
sending incremental file list
created directory /nobackup/proj/disk/somedirectory/somewhere/
[...]
[myarrheniususer@arrhenius11 ~]$

The Lmod module system¶

Arrhenius uses the Lmod module system to make installed software available and manage the end user environment in a consistent and convenient way from the command line. Invoking the special module help command will present a brief but comprehensive help text on how to use the module command. Also, here you can read about how Lmod abbreviates <action>, while the command itself can also be shortened to ml.

The basic schema for module system usage is...

module <action> <subject>

The primary purpose of the module system is to modify the user environment variables such that software becomes available on search paths and settings are implemented controlling this software. The <action> argument is a verb denoting what should be done with <subject>, which would typically be a named module in most day to day use.

Using the module system boils down to invoking the module command with arguments on the command line or in a script (any script, including batch job scripts).

There are four basic module command invocations which cover most usage.

For instance, invoking module avail buildenv will list all modules available on the system containing the string buildenv. This search function is character case insensitive.

Skipping the search string to module avail lists all modules available on the configured module search paths. Choosing one of them, e.g. buildenv-intel/2025b-eb, we can make the indicated software environment accessible to us with module load buildenv-intel/2025b-eb.

Assuming a successful loading, the module should be listed as loaded when invoking a module list.

Reversing the environment changes implemented by the module load <module_name> command is done with a module unload <module_name>.

GPU and CPU modules¶

The machine CPU architecture on Arrhenius is fundamentally different between GPU-equipped nodes and the CPU-only nodes, where the former is an ARM-based architecture, and the latter is an x86_64-based architecture. As a consequence, software built for one part of the machine is not usable on the other. This is reflected both in software installation paths and how compatible modules are presented via the module system. For instance

$ module avail

---------------------------------------------------------------------------------------- CPU partition (epyc9005) modules ----------------------------------------------------------------------------------------
   buildenv-gcc/recommendation (D)    buildenv-intel/recommendation (D)    buildtool-easybuild/5.2.1-hpca3ef7d197        CMake/3.31.11-bdist        Ninja/recommendation    (D)    PatchELF/0.15.5-bdist
   buildenv-gcc/2025b-eb              buildenv-intel/2023a-eb              CMake/recommendation                   (D)    CMake/4.3.2-bdist          Ninja/1.13.2-bdist
   buildenv-gcc/2026.03               buildenv-intel/2025b-eb              CMake/3.29.7-bdist                            Miniforge/26.1.1-3-hpc1    PatchELF/recommendation (D)

------------------------------------------------------------------------------------------ GPU partition (h200) modules ------------------------------------------------------------------------------------------
   GPU/buildenv-gcccuda/recommendation (D)    GPU/buildenv-nvhpc/25.9-cu13.0                    GPU/CMake/3.29.7-bdist     GPU/Ninja/recommendation    (D)    GPU/PatchELF/0.15.5-bdist
   GPU/buildenv-gcccuda/2026.03-cu13.0        GPU/buildtool-easybuild/5.2.1-hpca3ef7d197        GPU/CMake/3.31.11-bdist    GPU/Ninja/1.13.2-bdist
   GPU/buildenv-nvhpc/recommendation   (D)    GPU/CMake/recommendation                   (D)    GPU/CMake/4.3.1-bdist      GPU/PatchELF/recommendation (D)

  Where:
   D:  Default Module

Modules having names prefixed by a GPU/ string are exclusively usable on the GPU partition, while modules lacking this prefix are usable only on the CPU partition. Apart from this prefix string, you may well find identically named modules available for both the GPU partition and the CPU partition, very likely serving the identical software environment in the different contexts.

The programming environments¶

Programming (or software build) environments on an HPC system like Arrhenius are a collection of HPC tools facilitating the development and building of (typically parallelized) scientific software, all the while making the most efficient use possible of the underlying hardware. The programming environments typically contain optimizing compilers, libraries, debuggers and profilers as well as various supporting software to improve quality of life on the machine for the working scientist. All build environments on Arrhenius contain the following base components

• Compiler (e.g. GCC, Intel, NVHPC)
• MPI implementation (e.g. OpenMPI, Cray-MPICH, Intel MPI)
• Numerical libraries
  • BLAS/LAPACK implementation (e.g. Intel MKL, OpenBLAS, NVPL)
  • FFTW implementation (e.g. Intel MKL, FFTW, NVPL)
  • ScaLAPACK implementation (e.g. Intel MKL, NVPL)

The GPU specific build environments in addition contain a CUDA SDK, as well as some CUDA dependent numerical and communication libraries like NCCL, NVSHMEM and cuFFTmp.

Programming environments are mainly accessed via the module system on Arrhenius. One exception is the access to the Cray programming environments, which need an extra step due to them being encapsulated in software containers on Arrhenius. Inside of these containers is a module system which can be used like on the outside, only with different content native to that particular container.

There are two principal (but synonymous) names to programming environments on Arrhenius, buildenv- and PrgEnv-, where the latter is the name used to identify build/programming environments offered by HPE Cray, it is the name under which they offer their build and development environments. The former name indicate an environment provided by NAISS.

If you have no specific knowledge or requirements on what build environment you need, you are advised to choose buildenv-intel for the CPU partition and buildenv-nvhpc for the GPU partition. Use the latest version available. Otherwise choose a programming environment based on the particular needs of the software you are building. For instance:

• If the software benefits from using a certain compiler, choose a buildenv containing it
• If the software benefits from the use of an especially fast MPI implementation for the machine, choose Cray MPICH from a CPE container
• If the software requires (or benefits highly from using) a special set of numerical libraries choose a buildenv containing that set.

A solid choice to get started is buildenv-intel on the CPU partition and buildenv-nvhpc on the GPU partition. The CPE choice is a very good choice on both partitions, but requires some extra steps to use.

The detailed content of the different build environments are listed in the NAISS Provided Build Environments and Cray Programming Environments sections below.

NAISS Provided Build Environments¶

The NAISS provided build environments are at the time of writing these

• CPU partition
  • buildenv-intel/2023a-eb. Content predominantly from Intel oneAPI release 2023.1
    • Intel Classic compilers
    • Intel MPI 2021.9
    • Intel MKL for BLAS/LAPACK, FFT(W), ScaLAPACK
  • buildenv-intel/2025b-eb. Content predominantly from Intel oneAPI release 2025.2
    • Intel "modern" compilers
    • Intel MPI 2021.16
    • Intel MKL for BLAS/LAPACK, FFT(W), ScaLAPACK
  • buildenv-gcc/2025b-eb Content based on the foss25b EasyBuild toolchain
    • GCC 14.3.0
    • OpenMPI 5.0.8
    • FlexiBLAS, FFTW 3.3.10 and ScaLAPACK 2.2.2
  • buildenv-gcc/2026.03
    • GCC 14.3.0
    • MPICH 4.3.2
    • OpenBLAS 0.3.32, FFTW 3.3.11 and ScaLAPACK 2.2.3
• GPU partition
  • GPU/buildenv-nvhpc/25.9-cu13.0
    • NVHPC 25.9
    • CUDA 13.0
    • OpenMPI 5.0.10
    • NCCL 2.27.7 (+ AWS ofi plugin 1.19.2)
    • NVSHMEM 3.4.5
    • NVPL for BLAS/LAPACK, ScaLAPACK and FFT
    • cuFFTMp 11.4.0
  • GPU/buildenv-gcccuda/2026.03-cu13.0
    • GCC 14.3.0
    • CUDA 13.0.3
    • MPICH 4.3.2
    • NCCL 2.29.7 (+ AWS ofi plugin 1.19.2)
    • NVSHMEM 3.4.5
    • OpenBLAS 0.3.32, FFTW 3.3.11 and ScaLAPACK 2.2.2
    • cuFFTMp 12.1.3

All of the above environments can be loaded (module load <env>) and used as you would expect on an HPC system. Applications built with them can be launched on the cluster using launchers mpprun, Slurm's srun and the MPI native launchers (mpiexec/mpirun et. al.).

Cray Programming Environments¶

The Cray programming environments (CPE) on Arrhenius are offered in containers, specifically using the apptainer/singularity container format and runtime, which is generally available without needing to load a module. As a consequence of using an apptainer when building applications, there is an initial step where you enter an interactive shell session in the container, which is initialized in a way compliant with the CPE requirements. The base operating system of the container is the same as that of the login and compute nodes ensuring compatibility.

Applications built with the CPE must also be executed on the cluster using the same CPE version used for the build, see the Running Applications Built in the Cray Programming Environment section. The mpprun launcher wrapper provided on Arrhenius simplifies the parallel launch process for applications built in the CPE container environments, in essence you launch your application with the command mpprun --container <container_name> <application>. However, note that Slurm is not functional on the inside of the containers, that is, srun, salloc and similar Slurm commands will not work when in a container shell.

For general documentation detailing the use of CPE, please see the official documentation at https://cpe.ext.hpe.com/docs/latest/index.html.

Building applications in the Cray programming environment¶

Using the CPEs on Arrhenius typically involves these steps

1. Access a shell in the CPE container using the hpc_container_shell <container> command
2. Load appropriate modules for your requirements in the container shell.
   • The programming environment loaded by default in the shell you enter is PrgEnv-cray.
   • Common other choices would be PrgEnv-gnu and on the GPU-nodes additionally PrgEnv-nvidia.
3. Build your application and its requirements according to their documentation as you would do normally.
4. Exit the container with exit or Ctrl-d as you would any shell.
5. Run the CPE-built software.

To illustrate, first list the available Cray programming environments using the hpc_container_shell command

[<user>@arrhenius ~]$ hpc_container_shell

Usage: hpc_container_shell <container> [--reinit] [-m <module> [-m <module> ...]]

Arguments:
  <container> the name of the container to use (see below)
  --reinit    reinitialize module environment for program running inside the container (forgets PATH settings, etc.)
  -u          Don't clean environment
  -m          load modules inside the container (forces --reinit).
              To list available modules run: hpc_container_shell <container> --reinit bash -c 'module avail'

List of containers available in the current environment:

* cpe_25_09: Cray Programming Environment (CPE) v25.09

Loading one of the available CPEs listed will land you in a container having NAISS additions/enhancements added to the standard CPE experience allowing, among other things, simplified run-time launching of your applications built and an integrated EasyBuild system providing additional software titles and the opportunity for users to extend the software catalogue with their own titles.

[<user>@arrhenius ~]$ hpc_container_shell cpe_25_09
HPC_NOTE: hpc_container_shell running container cpe_25_09

APPTAINER[cpe_25_09] /home/exampleuser> module avail
HPC_NOTE: hpc_container_shell running container cpe_25_09

-------------------------------------------------------------------------------------- /software/sse2/el9_cpe_25_09/modules --------------------------------------------------------------------------------------
   Elk/recommendation              (D)    Python/recommendation (D)    buildenv-cpe_cray/recommendation (D)    buildtool-easybuild/5.2.1-hpca3ef7d197
   Elk/8.5.2-hpc1-cpeCray-20.09-eb        Python/3.14.2-hpc1           buildenv-cpe_cray/25.09-eb              testme/1.0

-------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mpi/crayclang/20.0/ofi/1.0/cray-mpich/8.0 ---------------------------------------------------------------------
   cray-hdf5-parallel/1.14.3.7    cray-mpixlate/1.0.9    cray-parallel-netcdf/1.12.3.19

-------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/perftools/25.09.0 ---------------------------------------------------------------------------------
   perftools-lite-events    perftools-lite-gpu    perftools-lite-hbm    perftools-lite-loops    perftools-lite    perftools-preload    perftools

--------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/cpu/x86-rome/1.0 ---------------------------------------------------------------------------------
   cray-fftw/3.3.10.11

-------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/comnet/crayclang/20.0/ofi/1.0 ---------------------------------------------------------------------------
   cray-mpich-abi/8.1.33    cray-mpich/8.1.33 (L)

----------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/net/ofi/1.0 ------------------------------------------------------------------------------------
   cray-openshmemx/11.8.0

----------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/compiler/crayclang/20.0 ------------------------------------------------------------------------------
   cray-hdf5/1.14.3.7    cray-libsci/25.09.0 (L)

---------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mix_compilers -----------------------------------------------------------------------------------
   cce-mixed/20.0.0    gcc-native-mixed/14

------------------------------------------------------------------------------ /opt/cray/pe/lmod/modulefiles/craype-targets/default ------------------------------------------------------------------------------
   craype-accel-amd-gfx908    craype-accel-host         craype-accel-nvidia90    craype-hugepages32M         craype-network-ucx    craype-x86-rome    (L)    craype-x86-turin
   craype-accel-amd-gfx90a    craype-accel-intel-max    craype-arm-grace         craype-hugepages512M        craype-x86-genoa      craype-x86-spr-hbm
   craype-accel-amd-gfx940    craype-accel-nvidia70     craype-hugepages1G       craype-network-none         craype-x86-milan-x    craype-x86-spr
   craype-accel-amd-gfx942    craype-accel-nvidia80     craype-hugepages2M       craype-network-ofi   (L)    craype-x86-milan      craype-x86-trento

--------------------------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/core ---------------------------------------------------------------------------------------
   PrgEnv-cray/8.6.0 (L)    cce/20.0.0    (L)    cray-ccdb/5.0.7         cray-dyninst/12.3.6    cray-zmqnet/1.3.2        gdb4hpc/4.16.5                sanitizers4hpc/1.1.6
   PrgEnv-gnu/8.6.0         cpe/25.09_fix        cray-cti/2.20.0         cray-mrnet/5.1.6       craype/2.7.35     (L)    papi/7.2.0.2                  valgrind4hpc/2.13.6
   atp/3.15.7               cpe/25.09     (D)    cray-dsmml/0.3.1 (L)    cray-stat/4.12.6       gcc-native/14            perftools-base/25.09.0 (L)

------------------------------------------------------------------------------------ /opt/cray/pe/lmod/lmod/modulefiles/Core -------------------------------------------------------------------------------------
   StdEnv (L)    lmod    settarg

--------------------------------------------------------------------------------------------- /opt/cray/modulefiles ----------------------------------------------------------------------------------------------
   xpmem/1.0.1-1.4_gf67ddfb318d4 (L)

------------------------------------------------------------ This is a list of module extensions. Use "module --nx avail ..." to not show extensions. ------------------------------------------------------------
    flit_core (E)     packaging (E)     pip (E)     setuptools (E)     setuptools_scm (E)     tomli (E)     typing_extensions (E)     wheel (E)

The standard set of modules loaded can be seen using the module list command directly upon entering the container.

APPTAINER[cpe_25_09] /home/exampleuser> module list

Currently Loaded Modules:
  1) cce/20.0.0           3) craype/2.7.35      5) PrgEnv-cray/8.6.0   7) xpmem/1.0.1-1.4_gf67ddfb318d4   9) perftools-base/25.09.0  11) StdEnv               13) hpc/.1.14.2-12-gae4120e (H,S)
  2) craype-network-ofi   4) cray-dsmml/0.3.1   6) craype-x86-rome     8) cray-libsci/25.09.0            10) cray-mpich/8.1.33       12) hpc_sysenv/.1 (H,S)

  Where:
   S:  Module is Sticky, requires --force to unload or purge
   H:             Hidden Module

Running Applications Built in the Cray Programming Environment¶

The recommended way to run CPE built software in a Slurm allocation is by using the mpprun launcher wrapper with the --container <container_name> switch. The application is ultimately launched by something like srun <srun_switches> apptainer exec <apptainer_switches> <container> <your_application>, but mpprun keeps track of some metadata and switches needed to make the experience smooth. An example script for running a job via the batch queue can look like

#!/bin/bash
#SBATCH -p cpu
#SBATCH -t 10
#SBATCH -n 64
#SBATCH -c 8

mpprun --container cpe_25_09 osu_mbw_mr

Be aware that the container used at run time must be the same as the one used at build time.

Building applications¶

As always when building applications, ensure you have access to the software and hardware environment required to fulfil the specific dependencies of the application being built. On Arrhenius this general approach is advised

1. Build your application in a hardware environment compatible with the software environment you intend to use. Modules with the GPU/ name prefix can only work on a GPU node, so you need to work in a GPU node allocation when building software intended to be used on the GPU partition. It is suggested you get into such an allocation using some variant of the interactive -p gpu --gpus 1 <other_salloc_args> command. The interactive command is a wrapper around salloc+srun which lands you in a shell on a physical node of the cluster. The CPU nodes are architecturally identical to the login node, so you can build CPU applications directly on the login node and later run on the CPU partition.
2. Once in an appropriate hardware context, load a buildenv module and any build time supporting modules containing tools and libraries such as for instance CMake, HDF5 etc. You can also rely on the host OS tooling available.
3. Ensure that application performance critical libraries/software are built as part of the application build process, or are loaded as a module, or are present in the environment by some other means (e.g. setting environment variables by hand). Preference should in most cases be given to using libraries and software provided via the module system. Search for these with module avail after you've loaded your buildenv module.
4. Build your application according to the instructions given by the software provider/developers. For MPI applications, typically use the MPI compiler wrappers (mpicc, mpicxx et. al.) as compilers, but for CPE instead use the HPE Cray compiler drivers primarily.
5. The code compilation process on Arrhenius builds into produced binaries all search paths to the libraries required for the software to run. This information is then be used by the parallel launch tool mpprun provided on Arrhenius to launch the parallel application.

Pre-built software titles¶

Often, parallelized and GPU-enabled scientific applications are complex to build and run, offering an inexperienced user plenty opportunity to build and install flawed or low-performing applications. To address this difficulty, Arrhenius offers a catalogue of pre-built scientific applications built by NAISS' domain application experts or other external experts, as the case is for EESSI software. The catalogue of installed software aims to make the use of the system both as simple as possible and as efficient as possible.

Interrogating which software titles are available can be done by browsing the Arrhenius online documentation or the output of module avail on the command line on Arrhenius. Running centrally installed software titles can be done with the mpprun launcher.

Running applications¶

All software built on Arrhenius by means of a buildenv- or PrgEnv- published centrally can be run in a Slurm resource allocation using the mpprun command, as well as with srun or the parallel framework native launcher (e.g. mpirun, mpiexec.hydra). NAISS recommends using mpprun to launch parallel applications as it uses configuration metadata for that particular MPI installation/implementation to make the application launch correctly. It also means there is no need to keep track of which build environment to load at launch time, this is resolved by mpprun using information from the application binary. For more full information on mpprun's capabilities, issue a mpprun --help on the command line.

Example Workflow Building and Running a Simple Application¶

The example below is appropriate for the CPU partition, for which you can build an application on a login node. It assumes you're logged in to the system and work on the command line. An MPI micro benchmark code is used for the example.

$ module load buildenv-intel/2025b-eb
$ wget https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.5.2.tar.gz
$ tar xf osu-micro-benchmarks-7.5.2.tar.gz
$ cd osu-micro-benchmarks-7.5.2
$ ./configure --prefix=/example/install/prefix CC=mpicc CXX=mpicxx
  8<---- output text omitted ---->8
$ make && make install
  8<---- output text omitted ---->8
## Put example benchmark on the search PATH ##
$ export PATH=/example/install/prefix/libexec/osu-micro-benchmarks/mpi/collective:$PATH
$ salloc -n 512 -t 10 # This allocates two CPU nodes
$ mpprun osu_allreduce
mpprun INFO: mpprun v1.14.2-12-gae4120e started
mpprun INFO: mpprun srun handler invoked on: /example/install/prefix/libexec/osu-micro-benchmarks/mpi/collective/osu_allreduce
mpprun INFO: Setting env variable: FI_PROVIDER=cxi
mpprun INFO: Setting env variable: SLURM_KILL_BAD_EXIT=1 (was unset; to override, set to 0 before invoking mpprun)
mpprun INFO: none of the known threading env variables set, all set by mpprun: OPENBLAS_NUM_THREADS=1, MKL_NUM_THREADS=1, NUMEXPR_NUM_THREADS=1, OMP_NUM_THREADS=1
mpprun INFO: mpprun v1.14.2-12-gae4120e executing: /usr/bin/srun --mpi pmi2 -m block:block /example/install/prefix/libexec/osu-micro-benchmarks/mpi/collective/osu_allreduce

# OSU MPI Allreduce Latency Test v7.5.2
# Datatype: MPI_INT.
# Size       Avg Latency(us)
4                       5.09
8                       5.06
16                      5.34
32                      5.36
64                      6.48
128                     7.24
256                     9.87
512                    11.25
1024                   14.01
2048                   20.35
4096                   32.99
8192                   21.47
16384                  23.70
32768                  26.63
65536                  33.51
131072                 50.79
262144                 82.98
524288                196.50
1048576               287.81
$ exit
exit
salloc: Relinquishing job allocation 29
salloc: Job allocation 29 has been revoked.

As can be seen, mpprun by default outputs a bit of information on how the launch was carried out. In this case, it sets a few environment variables and then launches the application using srun from Slurm. Most often mpprun can also be made to use the MPI-implementation's native launcher as well, which is sometimes useful. Nothing stops you from using srun directly, which is very often useful when doing specialized parallel launch invocations. The mpprun tool is merely a convenience for the most common use cases and settings.

Containers¶

Apptainer is available on Arrhenius to run and build containers. With apptainer containers you package your software and all of its dependencies into a single file. Making it a good alternative for distributed storage filesystems such as LustreFS which is used for project storage on Arrhenius.

If have a container from previously it is likely it wasn't built for the aarch64 CPU-architecture on the GPU-partition. If you want it for the GPU-partition you will need to rebuild it.

To get the recipe of your existing container you can run the inspect command:

apptainer inspect -d my_container.sif > my_recipe.def

Building containers¶

It is important that you build your container on the right node type. If you are going to use the container for the CPU partition, build on a CPU node and if you are going to use it for the GPU partition build it in a job on the GPU partition.

1. Get an interactive job interactive -A <... and other SLURM arguments>.
2. Build your container apptainer build my_container.sif my_recipe.def.
3. (If needed). If your build fails and you need to debug, you can use a sandbox on local disk.
   1. apptainer build --sandbox /tmp/my_sandbox.sif my_partial_recipe.def (don't use project storage or homedir)
   2. apptainer shell --writable /tmp/my_sandbox.sif/
   3. investigate the build interactively to learn what you need to fix your recipe
   4. build your container without sandbox on project storage

Multi-node¶

To use the fast slingshot interconnect to communicate between nodes your container needs to have been built to use slingshot capable libfabric.

We are currently looking at the easiest way to modify any existing container to enable it to make use of the slingshot interconnect. If internode communication is important for your application, please contact support through the support form in SUPR:

• https://supr.naiss.se/support/?centre_resource=r101

Submit a batch job to the queue¶

Arrhenius is a cluster using the Slurm resource manager. Being a Slurm controlled cluster means the generic Slurm documentation also applies here, and you can find out more by reading the canonical source of information on Slurm

If you are unfamiliar with Slurm here are some short pointers and some terminology to get you started

• Putting a job in the batch queue means submitting a script to the queue using the sbatch <script_name> command.
• Allocating compute resources is done using the Slurm commands salloc or sbatch.
• Launching applications in a resource allocation is preferentially done using either the mpprun or srun commands. The mpprun command is a wrapper around srun to simplify launching applications.
• Batch jobs are contained in bash scripts containing Slurm metadata directives (i.e. lines beginning with #SBATCH) read by Slurm to allocate compute resources.
• All resource request specifications can be set in the job script or on the command line, where the command line switches take precedence over the script.
• The Slurm configuration allows sharing nodes between users. You are in fact encouraged to only allocate the resources you need using the principal resource allocation pattern -n <num_tasks> -c <cpu_cores_per_task> on the CPU nodes and -n <num_tasks> -c <cpu_cores_per_task> --gpus <num_tasks> on the GPU nodes. Don't allocate full nodes unless you know you need it.
• The sinfo and squeue commands will respectively give you information about cluster resource occupation status and what the batch queue looks like.

Slurm Partitions¶

A Slurm partition is a collection of compute resources sharing a certain feature. The names of the partitions in Arrhenius are cpu, gpu and fat, indicating the type of compute resource they contain. The partition fat indicates CPU nodes with large memory. Allocating resources in a specific partition is done using the flag -p <partition_name> to your salloc/sbatch command.

You also need to supply your project account name using the switch -A <project_name> to your resource allocation (check your project name with the projinfo command). To get access to the GPU nodes, the project account name must end with a -gpu and correspondingly for the CPU nodes, it must end with a -cpu. The default partition is the cpu partition. All project account names ending with -cpu have access to this partition and the fat partition.

The walltime limit on Arrhenius' partitions can at any time be checked with the Slurm command sinfo. It is at the time of writing set to 3 days (72h).

Batch Job Scripts¶

Loosely following a previous example, below you can find example batch job scripts

CPU Batch Job Script¶

#!/bin/bash
#SBATCH -n 512
#SBATCH -c 1
#SBATCH -t 10 # Requested walltime
#SBATCH -A naiss<project_number>-cpu
#SBATCH -p cpu

# Just an example. Could be replaced with a "module load <relevant_module>" when such are available.
export PATH=/example/install/prefix/libexec/osu-micro-benchmarks/mpi/collective:$PATH
mpprun osu_allreduce

GPU Batch Job Script¶

#!/bin/bash
#SBATCH -n 8
#SBATCH -c 72
#SBATCH --gpus 8
#SBATCH -t 10 # Requested walltime
#SBATCH -A naiss<project_number>-gpu
#SBATCH -p gpu

# Just an example. Could be replaced with a "module load <relevant_module>" when such are available.
export PATH=/example/install/prefix/libexec/osu-micro-benchmarks/mpi/collective:$PATH
mpprun osu_allreduce

Migration guide¶

The NAISS systems Tetralith, Alvis and Dardel will be retired.

Users and projects from Tetralith, Alvis and Dardel will be migrated to Arrhenius.

For Tetralith and Alvis users and projects this process will happen during summer 2026, for Dardel later autumn 2026.

The main steps of migrating to Arrhenius are:

• Getting a storage and compute allocation on Arrhenius for the project
• Getting a personal login account on Arrhenius
• Testing your applications on Arrhenius
• Moving data from the project storage directory to Arrhenius
• Moving data from user's home directories to Arrhenius

These steps will be broadly similar no matter which system you are migrating from, but there will be minor differences. Please read the "When moving from XXXX" sections below carefully for the system you are migrating from.

When will the migration happen?¶

The HPC part of Arrhenius (CPU, GPU and HPC storage) became available to users (existing Tetralith and Alvis project as well as new Arrhenius projects) on 2026-06-01.

Tetralith and Alvis are scheduled to be shut down (as NAISS resources) on 2026-09-30. The shutdown date for Dardel is 2026-12-31. Dardel projects will get access to Arrhenius during 2026H2.

The size of Tetralith and Alvis (number of available compute nodes) will be reduced. Tentative start dates: 2026-06-10 for Tetralith and 2026-07-01 for Alvis.

We recommend that you start testing Arrhenius as soon as you get access and move your work to Arrhenius as soon as possible.

Arrhenius will not share any file systems with Tetralith, Dardel and Alvis. You will not be able to directly access files from project storage or home directories on Tetralith from Arrhenius and vice versa.

Getting a storage and compute allocation on Arrhenius for your project¶

Projects with an allocation on Tetralith or Alvis that ends after June 1^st 2026 automatically got an allocation on Arrhenius for the remaining time of the project.

Existing Dardel projects with an end date after the planned Dardel shutdown date will also get projects on Arrhenius automatically later on.

The size of the new allocation is scaled so that all projects from Tetralith, Alvis and Dardel (the systems that will be replaced by Arrhenius) will fit into the HPC part (there are also two much smaller sensitive data and persistent compute parts) of Arrhenius.

Removing "cold" data from Alvis¶

See the Alvis data migration page

Removing "cold" data from Tetralith¶

On Tetralith, you can use the command storagereport to get an overview of how much "cold" data the project has (data that has not been accessed in a long time). Such data should not be stored on Tetralith or Arrhenius, and you will likely need to remove some of it before your data will fit into your storage allocation on Arrhenius.

The PI or proxy for the PI can request assistance from NSC Support to access or remove user's data in the project storage directory.

Getting a personal login account on Arrhenius¶

All users who will be using Arrhenius need to request a login account in SUPR. When you do this, you will get to choose your username, password, setup two-factor authentication, etc.

This is described in more detail near the top of this page.

Testing your applications on Arrhenius¶

We recommend that projects login and get familiar with Arrhenius as soon as they get access. This includes reading the documentation, making sure you can login, transfer files, check if the applications you need are already installed, etc.

We recommend that you then copy or move a small amount of data (see "Transfer of files" on this page) to Arrhenius and run some test jobs. If you run into problems, do not hesitate to contact NAISS Support.

When you are happy with how Arrhenius works, it is time to move your data and start using Arrhenius instead of the previous system.

When migrating from Alvis¶

See the Alvis data migration page

When migrating from Tetralith¶

There will be a period of time where both Tetralith and Arrhenius are available (2026-06-01 to 2026-09-30). The project PI can decide when during this period the project will move to Arrhenius.

This overlap period is intended to make the move to Arrhenius easier, not to provide extra computing resources. We expect that projects that have migrated to Arrhenius will stop using Tetralith. The size of Tetralith will be gradually reduced during the overlap period, both to save on operating costs and to encourage migration.

NAISS recommends that projects move as early as possible. The main reason is that the later you move, the less time is available to fix any problems you might encounter on Arrhenius until you can no longer use Tetralith.

Moving data from the project storage directory to Arrhenius¶

When migrating from Alvis¶

See the Alvis data migration page

When migrating from Tetralith¶

When the project is ready to move, the project PI needs to decide what data from the project storage directory (e.g /proj/DIRECTORYNAME) should be moved to Arrhenius and when.

NAISS strongly recommends that projects request that NAISS move the data.

Experience from previous migrations show that it is very easy to make mistakes, fail to transfer data you cannot read, people are on parental leave, etc.

The PI (or another person the PI delegates this task to) can request a data transfer from NAISS using the command storagemigrate on Tetralith (recommended) or by contacting NSC Support. You will then be asked to provide the "what" and "when" for the data transfer.

We will do our best to transfer your data on the date you have chosen. If we need to change the schedule we will inform the person that made the request.

You can change or cancel your request by simply re-running storagemigrate later. Around 7 days before the requested date, we will start a preliminary transfer of data (to speed up the final one). This data is transferred in the background but will show up in the storagequota output on Arrhenius. Once the background transfer has started you will need to contact NSC Support to change your request.

storagemigrate will warn you if you request a date when other projects have already scheduled a large amount of data to be transferred. If you choose that date anyway your transfer is more likely to be delayed. Projects are transferred in the order they submitted their request.

You can use the command storagemigrate_overview to see which days other projects have already requested transfers. If possible, try to request your transfer for a day that is less busy, it will speed up your data transfer. storagemigrate_overview will also show ongoing transfers so you can use it to see if the transfer has started.

We will send an email notification to the person that submitted the transfer request when the transfer starts and when it is completed.

The options for what the project wants NAISS to transfer are:

• Request a full move - NAISS moves the entire project storage directory.

• Request a partial move - NAISS moves ONLY the data you have put into the "to-arrhenius" subdirectory (e.g /proj/someproject/to-arrhenius) This can be useful if your data does not fit into your storage allocation on Arrhenius and you want to move most of the data to Arrhenius now but leave some on Tetralith to be handled later. You need to create the to-arrhenius directory.

• No move - the project will move its own data

• No move - the project has no data that needs to move

(the last two options will be handled in the same way by NAISS, we just want to know the reason for why you did not want any data to be moved)

The options for when the project wants the transfer to happen are:

• Starting as soon as possible (if no other transfer is ongoing the transfer should start within 10 minutes)

• Starting on a fixed date, e.g "2026-07-14" means "starting 2026-07-14 as soon as possible after 00:00 CEST"

• Starting the day after Tetralith is shut down (as this is planned for 2026-09-30, the transfer will start as soon as possible after 2026-10-01 00:00 CEST.

As the system is automated transfers can start at any time including weekends, but staff will only be able to handle problems during normal working hours.

If a project has not requested a transfer or a "no move" by the time Tetralith is shut down, NAISS will move the entire project storage directory to Arrhenius. This is done so that projects who do not read this document (parental leave, etc.) will not lose data.

Each project can only request one data transfer.

NAISS will only move data, not copy it. If you absolutely need a writable copy left on Tetralith, contact NSC Support to discuss your options.

Data will be moved to the from-tetralith subdirectory of your Arrhenius project storage directory, e.g /proj/DIRECTORYNAME on Tetralith will be moved to /nobackup/proj/disk/DIRECTORYNAME/from-tetralith. This is done so we don't overwrite data you have already created on Arrhenius or data that is being migrated from Alvis. You can then move the data from from-tetralith to another location in the project storage directory tree, or leave it there.

Due to the large number of projects to be transferred we have a very limited ability to accommodate any custom requests outside of the options described above. If none of the above options work for you, contact NSC Support.

Since most projects will get a smaller storage allocation on Arrhenius, most projects will need to remove or move some data before the data can be transferred.

It is the project's responsibility to make sure that the data you request a transfer for will fit into your storage allocation on Arrhenius!

If the data you requested a transfer for will not fit into your storage allocation on Arrhenius the transfer may fail or you may end up being over quota on Arrhenius.

When the data transfer begins, the data becomes inaccessible on Tetralith. When the data transfer is complete, the data will become accessible on Arrhenius. I.e you will never see partial data anywhere and you will not be able to access the data while the transfer is in progress.

The transfer will take some time, from minutes for small projects to maybe a day or more for the largest ones.

The transfer will normally start at 00:00 CEST on the day requested, but if many projects choose the same day, we will not start all at the same time. In this case, transfers are started in the order the transfer request was made. So if you still see your data on Tetralith at 01:00, that is not necessarily an error.

If you request a partial transfer of data and leave some data behind on Tetralith after the shutdown date, you will then have 14 days to retrieve this data (i.e until 2026-10-14). NAISS will then inform the PI's university, give them 14 days to retrieve the data and then delete any remaining data. This is an accelerated version of the normal NAISS Data Decommissioning process

Moving data from user's home directories to Arrhenius¶

When migrating from Alvis¶

See the Alvis data migration page

When migrating from Tetralith¶

Data in a user's home directory (/home/USERNAME) will not be automatically transferred to Arrhenius.

The main reason for this is that many users are members of several projects that will probably not move to Arrhenius at the same time, so NAISS cannot know when we should move a specific home directory.

Instead, a command copy_home_directory_to_arrhenius is available that you can run on Tetralith that will copy the contents of your home directory to Arrhenius. To ensure that you can still use Tetralith afterwards, the home directory is copied, not moved.

The data will be copied to a subdirectory "from-tetralith" in the Arrhenius home directory in order not to overwrite any Arrhenius settings or data that might already be there.

The script will ask for confirmation before doing anything and show you the rsync command that will be used.

If you leave any data in your home directory on Tetralith, you will have 14 days after the Tetralith shutdown date to move that elsewhere, then anything that remains will be deleted.

References¶