How to Run Jobs¶

Many researchers run their program on NAISS computer systems, often simultaneously. For this, the computer systems need workload management and job scheduling. For job scheduling Arrhenius uses Slurm Workload Manager .

When you login to the supercomputer with ssh, you login to a designated login node in your home directory. Here you can modify your scripts and manage your files.

To run your script/program on the computer nodes, you either write a job script describing what resources you need and submit this using the sbatch command, or run the script/program interactively.

How jobs are scheduled¶

The queue system uses two main methods to decide which jobs are run. These are called fair-share and backfill. Unlike some other centers, the time a job has been in the queue is not a factor.

The goal of the fair share algorithm is to make sure that all projects can use their fair share of the available resources within a reasonable time frame. The priority that a job (belonging to a particular project) is given will depend on how much of that project’s time quota has been used recently in relation to the quotas of jobs belonging to other projects - the effect of this on the priority declines gradually with a half-life of 14 days. So jobs submitted by projects that have not used much of their quota recently will be given high priority, and vice versa.

Backfill¶

As well as having a main queue to ensure that the systems are as full as possible, the job scheduling system also implements “backfill”. If the next job in the queue is large (i.e., it requires a significant number of nodes to run), the scheduler accumulates nodes as they become available until there are enough to start running the large job. Backfill means that the scheduler looks for smaller jobs that could start on nodes that are free now, and which would finish before there are enough nodes free for the large job to start. For backfill to work well, the scheduler needs to know how long jobs will take. So, to take advantage of the possibility of backfill, you should set the maximum time your job needs to run as accurately as possible in your submit scripts.

Warning

All researchers sharing a particular time allocation have the same priority. This means that if other people in your time project have used up lots of the allocated time recently, then any jobs you (or they) submit within that project will be given the same low priority.

Example of scheduling¶

Consider a couple of researchers, Anna and Björn with projects A and B. Of course both Anna and Björn would like their jobs to be run as soon as possible.

Assume now that project B has used less of its time allocation than project A. In this case, the scheduler will give priority to Björn’s job.

Even if Anna has not used any time herself, this does not make any difference as it is the total amount of time recently used by each project that is taken into consideration when deciding which job will be scheduled next.

Arrhenius partitions¶

A Slurm partition is a collection of compute resources sharing a certain feature. The names of the partitions in Arrhenius are cpu, gpu and fat, indicating the type of compute resource they contain. The partition fat indicates CPU nodes with large memory. Allocating resources in a specific partition is done using the flag -p <partition_name> to your salloc/sbatch command.

You also need to supply your project account name using the switch -A <project_name> to your resource allocation (check your project name with the projinfo command). To get access to the GPU nodes, the project account name must end with a -gpu and correspondingly for the CPU nodes, it must end with a -cpu. The default partition is the cpu partition. All project account names ending with -cpu have access to this partition and the fat partition.

The walltime limit on Arrhenius' partitions can at any time be checked with the Slurm command sinfo. If you want further information on the nodes listed you can use the command scontrol show node [nodename].

Shared partitions¶

Arrhenius does only have shared partitions meaning that several jobs for several users can run on the same node. Running on the shared partition is a little bit different than running on exclusive nodes. First you need to specify the number of cores you will be using, and, at the same time, you get an equivalent size of RAM for your job.

Defining the number of cores or memory¶

When running on shared nodes, you need to add the number of cores you will be using or the amount of memory you need. The amount of memory is equivalent to the amount of cores you are asking for and viceversa. For example, we are using a node with 256 logical cores (128 physical cores) and 256 Gbytes of RAM. If you are asking for 20 cores, you will receive 17 Gbytes of RAM. Instead of you are asking for 80 Gbytes of RAM, you will only receive 1 core but will be automatically billed for 94 cores. The cores or memory that is the largest for the job will dictate what is billed.

Parameters needed on shared nodes¶

Parameter	Description
`-N [nnodes]`	Allocates nnodes. Set this to 1 in order to force all cores to be on the same node
`-n [ntasks]`	Allocates ntasks
`--cpus-per-task=[cores]`	Allocates logical [cores]=ntasks*cpu-per-task. (Default: cpus-per-task=1)
`--mem=[RAM in Mbytes]`	The max amount of RAM allocated for your job

Example 1: On a shared node with 128 physical cores, 256 Gbytes RAM. In this case you will receive 20 logical cores, 17 GBytes RAM.

#SBATCH -N 1
#SBATCH --ntasks=10
#SBATCH --cpus-per-task=2

Example 2: On a shared node with 128 physical cores, 256 Gbytes RAM. In this case you will receive 1 logical core, 40 GBytes RAM. However you are billed for 48 logical cores, as you consume the amount of RAM for these cores.

#SBATCH -N 1
#SBATCH --ntasks=2
#SBATCH --mem=40G