Centrally provided datasets¶

To simplify for our researchers and saving space on the filesystem, we provide a number of datasets centrally on Arrhenius. You can find the available datasets by using module avail and looking at the modules under the Dataset modules header.

When you have loaded a module an environment variable with the path to the data should be made available to you. If the name isn't clear after loading the module, you can check with module show <MODULE> (replace <MODULE> with the name of the module).

Requesting more datasets¶

We consider adding any datasets which seems popular among our users and which is licensed such that we may provide it centrally. To make a request for a specific dataset contact support.

Gated datasets¶

Some datasets have licenses such that we can't provide them openly to everyone on the cluster. To get access to these datasets join the corresponding group on SUPR:

• https://supr.naiss.se/group/

When you have successfully joined a group, your group membership will be synced to the cluster within 10 minutes and you will have to log-out and log-in again to update your unix group memberships. You can check the groups you are member of with the command groups.

Special formats¶

When datasets are provided the format is chosen after weighing ease of use against performance and load on the filesystem. When a datasets consists of lots of small files an alternative format is chosen for this reason.

In case you want help with loading any particular dataset, contact support.

-hf-* format¶

The -hf format means that the files have been created using HuggingFace dataset package and saved using dataset.save_to_disk(). These can be used with the corresponding load_from_disk() function.

Here is an example for the module CIFAR-10-data/20240104-hf-2025b:

import os

from datasets import load_from_disk


ds = load_from_disk(os.environ['CIFAR_10_DATA_DIR'])
print(ds['train'])

The year/version looking suffix (e.g. 2025b) after -hf- is related to which HF-Datasets module was used to create this dataset. The same version does not need to be used when loading it.

-zip format¶

In these cases the files have been collected into one or a few uncompressed (-0) .zip files. The zip format contains a central directory which makes it reasonably efficient to read even in a random order and can often be read without any additional dependencies (such as with Python's zipfile standard library.)

Here is an example of how to use ImageNet-1k-data/20210311-zip with PyTorch's dataloader:

import io
import os
from pathlib import Path
from zipfile import ZipFile

from PIL import Image
from torch.utils.data import Dataset


class ImageNetDataset(Dataset):
    def __init__(self, dataroot: str, train: bool = True):
        dataroot = Path(dataroot)
        self.zfpath = dataroot / f"{'train' if train else 'val'}.zip"

        # Avoid reusing the file handle created here, for known issue with multi-worker:
        # https://discuss.pytorch.org/t/dataloader-with-zipfile-failed/42795
        self.zf = None
        with ZipFile(self.zfpath) as zf:
            self.imglist: list[str] = [
                name for name in zf.namelist()
                if name.endswith(".jpg")
            ]

        # Images are structured in directories based on class
        with open(dataroot / "devkit" / "data" / "map_clsloc.txt") as f:
            def parse_row(row: str) -> tuple[str, int]:
                classname, classnum, _ = row.split()
                return classname, (int(classnum) - 1)
            self.classes: dict[str, int] = dict(parse_row(row) for row in f)

    def get_label(self, imgpath: str) -> int:
        if not imgpath.endswith(".jpg"):
            raise ValueError(f"Expected path to image, got {imgpath}")
        classname: str = imgpath.split("/")[-2]
        return self.classes[classname]

    def __len__(self):
        return len(self.imglist)

    def __getitem__(self, idx: int) -> tuple[Image.Image, int]:
        if self.zf is None:
            self.zf=ZipFile(self.zfpath)

        imgpath = self.imglist[idx]
        try:
            img = Image.open(io.BytesIO(self.zf.read(imgpath)))
        except zipfile.BadZipfile:
            # It seems that sometimes the zipfile handle can become bad
            self.zf = ZipFile(self.zfpath)
            img = Image.open(io.BytesIO(self.zf.read(imgpath)))
        label = self.get_label(imgpath)
        return img, label


for train in [True, False]:
    dataset = ImageNetDataset(os.environ['IMAGENET_1K_DATA_DIR'], train=train)
    print(dataset[1032])