Santis¶

Santis is the main Climate and Weather Platform cluster that provides compute nodes and file systems for GPU-enabled climate and weather workloads.

Cluster specification¶

Compute nodes¶

Santis consists of 302 Grace-Hopper nodes.

The number of nodes can vary as nodes are added or removed from other clusters on Alps. See the Slurm documentation for information on how to check the number of nodes.

There are two login nodes, santis-ln00[1,2], which are repurposed compute nodes from the pool of 302 GH200 nodes. You will be assigned to one of the two login nodes when you ssh onto the system, from where you can edit files, compile applications and launch batch jobs. This leaves 300 nodes available for compute jobs.

Storage and file systems¶

Santis uses the CWP filesystems and storage policies.

Getting started¶

Logging into Santis¶

To connect to Santis via SSH, first refer to the ssh guide.

Host santis
    HostName santis.alps.cscs.ch
    ProxyJump ela
    User cscsusername
    IdentityFile ~/.ssh/cscs-key
    IdentitiesOnly yes

Software¶

uenv¶

Santis provides uenv to deliver programming environments and application software for the climate and weather community. Please refer to the uenv documentation for detailed information on how to use the uenv tools on the system.

To list all uenv that are available on Santis, run uenv image find.

# list all images available for daint
uenv image find @daint

# download an image for daint
uenv image pull namd/3.0:v3@daint

# start the uenv
uenv start --view=namd namd/3.0:v3@daint

Containers¶

Santis supports container workloads using the container engine.

To build images, see the guide to building container images on Alps.

Running jobs on Santis¶

Slurm¶

Santis uses Slurm as the workload manager, which is used to launch and monitor compute-intensive workloads.

There are four Slurm partitions on the system:

Slurm uses the default time when a job does not set --time.

Node sharing and GPU requests¶

All GH200 partitions on Santis use node sharing.

By default MPI ranks on Santis are given a single CPU core and no GPUs.

$ srun --ntasks=2 --account=p1234 ./affinity.cuda
GPU affinity test for 2 MPI ranks
rank      0 @ nid005110
 cores   : 0
rank      1 @ nid005110
 cores   : 72

Slurm is configured to allocate resources in units of one GH200 chip for each GPU that the job requests. The GPUs assigned to each rank are configured using the --gpus-per-task and --gpus-per-node slurm flags:

• 1 GPU is a whole Grace-Hopper super-chip: 72 CPU cores, one GPU (96 GB HBM), and 115 GB host memory by default.

$ srun --ntasks=1 --gpus-per-task=1 --account=p1234 ./affinity.cuda
srun: job 898836 queued and waiting for resources
srun: job 898836 has been allocated resources
GPU affinity test for 1 MPI ranks
rank      0 @ nid005108
 cores   : [216:287]
 gpu   0 : GPU-cc201279-83e4-96d2-ad01-87b35f2a7bc7

$ srun --ntasks=2 --gpus-per-task=1 --account=p1234 ./affinity.cuda
GPU affinity test for 2 MPI ranks
rank      0 @ nid005012
 cores   : [0:71]
 gpu   0 : GPU-5b19f465-b587-4b9e-31c4-60bf65296733
rank      1 @ nid005012
 cores   : [72:143]
 gpu   0 : GPU-ac10eafa-4462-5be4-152e-7b92f9780b80

$ srun --ntasks=2 --gpus-per-node=4 --account=p1234 ./affinity.cuda
GPU affinity test for 2 MPI ranks
rank      0 @ nid005303
 cores   : [0:287]
 gpu   0 : GPU-29180fbd-a8f8-0502-2545-5787e427361b
 gpu   1 : GPU-47948342-2073-f024-84a1-db54c3084812
 gpu   2 : GPU-16d309ce-af9d-6814-032a-d5109f6809d9
 gpu   3 : GPU-3fa8aa36-b463-83be-edab-f7b281c0d359
rank      1 @ nid005303
 cores   : [0:287]
 gpu   0 : GPU-29180fbd-a8f8-0502-2545-5787e427361b
 gpu   1 : GPU-47948342-2073-f024-84a1-db54c3084812
 gpu   2 : GPU-16d309ce-af9d-6814-032a-d5109f6809d9
 gpu   3 : GPU-3fa8aa36-b463-83be-edab-f7b281c0d359

Multi-GPU jobs with P2P/IPC¶

Because device isolation is enforced via cgroups, multi-GPU jobs that rely on GPU P2P/IPC (for example MPI across GPUs on the same node) must add to the srun command (NB: this does not work being specified in the #SBATCH definition block):

--gres-flags=allow-task-sharing

This keeps all GPUs allocated to a job visible to all tasks of that job while preserving per-task CUDA_VISIBLE_DEVICES bindings. Without this flag, intra-node GPU-GPU communication will fail.

CPU Power Capping¶

This power capping feature allows to optimize power distribution on the GH200 nodes in favour of the GPUs for applications using CPU and GPU simultaneously. You can find more details and instructions how to activate the power capping in your runs here.

Low-priority overflow partition¶

The low partition is available to all users for work that should only run when the higher-priority partitions have idle capacity. It is intended for:

• quota-exhausted projects that still need some resources before the next quarterly reset
• work that does not need fast turnaround

Jobs in low have lower scheduling priority and are not preempted.

Concurrent job limit¶

Slurm limits the number of jobs that each user can run at the same time:

Jobs above the running limit wait in the queue until one of your running jobs finishes. In the low partition, Slurm rejects new submissions when you already have 20 jobs (running or waiting) in that partition.

To show the current limits, run:

$ sacctmgr show qos normal,slowdown format=name,maxjobspu,maxsubmitpu
      Name MaxJobsPU MaxSubmitPU
---------- --------- -----------
    normal        20
  slowdown        10          20

See the Slurm documentation for instructions on how to run jobs on the Grace-Hopper nodes.

FirecREST¶

Santis can also be accessed using FirecREST at the https://api.cscs.ch/cw/firecrest/v2 API endpoint.

Maintenance and status¶

Scheduled maintenance¶

One Wednesday per month is reserved for planned maintenance (usually around the middle of the month, 8-12 CET), with services potentially unavailable during this timeframe. If the queues must be drained (redeployment of node images, rebooting of compute nodes, etc) then a Slurm reservation will be in place that will prevent jobs from running into the maintenance window.

Exceptional and non-disruptive updates may happen outside this time frame and will be announced via the CSCS status page.

Change log¶

Known issues¶