Using uenvs¶
To use one, it must first be loaded into your running environment. There are three ways to use a uenv:
- Start a shell with a uenv environment for interactive use;
- Run a single command in a uenv, for use in scripts or in workflows that use software provided by more than one uenv.
- Configure a Slurm job to use a uenv on compute nodes.
How uenv works¶
A uenv provides full software stacks that provide specific applications, or programming/development environments. They are stored as SquashFS images, a compressed file that contains a directory tree. The SquashFS image of a uenv is a directory that contains all of the software provided by the uenv, along with useful meta data.
When you use uenv start, uenv run, or use the --uenv flag with Slurm, the SquashFS file is mounted at the mount location for the uenv, which is most often /user-environment.
The following example demonstrates how a uenv mounts the software at /user-environment:
# log into daint
$ ssh daint
# /user-environment is empty
$ ls -l /user-environment
total 0
# start a uenv
$ uenv start prgenv-nvfortran/24.11:v1
# the uenv software is now available
$ ls /user-environment/
bin config env linux-sles15-neoverse_v2 meta modules repo
# findmnt verifies that a squashfs image has been mounted
$ findmnt /user-environment
TARGET SOURCE FSTYPE OPTIONS
/user-environment /dev/loop25 squashfs ro,nosuid,nodev,relatime,errors=continue
# end the session and verify that the uenv is not longer mounted
$ exit
$ ls -l /user-environment
total 0
The software is available in the shell started by uenv start.
Note that the mounted software is only visible inside the uenv call: other users and sessions on the node do not see the mounted software at /user-environment - and they can mount different environments at the same time.
When the uenv was started above, the only change was the software stack mounted at /user-environment:
- the scratch, store, and home filesystems are still available;
- the environment variables are not modified.
As such, the environment is not necessarily very useful, because environment variables like PATH, LD_LIBRARY_PATH and CUDA_HOME are no set to make the software in the mounted path useable.
For example, using the uenv run command that executes a single command inside an environment before returning, we see that the same gcc 7 is detected inside the uenv - despite it providing gcc 14:
# the default gcc is installed in /usr/bin
$ which gcc
/usr/bin/gcc
$ gcc --version
gcc (SUSE Linux) 7.5.0
# inside the uenv we still see the system gcc
$ uenv run prgenv-gnu/25.6:v2 -- which gcc
/usr/bin/gcc
$ uenv run prgenv-gnu/25.6:v2 -- gcc --version
gcc (SUSE Linux) 7.5.0
To set environment variables, uenv provide views, which set additional environment variables inside the uenv:
# use the default view to make all software available
$ uenv run --view=default prgenv-gnu/25.6:v2 -- gcc --version
gcc (Spack GCC) 14.2.0
$ uenv run --view=default prgenv-gnu/25.6:v2 -- which gcc
/user-environment/env/default/bin/gcc
# use the modules view to make modules available
$ uenv run --view=modules prgenv-gnu/25.6:v2 -- bash -c 'module avail'
------- /user-environment/modules -------
aws-ofi-nccl/1.16.0 gcc/14.2.0
boost/1.88.0 gsl/2.8
...
Starting a uenv session¶
The uenv start command will start a new shell with one or more uenv images mounted.
This is very useful for interactive sessions, for example if you want to work in the terminal to compile an application, or set up a Python virtual environment.
start an interactive shell to compile an application
Here we want to compile an MPI + CUDA application “affinity”.
# start the prgenv-gnu uenv, which provides MPI, cuda and CMake
# use the "default" view, which will load all of the software in the uenv
$ uenv start prgenv-gnu/25.6:v2 --view=default
# clone the software and set up the build directory
$ git clone https://github.com/bcumming/affinity.git
$ mkdir -p affinity/build
$ cd affinity/build/
# configure the build with CMake, then call make to build
# mpi, gcc, cuda and cmake are all provided by the uenv
$ CXX=g++ CC=gcc cmake .. -DAFFINITY_GPU_BACKEND=cuda
$ make -j
# run the affinity executable on two nodes - note how the uenv is
# automatically loaded by Slurm on the compute nodes, which provides CUDA and MPI from
# the uenv so that the uenv can run.
$ srun -n2 -N2 ./affinity.cuda
GPU affinity test for 2 MPI ranks
rank 0 @ nid005636
cores : 0-287
gpu 0 : GPU-13a62579-bf3c-fb6b-667f-f2c588f4667b
gpu 1 : GPU-74968c03-7401-9013-0590-8445b3623208
gpu 2 : GPU-dfbd9ec1-a4b7-4a8d-603e-ebcc360f55a3
gpu 3 : GPU-6a44522d-bf84-9864-decf-6d3e85078442
rank 1 @ nid006322
cores : 0-287
gpu 0 : GPU-6d96b1d5-69e9-7bd4-f59a-a37ec1f5da1c
gpu 1 : GPU-c0508d69-a357-934e-87a0-be04adf4eee9
gpu 2 : GPU-02a7fd85-ff41-1d81-d010-d7a85f6134d8
gpu 3 : GPU-e07d996e-4d67-c9f4-cf75-81cfd45a1ae1
# finish the uenv session
$ exit
which shell is used?
uenv start starts a new shell, and by default it will use the default shell for the user.
You can see the default shell by looking at the $SHELL environment variable.
If you want to force a different shell:
C Shell / tcsh users
uenv is tested extensively with bash (the default shell), and zsh. C shell is not tested properly, and we will not make significant changes to uenv to maintain support for C shell.
If your are one of the handful of users using tcsh (C shell) and you want to use uenv, we strongly recommend creating a request at the CSCS service desk to change to either bash or zsh as your default.
Failed to unshare the mount namespace
If you get the following error message when starting a uenv:
$ uenv start linalg/24.11:v1
squashfs-mount: Failed to unshare the mount namespace: Operation not permitted
uenv status command will report that you have a uenv loaded if that is the case:
Unload the active uenv by exiting the current shell before loading the new uenv.
The basic syntax of uenv start is uenv start image where image is the uenv to start.
The image can be a label, the hash/id of the uenv, or a file:
uenv start
do not use uenv start in scripts or ~/.bashrc
The uenv start command is only for creating interactive environments, because it creates an interactive shell.
For Slurm jobs, and use inside scripts, use the uenv run and Slurm integration.
Because uenv start and uenv run execute commands in a new environment, they can’t be used in bashrc to configure your environment during login.
See our guide for creating convenient custom environments with uenv for alternatives to module load in your ~/.basrhc.
Running a uenv¶
The uenv run command can be used to run an application or script in a uenv environment, and return control to the calling shell when the command has finished running.
While the uenv start command is useful for interactive sessions and experimentation, the uenv run command is for use inside scripts, Slurm jobs and automated pipelines.
It can also be used to create customized environments. See the guide to creating custom environments for an example.
how is uenv run different from uenv start?
uenv start sets up the uenv environment, then starts an interactive shell in that environment.
When you are finished, you can type exit to finish the session.
uenv run is more generic - instead of running a shell in environment, it takes the executable and arguments to run in the shell.
The following commands are equivalent:
running commands from within a uenv
Call the cmake provided by the uenv to configure a build with the default view loaded:
Run the GROMACS executable from inside the gromacs uenv.
uenv run is useful for running multiple applications or scripts in a pipeline or workflow, where each application has separate requirements.
In this example the pre and post processing stages use prgenv-gnu, while the simulation stage uses the gromacs uenv.
# run multiple applications, one after the other, that have different requirements
$ uenv run --view=default prgenv-gnu/24.11:v1 -- ./pre-processing-script.sh
$ uenv run --view=gromacs gromacs/2024:v1 -- gmx_mpi $gromacs_args
$ uenv run --view=default prgenv-gnu/24.11:v1 -- ./post-processing-script.sh
using alias to create shortcuts
It can be verbose typing the full uenv run --view=... name/version:tag -- in front of each command.
If you want to use commands from a specific uenv frequently, create an alias:
# create a shortcut alias for running commands in prgenv-gnu
$ alias prg='uenv run --view=default prgenv-gnu/25.11:v1 --'
#
# tools like cmake and gcc can be called: the entire command will run inside
# the uenv: isolating the call
#
$ prg cmake --version
cmake version 3.31.9
$ prg which gcc
/user-environment/env/default/bin/gcc
If you have a command that you like to execute from a uenv, it can also be turned into an alias.
For example, if you want to use nvim from the editors uenv:
$ alias nvim='uenv run --view=ed editors/24.7 -- nvim'
# now edit a file using neovim from the uenv
$ nvim main.f90
aliases do not work inside scripts
Aliases are a useful for reducing keystrokes and usability in interactive shells. They do no work by default inside bash scripts, and have other quirks that mean you should prefer using functions instead in scripts.
Slurm integration¶
The environment to load can be provided directly to Slurm via three arguments:
--uenv: a comma-separated list of uenv to mount--view: a comma-separated list of views to load--repo: specify the repositories to search for the requested uenv. See the--repoflag docs for more information.--uenv-passthrough: uenv v10.0 configure how sbatch and srun load or ignore uenv environments that are already loaded.--no-default-view: uenv v10.0 disable default views.
For example, the flags can be used with srun:
# mount the uenv prgenv-gnu with the view named default
$ srun --uenv=prgenv-gnu/24.7:v3 --view=default ...
# mount an image at an explicit location (/user-tools)
$ srun --uenv=$IMAGES/myenv.squashfs:/user-tools ...
# mount multiple images: use a comma to separate the different options
$ srun --uenv=prgenv-gnu/24.7:v3,editors/24.7:v2 --view=default,editors:modules ...
The commands can also be used in sbatch scripts to have fine-grained control:
sbatch script for uenv
It is possible to provide a uenv that is loaded inside the script, and will be loaded by default by all srun commands that do not override it with their own --uenv parameters.
#!/bin/bash
#SBATCH --uenv=editors/24.7:v2
#SBATCH --view=editors:ed
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH --output=out-%j.out
#SBATCH --error=out-%j.out
echo "==== test in script ===="
# the fd command is provided by the ed view
# use it to inspect the meta data in the mounted image
fd . /user-tools/meta/recipe
echo "==== test in srun ===="
# use srun to launch the parallel job
srun -n4 bash -c 'echo $SLURM_PROCID on $(hostname): $(which emacs)'
echo "==== alternative mount ===="
srun -n4 --uenv=prgenv-gnu --view=prgenv-gnu:default \
bash -c 'echo $SLURM_PROCID on $(hostname): $(which mpicc)'
The sbatch job above would generate output like the following:
==== test in script ====
/user-tools/meta/recipe/compilers.yaml
/user-tools/meta/recipe/config.yaml
/user-tools/meta/recipe/environments.yaml
/user-tools/meta/recipe/modules.yaml
==== test in srun ====
1 on nid007144: /user-tools/env/ed/bin/emacs
3 on nid007144: /user-tools/env/ed/bin/emacs
0 on nid007144: /user-tools/env/ed/bin/emacs
2 on nid007144: /user-tools/env/ed/bin/emacs
==== alternative mount ====
0 on nid007144: /user-environment/env/default/bin/mpicc
1 on nid007144: /user-environment/env/default/bin/mpicc
2 on nid007144: /user-environment/env/default/bin/mpicc
3 on nid007144: /user-environment/env/default/bin/mpicc
In the example above, the #SBATCH --uenv and #SBATCH --view parameters in the preamble of the sbatch script set the default uenv to editors with the view ed.
editorsis mounted and the view set in the script (the “test in script” part)editorsis also mounted in the first call to srun (which does not provide a`–-uenvflag)
it is possible to override the default uenv by passing a different --uenv and --view flags to an srun call inside the script, as is done in the second srun call.
- Note how the second call has access to
mpicc, provided byprgenv-gnu.
Slurm environment variables¶
The SBATCH_UENV, SBATCH_UENV_VIEW, and SBATCH_UENV_REPO environment variables can be used to pre-configure the uenv for a batch job.
Useful in CI/CD pipelines or wrapper scripts where passing flags to sbatch is inconvenient:
The following example is equivalent to adding --uenv and --view to the sbatch call or inside the sbatch script.
export SBATCH_UENV=prgenv-gnu/25.6:v2
export SBATCH_UENV_VIEW=default
sbatch my-job.sh
These variables are cleared inside the job, and do not propagate to nested srun or sbatch calls.
Passthrough behaviour¶
uenv v10.0
The --uenv-passthrough flag controls how a uenv that is already loaded in the calling environment is treated when launching a Slurm step.
The passthrough option can take one of three values:
| value | meaning |
|---|---|
use |
forward the currently loaded uenv into the job (default for srun) |
ignore |
start the job with a clean environment, discarding the loaded uenv |
disable |
produce an error if a uenv is loaded and no --uenv flag is given (default for sbatch/salloc) |
srun passthrough behavior
By default, srun will recreate the uenv environment.
Below the prgenv-gnu uenv is mounted remotely, and the copy of gcc installed in the uenv is found
$ uenv start --view=default prgenv-gnu/26.3:v1
$ srun which gcc
/user-environment/env/default/bin/gcc
To ignore, use --passthrough=ignore option:
$ uenv start --view=default prgenv-gnu/26.3:v1
$ srun --uenv-passthrough=ignore which gcc
/usr/bin/gcc
Note how the default gcc in /usr/bin/gcc is found because the prgenv-gnu uenv is not mounted on the compute node in the srun call.
By default, sbatch will raise an error if it is run with a uenv already mounted.
sbatch default passthrough behavior
Calling sbatch from within a uenv environment is treated as an error because sbatch jobs should be isolated and reproducible.
As such, we want to avoid getting different results depending on whether a uenv was mounted when a job is launched.
Most of the time, the fix for this error is to not start uenv in a running uenv environment.
However, there are workflows and situations where sbatch must be called with a uenv mounted.
In these situations, the --uenv-passthrough flag can be used to explicitly state how to treat uenv in the calling environment:
using --uenv-passthrough with sbatch
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=4
#SBATCH --uenv=prgenv-gnu
#SBATCH --view=default
# this srun will use prgenv-gnu because srun defaults to --uenv-passthrough=use
srun ./a.out
# this srun will not use any uenv
srun --uenv-passthrough=ignore ./my.exe
# this srun ignores prgenv-gnu from the calling environment, and uses cp2k instead.
srun --uenv=cp2k --view=develop ./build
# start another sbatch job that does not automatically load
# the prgenv-gnu uenv
sbatch --uenv-passthrough=ignore ./nested-job
# start another sbatch job that uses the prgenv-gnu uenv
sbatch --uenv-passthrough=use ./nested-job
Breaking change: calling sbatch from inside a uenv session
In v10, calling sbatch or salloc from inside an active uenv start session is disabled by default and will produce an error:
$ uenv start prgenv-gnu/25.6:v2 --view=default
$ sbatch my-job.sh
error: Calling sbatch/salloc from inside a uenv session is disabled by default.
Use --uenv-passthrough to declare your intent explicitly:
Views¶
Starting a uenv runs in a process with the software mounted at /user-environment or /user-tools, however no changes are made to environment variables like $PATH.
Uenv images provide views, which will set environment variables that load the software into your environment.
Views are loaded using the --view flag for uenv start, uenv run and the Slurm plugin (all documented above).
loading views
# activate the view named default in prgenv-gnu
$ uenv start --view=default prgenv-gnu/24.11:v1
# activate both the spack and modules views in prgenv-gnu using
# a comma-separated list of view names
$ uenv start --view=spack,modules prgenv-gnu/24.11:v1
# when starting multiple uenv, you can disambiguate using uenvname:viewname
$ uenv start --view=prgenv-gnu:default,editors:ed prgenv-gnu/24.11:v1,editors
Each uenv can provide more than one view.
The modules and spack are standard views provided by nearly all uenv.
To find a list of the views in a uenv, use uenv image inspect without having to mount it first.
listing views in a uenv
why is the default view not the default?
Some uenv, for example the prgenv-gnu, have a view named default.
This view is not loaded by default if no view is specified with the --view flag.
Uenv v10 introduces support for default views.
installing Python venv with uenv
Python virtual environments can be created on top of a uenv view.
However, to ensure that the Python interpreter and packages from the uenv view are used, the PYTHONPATH and PYTHONUSERBASE environment variables must be set correctly, see our guide on building Python virtual environments with uenv.
Default views¶
uenv v10.0
Uenv images can provide a default view.
When a default view is declared and no --view flag is given, that view is loaded automatically.
- To opt out of the automatic default view, use the
--no-default-viewflag; - Use
uenv image inspectto check whether a given uenv has one.
using default views
The uenv inspect command shows the views in a uenv, and marks defaults (if there are any).
In the linaro-forge image the forge view is a default view:
$ uenv inspect linaro-forge
repo default:/ritom/scratch/cscs/bcumming/.uenv-images
linaro-forge/25.1:v2@starlex%gh200 mount at /user-tools
views:
spack: configure spack upstream
forge (default):
When the uenv is started without a view, the forge view is loaded by default.
Note
Not all uenv provide a default view. This is a new feature, so uenv built before it was introduced will have no default.
Uenv with no default view work exactly as they did before.
Modules view¶
Most uenvs provide the modules, that can be accessed using the module command.
By default, the modules are not activated when a uenv is started, and need to be explicitly activated using the module view.
using the module view
$ uenv start prgenv-gnu/24.11:v1 --view=modules
$ module avail
---------------------------- /user-environment/modules ----------------------------
aws-ofi-nccl/git.v1.9.2-aws_1.9.2 lua/5.4.6
boost/1.86.0 lz4/1.10.0
cmake/3.30.5 meson/1.5.1
cray-mpich/8.1.30 nccl-tests/2.13.6
cuda/12.6.2 nccl/2.22.3-1
fftw/3.3.10 netlib-scalapack/2.2.0
fmt/11.0.2 ninja/1.12.1
gcc/13.3.0 openblas/0.3.28
gsl/2.8 osu-micro-benchmarks/5.9
hdf5/1.14.5 papi/7.1.0
kokkos-kernels/4.4.01 python/3.12.5
kokkos-tools/develop superlu/5.3.0
kokkos/4.4.01 zlib-ng/2.2.1
libtree/3.1.1
$ module load cuda gcc cmake
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Cuda compilation tools, release 12.6, V12.6.77
$ gcc --version
gcc (Spack GCC) 13.3.0
$ cmake --version
cmake version 3.30.5
bash: module: command not found
Version 9.0.0 of uenv, installed on October 22 2025, has a bug that removes the module command on Santis and Clariden.
Note
The issue does not affect uenv in Slurm jobs, or on Daint and Eiger.
Note
A fix has been implemented, and will be installed as soon as possible.
Specifically, the module command is not available inside uenv start sessions:
The workaround is to manually load the module tool after starting your uenv session:
Spack view¶
uenv images provide a full upstream Spack configuration to facilitate building your own software with Spack using the packages installed inside as dependencies.
No view needs to be loaded to use Spack, however all uenv provide a spack view that sets some environment variables that contain useful information like the location of the Spack configuration, and the version of Spack that was used to build the uenv.
For more information, see our guide on building software with Spack and uenv.