Research IT Cluster Usage - Lanzcos Cluster

Date: 2023-08-03
Revision: v0.1

Contact ops@tchpc.tcd.ie for any queries relating to this documentation.

Terminology

Hermit -> has been chosen to represent Lanczos users with additional privileges to download software at a system level.

Inspired loosely by the Lanczos Algorithm devised by Cornelius Lanczos to find the m eignvalues and eigenvectors of a square Hermitian matrix. This term is malleable.

Software locations

Hermits have additional privileges to:

/home/software/spack -> Install prefix for Hermit-installed spack packages
/home/software/pkgs -> Install prefix for other Hermit-installed packages

Read below for further information and guidelines on install software to the spack directory and to the pkgs directory.

Spack

Full documentation for Spack v0.20 on Lanczos.

Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share commoninstallations of software on clusters with exotic architectures, using libraries that do not have a standard ABI. Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system.

Spack Userspace Idea

Spack supports six configuration scopes to handle default behaviour. These scopes in order of decreasing priority are:

Scope	Directory
Command-line	N/A
Environment	In environment base directory (in `spack.yaml`)
Custom	Custom directory, specified with `--config-scope`
User	`~/.spack/`
Site	`$SPACK_ROOT/etc/spack/`
System	`/etc/spack/`
Defaults	`$SPACK_ROOT/etc/spack/defaults/`

Sysadmins already manage Defaults, and Site scopes to install software at a system level for all users. The plan is to utilize the User scope to allow dedicated persons to install software at a "group level" for users of Lanczos.

There may be no major distinction between software installed by RIT staff, and software installed by privileged Lanczos users apart from the install directory, and module location. If a distinction is required, perhaps adding a simple module prefix/suffix may be suitable.

Example: zlib/4.1.0 installed by RIT versus zlib/4.1.0 installed by a Hermitian, with compiler gcc/9.3.0 and same configuration (ie same hash).

zlib/4.1.0-gcc-9.3.0-hash
h/zlib/4.1.0-gcc-9.3.0-hash

Accessing Spack Install

Latest version of Spack is installed at /home/support/spack/spack. The following lines can be added to your .bashrc to access Spack application, though you will probably not be able to install software without following the configuration steps in the section below. Add path for new Spack modules with module use command.

# [In your ~/.bashrc file](#in-your-bashrc-file)
export SPACK_ROOT="/home/support/spack/spack"
source $SPACK_ROOT/share/spack/setup-env.sh
module use /home/software/modulefiles

Test Spack is now accessible with which spack.

Spack Configuration

Spack has many configuration files (.yaml) to define behaviour of compilers, build and install locations, module configurations and locations, etc etc. Read more about spack configuration files.

Most of these details are configured on the Default/System scope by sysadmins, with only minimal changes needed at the Hermit User scope. The config specifications at user level will override system and default config (read more about multiple user-scopes).

Users will change the following:

Install prefix (changing to /home/software/spack)
Module prefix (changing to /home/software/modulefiles)
Set system version of Spack as upstream (add upstream directory $SPACK_ROOT/opt/spack)

Required file changes:

Config: ~/.spack/config.yaml

config:
  install_tree:
    root: /home/software/spack
  source_cache: $user_cache_path/var/spack/cache

Modules: ~/.spack/modules.yaml

modules:
  default:
    roots:
      tcl: /home/software/modulefiles

Upstreams: ~/.spack/upstreams.yaml

upstreams:
  spack-rit:
    install_tree: $SPACK_ROOT/opt/spack/

Test Spack Configuration

Verify the following commands produce similar output:

Spack is on path:

[dkierans@lanczos01 ~]$ which spack
spack ()
{
    : this is a shell function from: /home/support/spack/0.20.1/share/spack/setup-env.sh;
    : the real spack script is here: /home/support/spack/0.20.1/bin/spack;
    _spack_shell_wrapper "$@";
    return $?
}

Read permissions on configuration files (gcc@13.1.0 should be available):

[dkierans@lanczos01 ~]$ spack compilers
==> Available compilers
-- gcc rocky8-x86_64 --------------------------------------------
gcc@8.5.0  gcc@13.1.0

Upstream packages are recognized. spack find should list packages installed

upstream by system version of spack.

[dkierans@lanczos01 ~]$ spack find
-- linux-rocky8-broadwell / gcc@8.5.0 ---------------------------
autoconf@2.69                bzip2@1.0.8    gdbm@1.23       libiconv@1.17    m4@1.4.19    perl@5.36.0    tar@1.34       zstd@1.5.5
autoconf-archive@2023.02.20  diffutils@3.9  gettext@0.21.1  libsigsegv@2.14  mpc@1.3.1    pigz@2.7       texinfo@7.0.3
automake@1.16.5              gawk@5.2.1     gmake@4.4.1     libtool@2.4.7    mpfr@4.2.0   pkgconf@1.9.5  xz@5.4.1
berkeley-db@18.1.40          gcc@13.1.0     gmp@6.2.1       libxml2@2.10.3   ncurses@6.4  readline@8.2   zlib@1.2.13

-- linux-rocky8-broadwell / gcc@13.1.0 --------------------------
berkeley-db@18.1.40  gdbm@1.23       libiconv@1.17     ncurses@6.4    python@3.10.10  util-linux-uuid@2.38.1
bzip2@1.0.8          gettext@0.21.1  libmd@1.0.4       perl@5.36.0    readline@8.2    xz@5.4.1
diffutils@3.9        libbsd@0.11.7   libxcrypt@4.4.33  pigz@2.7       sqlite@3.40.1   zlib@1.2.13
expat@2.5.0          libffi@3.4.4    libxml2@2.10.3    pkgconf@1.9.5  tar@1.34        zstd@1.5.5
==> 53 installed packages

Can install new zlib package (small package with no dependencies).

[dkierans@lanczos01 ~]$ # Check currently installed versions
[dkierans@lanczos01 ~]$ spack find zlib%gcc@13.1.0
-- linux-rocky8-broadwell / gcc@13.1.0 --------------------------
zlib@1.2.13
==> 1 installed package
[dkierans@lanczos01 ~]$ # install different version (check `spack info zlib` for available versions)
[dkierans@lanczos01 ~]$ spack install zlib@1.2.8
==> Warning: using "zlib@1.2.8" which is a deprecated version
==> Installing zlib-1.2.8-7n7inqvtq245fgxi6fjfczjrio7dephv
==> No binary for zlib-1.2.8-7n7inqvtq245fgxi6fjfczjrio7dephv found: installing from source
==> Warning: zlib@1.2.8 is deprecated and may be removed in a future Spack release.
==>   Fetch anyway? [y/N] y
==> Fetching https://mirror.spack.io/_source-cache/archive/36/36658cb768a54c1d4dec43c3116c27ed893e88b02ecfcb44f2166f9c0b7f2a0d.tar.gz
==> No patches needed for zlib
==> zlib: Executing phase: 'edit'
==> zlib: Executing phase: 'build'
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.8-7n7inqvtq245fgxi6fjfczjrio7dephv
  Stage: 3.88s.  Edit: 1.11s.  Build: 1.19s.  Install: 0.63s.  Post-install: 0.35s.  Total: 7.23s
[+] /home/software/spack/linux-rocky8-broadwell/gcc-13.1.0/zlib-1.2.8-7n7inqvtq245fgxi6fjfczjrio7dephv

Module for new zlib version is automatically generated at /home/software/modulefiles/

[dkierans@lanczos01 ~]$ ls -l /home/software/modulefiles/zlib/
total 4
-rw-r--r-- 1 dkierans dkierans 1491 Aug  3 19:09 1.2.8-gcc-13.1.0-7n7i
[dkierans@lanczos01 ~]$
[dkierans@lanczos01 ~]$ module av zlib
---------------------------------------------------- /home/software/modulefiles -----------------------------------------------------
zlib/1.2.8-gcc-13.1.0-7n7i

That is all. You should have free reign to install Spack packages in a location available to all users to use.

pkgs

We will try standardize the installation of software from source as much as possible. This is only a rough guide, which is suitable for installing software with:

Different versions of the software
Different compilers

And optionally:

Different mpi flavours
Different blas flavours

Additional customization would likely be needed if installing the software with different add-ons or features enabled. Ideally you would have one main install with as many features as needed enabled. This main install could be updated over time using the module system to hide the underlying changes.

The process will be explained via a sample install of siesta below.

Relevant Directories

The following directories will be relevant:

Directory	Explanation
`/home/software/pkgs/src/<PACKAGE>/`	Source code directory
`/home/software/pkgs/<PACKAGE>/<VERSION>-<COMPILER>-<COMPILER_VERSION>/`	Install prefix directory
`/home/software/modulefiles/<PACKAGE>/`	Modulefile directory

Sample Install

The DFT code Siesta will be used as an example for good practice on how to install code. Below is an 8-ish step guide on how to install and setup Siesta.

Step 1 - Set up directories

> PACKAGE=siesta
> mkdir -p /home/software/pkgs/src/$PACKAGE
> mkdir -p /home/software/pkgs/$PACKAGE
> mkdir -p /home/software/modulefiles/$PACKAGE

Step 2 - Get source code

Source code for Siesta can be found on gitlab.

> cd /home/software/pkgs/src/$PACKAGE
> git clone https://gitlab.com/siesta-project/siesta.git
> cd siesta

Alternatively you may be able to download a tarball containing source code from the web. Eg OpenMPI v4.1 releases:

> cd /home/software/pkgs/src/openmpi
> wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.5.tar.gz
> tar xvzf openmpi-4.1.5.tar.gz
> cd openmpi-4.1.5

Step 3 - Configure install

Load any necessary dependencies and configure install prefix.

This step will vary based on installation methods (eg configure.sh, cmake, gnu make etc), SIESTA supports Cmake to configure your install. This build uses Scalapack (dependencies Intel MPI \& MKL).

Most important here is the PKG-VERSION variable set in the first line. This is an identifier which will be used to define the details of your install and install location. Good formats are dependent on the package and it's dependencies:

Purpose	PKG_VERSION Format	Samples
Default	`<VERSION>-<COMPILER>-<COMPILER_VERSION>`	`siesta/4.1.5-intel-2023.1.0`
Multiple blas variants	`<VERSION>-<COMPILER>-<COMPILER_VERSION>-<BLAS>-<BLAS_VERSION>`	`siesta/4.1.5-intel-2023.1.0-mkl-2023.1.0`
Multiple mpi variants	`<VERSION>-<COMPILER>-<COMPILER_VERSION>-<MPI>-<MPI_VERSION>`	`siesta/4.1.5-intel-2023.1-openmpi-4.1.5`

Similarly you may want to specify other extensions and features, eg

CP2K without libbeef (default): cp2k/1.2.3
CP2K with libbeef: cp2k/1.2.3-libbeef

> PKG-VERSION="YOUR_PACKAGE_VERSION_GOES_HERE"
> # Load any build dependencies
> module load cmake
> module load oneapi-compiler/2023.1.0 oneapi-mpi/2021.9.0 oneapi-mkl/2023.1.0
> # Run config setup
> cmake -S. -B_build -DCMAKE_INSTALL_PREFIX=/home/software/pkgs/$PACKAGE/$PKG-VERSION ...
> cmake --build _build -j 8 # building in parallel with 8 procs

Step 4 - Test Configuration (optional)

Next it is recommended to test the configuration before completing install, using any provided test suites. Read package documentation for more information on this. Siesta supports testing using ctest. Gnu Makefiles often use commands make test or make check to test configuration.

Step 5 - Install

Complete installation if you are happy with test results from step 4.

> cmake --install _build

Step 6 - Create Module File

Modulefiles on Lanczos are written in tcl - read documentation.

Copy the following modulefile template and symlink to main module tree:

> cp /home/software/modulefiles/template.mod /home/software/$PACKAGE/$PACKAGE_VERSION/modulefiles/$PACKAGE_VERSION
> ln -s /home/software/$PACKAGE/$PACKAGE_VERSION/modulefiles/$PACKAGE_VERSION /home/software/modulefiles/$PACKAGE

View of the tlc module template:

[dkierans@lanczos01 siesta]$ cat -n /home/software/modulefiles/template.mod
     1  #%Module1.0
     2  ## author: <author_name> (<author_email>)
     3  ## date: <yyyy-mm-dd>
     4  ##
     5
     6  ## LOCAL VARIABLES
     7  # Eg: set pkg "cp2k"
     8  set pkg "<package_name>"
     9  # Eg: set version "2023.1-gcc-13.1.0"
    10  set version "<package_version>"
    11  # Shouldn't need to change `base` if pkg and version set correctly
    12  set base "/home/software/${pkg}/${version}/"
    13
    14  ## MODULE DESCRIPTIONS & HELP
    15  module-whatis "Name : ${pkg}"
    16  module-whatis "Version : ${version}"
    17  module-whatis "Target : x86_64"
    18  module-whatis "Short description : <short_package_descript>"
    19  module-whatis "Prerequisites: <prereq_modules>"
    20  module-whatis "Configuration: <path_to_config_script> or <config_command>"
    21  module-whatis ""
    22
    23  proc ModulesHelp { } {
    24  puts stderr "<long_package_description>"
    25  puts stderr "\n source: <url_to_main_website>"
    26  }
    27
    28  ## MODULE CONFLICTS
    29  conflict "${pkg}"
    30
    31  ## MODULE DEPENDENCIES
    32  # Uncomment if needed, repeat for multiple dependencies
    33  # eg Package requires DEP="oneapi-mkl/2023.1.0" to be loaded
    34  #DEP="<module_dependency>"
    35  #if { ![ is-loaded "$DEP" ] } { module load "$DEP" }
    36
    37  ## PATHS & ENVIRONMENT VARIABLES
    38  setenv [string toupper $pkg]_ROOT "${base}"
    39  prepend-path PATH "${base}/bin"
    40  prepend-path LD_LIBRARY_PATH "${base}/lib"
    41  prepend-path LIBRARY_PATH "${base}/lib"
    42  prepend-path PKG_CONFIG_PATH "${base}/lib/pkgconfig"
    43  prepend-path CPATH "${base}/include"
    44  prepend-path CMAKE_PREFIX_PATH "${base}/"
    45  prepend-path MANPATH "${base}/share/man"
    46
    47  ## ADDITIONAL PATH & ENVIRONMENT VARIABLES
    48

You will want to replace the following in your modulefile /home/software/modulefiles/$PACKAGE/$PACKAGE_VERSION

Line	Variable Name	Variable Description
2	`<author_name>` and, `<author_email>`	Name and contact email (optional) of person who installed software
3	`<yyyy_mm_dd>`	Date package was installed
8	`<package_name>`	Name of package (as per `PACKAGE` variable used for install path)
10	`<package_version>`	Version of package (as per `PKG-VERSION` variable used for install path)
18	`<short_package_description>`	Short description of package
19	`<prereq_modules>`	List of any prerequisite module dependencies (used later)
20	`<path_to_config_script>` or, `<config_command>`	Path to configuration script used to install software, or configuration command used to setup software.
24	`<long_package_description>`	Long description of package (usually 1 paragraph taken from `About` section of relevant website)
25	`<url_to_main_website>`	Link to website of package (eg `https://siesta-project.org/siesta/About/overview.html`)
34	`<module_dependency>`	Uncomment lines 34&35 if any modules required as dependencies, eg blas library. Repeat lines 34&35 with each dependency if multiple required

Line 38 sets environment variable for the package install root, this line usually does not need to be changed. Eg if OpenMPI v4.1.4 is installed with GCC v13.1.0 compiler, it is equivalent to setting:

> export OPENMPI_ROOT="/home/software/openmpi/4.1.4-gcc-13.1.0"

Lines 39-45 cover the basic PATH variables needed for most software, but this can vary a lot. Read package documentation for more information. However as a general rule of thumb;

if ./bin exists, add it to PATH
if ./lib exists, add it to LD_LIBRARY_PATH and LIBRARY_PATH
if ./lib64 exists, add it to LD_LIBRARY_PATH and LIBRARY_PATH
if ./lib/pkgconfig or ./lib64/pkgconfig exists, add it to PKG_CONFIG_PATH
if ./include exists, add it to CPATH
if ./share/man exists, add it to MANPATH

Otherwise you can comment out or remove the relevant line.

Step 7 - Test Installation

You should be able to load the module using:

> module load <PACKAGE>/<PACKAGE_VERSION>

Run some tests on the software to make sure it's running as expected!

Step 8 - Debugging

Hopefully all the tests run successfully but often they might not. Review your configuration, make sure all depedencies are loaded, google any error codes/messages, etc.

Contact ops@tchpc.tcd.ie if you are having further trouble. Read our support page for information how to report issues.

But hopefully if all went well then you're installation is done. All lanczos users should be able to load your module and run the software, but only elevated users (Hermits) should be able to re-configure or edit the installation and source files.

[...Work in progress...]