Getting Started with DisCoTec

Installing DisCoTec

There are two established ways of installing: Spack and CMake. After installing all dependencies, Spack actually delegates to CMake to build DisCoTec.

… with Spack

DisCoTec can be installed via spack, which handles all dependencies.

If you want to use DisCoTec as a library only, follow the usual Spack installation instructions until the command spack install discotec@main (or an extended version of it) succeeds.

If you want to develop DisCoTec code or examples, the spack dev-build workflow is recommended as follows.

Clone both spack and DisCoTec to find or build the dependencies and then compile DisCoTec:

git clone https://github.com/spack/spack.git
./spack/bin/spack external find  # find already-installed packages
./spack/bin/spack compiler find  # find compilers present on system
./spack/bin/spack info discotec@main  # shows DisCoTec's variants

 # shows DisCoTec's dependency tree and which parts are already found
./spack/bin/spack spec discotec@main
# optional: configuring external packages according to Spack documentation

git clone https://github.com/SGpp/DisCoTec.git
cd DisCoTec
../spack/bin/spack dev-build -b install discotec@main

This will first build all dependencies, and then build DisCoTec inside the cloned folder. The executables are placed in the respective example and test folders.

To use DisCoTec in another CMake project, you can use find_package:

find_package(discotec REQUIRED)
target_link_libraries(your_target discotec::discotec)

Alternatively, for development builds you can use add_subdirectory(DisCoTec/src).

If you encounter problems with the Spack installation, check out the Spack command documentation, and see if other users had the same spack issues or DisCoTec issues before.

… with CMake

Dependencies

cmake >= (3.24.2), libmpich-dev (>= 3.2-6), or other MPI library libboost-all-dev (>= 1.60) (or the subset test, serialization, filesystem, system, program_options, date_time Boost libraries)

Additional (optional) dependencies:

  • OpenMP

  • HDF5

  • HighFive

  • lz4

  • glpk

  • paliwa (with DDC and Kokkos, for wavelet-based hierarchization)

You can also install the dependencies with Spack, and spack load them, before executing the next steps.

Complete build

git clone https://github.com/SGpp/DisCoTec.git DisCoTec
cmake -S DisCoTec -B DisCoTec/build
cmake --build DisCoTec/build -j

Advanced build

All examples and the tools can be built on their own. To build a specific example, run the specific cmake command in the example folder. For example, run the following commands

git clone https://github.com/SGpp/DisCoTec.git DisCoTec
cd DisCoTec/examples/combi_example
cmake -S . -B build
cmake --build build -j

to build the combi_example.

For building only the DisCoTec library, run cmake with the src folder as source folder.

Optional CMake Options

  • DISCOTEC_TEST=**ON**|OFF - Build tests if you build the complete project.

  • DISCOTEC_TIMING=**ON**|OFF - Enables internal timing

  • DISCOTEC_USE_HIGHFIVE=**ON**|OFF - Enables HDF5 support via HighFive.

  • DISCOTEC_OPENMP=ON|**OFF** - Enables OpenMP support.

  • DISCOTEC_ENABLEFT=ON|**OFF** - Enables algorithm-based fault tolerance.

  • DISCOTEC_USE_LTO=**ON**|OFF - Enables link time optimization if the compiler supports it.

  • DISCOTEC_WITH_COMPRESSION=ON|**OFF** - Enables compressed output with LZ4.

  • DISCOTEC_USE_PALIWA=ON|**OFF** - Enables wavelet-based hierarchization via the paliwa library. Requires paliwa, DDC, and Kokkos.

  • DISCOTEC_WITH_SELALIB=ON|**OFF** - Looks for SeLaLib dependencies and compiles the matching example

Run the tests

Beware: Most of the tests in DisCoTec need significant resources (eight to nine MPI processes, file I/O, some performance tests when built in Release mode). Make sure you run them on appropriate hardware.

To run the compiled tests, go to folder tests and make sure the correct MPI runtime is loaded to your path, e.g. with

. ../spack/share/spack/setup-env.sh
spack load openmpi # <- use the one you compiled with
mpiexec --version # <- expected output?

Then, run

mpiexec -np 9 ./test_distributedcombigrid_boost

where you can use all the parameters of the boost test suite. Beware that the tests can already use significant resources. If timings matter, consider pinning.

Run an Example

DisCoTec executables are typically configured through ctparam files, which are parsed on startup. The ctparam file will contain the combination technique parameters (dimension, minimum and maximum level) as well as parallelization parameters (number and size of process groups, parallelism within process groups) in .ini file format.

Any DisCoTec executable must be run through MPI (either mpirun or mpiexec), and if no argument to the binary is specified, it will use the file called ctparam in the current working directory. Make sure that it exists and describes the parameters you want to run.

As with the tests, make sure the correct MPI is loaded to your path.

The exact format and naming in ctparam is not (yet) standardized, to allow adaptation for different PDE solver applications. Please refer to existing parameter files and example implementations.

The different examples intend to illustrate various aspects of DisCoTec:

  • combi_example: manager-worker setup with a generic TaskExample that interpolates a cosine function on the component grids

  • combi_example_faults: like combi_example, but with fault tolerance functionality (requires that DisCoTec is built with the DISCOTEC_ENABLEFT flag)

  • distributed_advection: manager-worker setup using a first-order upwinding advection solver, illustrates how to evaluate the Monte-Carlo norms (and error norms, as an analytical solution is given for this PDE problem)

  • combi_workers_only: like distributed_advection, but with worker-only setup; illustrates the static task assignment required for worker-only scenarios

  • distributed_third_level: like distributed_advection and combi_workers_only; adds logic for widely-distributed simulations

  • GENE examples, starting with gene_: show the interaction with the GENE gyrokinetic solver (using a complicated build/link setup that we don’t recommend to use any more, see next example instead)

  • selalib_distributed: shows the interaction with a Semi-Lagrangian Vlasov solver in SeLaLib: a static library is built as part of SeLaLib, and linked through DisCoTec’s Task interface in SelalibTask.

Pinning With Various MPI Implementations

Correct pinning is of utmost importance to performance, especially if DisCoTec was compiled with OpenMP support. The desired pinning is the simplest one you can imagine: Each node and, within a node, each socket is filled consecutively with MPI rank numbers. If OpenMP is used, MPI pinning is strided by the number of threads. The threads should be placed close to their MPI rank. This means for all compilers, if the desired number of OpenMP threads per rank is N, one needs

export OMP_NUM_THREADS=$N;export OMP_PLACES=cores;export OMP_PROC_BIND=close

If OpenMP is used for compilation but should not be used for execution, N should be set to 1 to avoid unintended effects.

The number and size of process groups (in MPI ranks) can be read from the ctparam files:

NGROUP=$(grep ngroup ctparam | awk -F"=" '{print $2}')
NPROCS=$(grep nprocs ctparam | awk -F"=" '{print $2}')

Intel-MPI

Intel-MPI requires some environment variables, in particular for OpenMP:

mpiexec -n $(($NGROUP*$NPROCS)) -genv I_MPI_PIN_PROCESSOR_LIST="allcores" -genv I_MPI_PIN_DOMAIN=omp $DISCOTEC_EXECUTABLE

In SLURM, it is important to set --ntasks-per-node to match the number of desired tasks ($CORES_PER_NODE/$OMP_NUM_THREADS). Validate with verbose output: export I_MPI_DEBUG=4

OpenMPI

OpenMPI uses command line arguments, pinning may clash with SLURM settings depending on the exact call.

mpiexec.openmpi --rank-by core --map-by node:PE=${OMP_NUM_THREADS} -n $(($NGROUP*$NPROCS)) $DISCOTEC_EXECUTABLE

Validate with verbose output: --report-bindings

MPT

With mpt, one uses the omplace wrapper tool to set the correct pinning.

mpirun -n $(($NGROUP*$NPROCS)) omplace -v -nt ${OMP_NUM_THREADS} $DISCOTEC_EXECUTABLE

Validate with very verbose output: -vv .